Activity Graph

Page 1 of 22 | Next »


[20:57:23] bradland: has joined #ruby
[20:58:16] bradland: has joined #RubyOnRails
[20:59:49] bradland: I'm using Apache mod_unique_id to send a request ID through to Rails. This request ID is logged for each request, so that I can tie requests together across logs (web server/app server).
[21:00:16] bradland: However! The request IDs don't always match.
[21:00:30] bradland: Specifically, it looks like the at symbol is being lost somewhere along the line.
[21:01:01] bradland: For example: Apache log WkVXE0LkM@AAAGUvTSsAAABP; Rails log WkVXE0LkMAAAGUvTSsAAABP.
[21:01:31] bradland: The ID is being passed in through an Apache directive: RequestHeader set X-Request-Id "%{UNIQUE_ID}e"
[21:01:48] bradland: Anyone have any ideas what's stripping the @ symbol? Is it in Rack?
[21:18:04] bradland: I think I found my culprit. The RequestId middleware stripping anything that isn't in [\w\-].
[21:18:07] bradland:
[21:36:51] bradland: Quit: bradland


[16:46:58] bradland: Looks like {timezone: "EDT", end: SECONDS, start: SECONDS, gmtoffset: SECONDS} to me
[16:47:33] bradland: Wixy: ActiveSupport has a module for that:
[16:48:06] bradland: What ruby version?
[16:50:04] bradland: Have a look at Time in StdLib as well. There are some additional methos in there related to time zones.
[16:50:52] bradland: Before you get too deep down this rabbit hole, consider this advice: always use UTC internally, and consider time zones a localication matter.
[16:51:44] bradland: just add the seconds and you have UTC.
[16:52:59] bradland: Quit: bradland


[16:32:31] bradland: has joined #ruby
[19:16:22] bradland: adaedra: that means it's his first language.
[19:19:11] bradland: you're curious about matz changing the ruby core code and not noting it in english?
[19:21:28] bradland: fujinuma: Ruby is open source. It has many, many contributors, and you can watch its development openly on Github.
[19:22:13] bradland: If you click on the "commits" link near the top of that page, you can see all of the changes that are happening to Ruby on a daily basis.
[19:23:28] bradland: fujinuma: i'm curious why you're asking?
[19:24:45] bradland: ok, that's what i thought. i have no interest in helping anyone disparage matz. good day.


[21:01:20] bradland: Quit: Lost terminal
[21:01:45] bradland: has joined #ruby
[21:02:06] bradland: Client Quit
[21:02:27] bradland: has joined #ruby
[21:03:24] bradland: looking for a hand with some regex.
[21:04:00] bradland: i'm using lograge with the lines tem for formatting the payload, but some of my log entries are appended with this ellipse, which the Lines parser chokes on.
[21:04:38] bradland: i can always gsub that '...' off after the fact but that adds quite a bit of time to file processing when handling lots of lines.
[21:04:47] bradland: so i'd prefer to simply ignore it in my regex capture.
[21:06:09] bradland: i would kind of like it to be in its own capture group so that i can do some analysis on when it's showing up.
[21:06:14] bradland: that's kind of its own problem.
[21:06:31] bradland: yeah, shevy i've been heads down over at rubular for a while.
[21:10:51] bradland: atmosx: not a bad idea
[21:20:24] bradland: jhass: turns out making payload non-greedy works, but it requires more steps than simply making the ellipse group optional like so
[21:20:50] bradland: was going to make it non-capturing, but i need to debug this behavior anyway
[21:21:12] bradland: so i can easily find out which requests are appending the ellipse if it's in a group
[21:23:18] bradland: just noticed that very thing
[22:11:12] bradland: Quit: bradland


[18:44:54] bradland: has joined #ruby


[23:12:57] bradland: *.net *.split


[20:47:21] bradland: *.net *.split
[21:50:23] bradland: has joined #ruby


[22:32:03] bradland: Ping timeout: 240 seconds
[22:39:42] bradland: has joined #ruby


[02:23:51] bradland: Ping timeout: 248 seconds
[02:26:56] bradland: has joined #ruby


[18:50:59] bradland: Ping timeout: 268 seconds
[18:51:37] bradland: has joined #ruby


[22:40:58] bradland: Ping timeout: 240 seconds
[23:06:09] bradland: has joined #ruby


[12:25:04] bradland: *.net *.split
[12:28:16] bradland: has joined #ruby


[21:31:21] bradland: silly question, but is there a shortcut for writing CSV::Table objects back to a CSV file?
[21:32:36] bradland: it looks like #to_csv and writing to a string is it.


[13:39:12] bradland: Ping timeout: 265 seconds
[14:01:45] bradland: has joined #ruby


[07:43:45] bradland: Ping timeout: 260 seconds
[08:01:21] bradland: has joined #ruby


[16:43:35] bradland: Ping timeout: 240 seconds
[16:56:00] bradland: has joined #ruby


[20:00:56] bradland: Ping timeout: 244 seconds
[20:40:12] bradland: has joined #ruby


[00:00:45] bradland: so, i have a UDP stream coming in on 5606
[00:00:49] bradland: i know how to read from a socket
[00:01:09] bradland: and i'm about to set about the task of handling the data that arrives
[00:01:14] bradland: some background might help
[00:01:20] bradland: i have a game called ProjectCARS
[00:01:33] bradland: the game broadcasts telemetry data on port 5606 at a configurable interval
[00:01:48] bradland: i'm at the starting phase, but this seems like it might be a solved problem
[00:01:52] bradland: i'm just not sure where to start searching
[00:02:05] bradland: i think it might be called strem processing
[00:02:11] bradland: *stream processing
[00:02:14] bradland: but i'm not sure
[00:02:32] bradland: i want to listen on the port, detect, and process the data that arrives
[00:02:51] bradland: jbrhbr: that's a good way to put it
[00:03:12] bradland: for example, the payloads that arrive easily processed by the bindata gem
[00:03:41] bradland: but there's the matter of listening for the data, detecting the type of payload (it's in the first few bytes), and dispatching to the correct handler
[00:03:48] bradland: all of this seems like a "solved probleM
[00:04:10] bradland: so rather than setting out to my own poorly engineered solution, i was hoping to look at other code that does similar things
[00:04:57] bradland: yeah, i've managed to fetch some bytes from the wire
[00:05:05] bradland: but i think i need threads or an event loop
[00:05:12] bradland: because the bytes are going to keep coming
[00:05:21] bradland: so they need to be read in to a buffer that is processed by the parser
[00:05:37] bradland: ah, good to know
[00:06:01] bradland: so, for example, i could read 1024 bytes, process them, then read more?
[00:08:02] bradland: i have captured some data using this
[00:08:18] bradland: i used that to collect the data, then have a look at what's in there (in a hex editor)
[00:09:21] bradland: so from what i've observed, it's a stream of null bytes until some data shows up
[00:09:35] bradland: so, i'll get nul, then \x74
[00:09:52] bradland: that's always the signature of the beginning of the blob
[00:10:18] bradland: so, unfortunately it's not just a matter of waiting on data to arrive
[00:10:34] bradland: because \x00\x00\x00... is always showing up
[00:10:47] bradland: or at least that's how it looks from my capture code
[00:11:06] bradland: so my listener is going to have to do some detecting work
[00:12:12] bradland: at the first occurance of non-null bytes, pop 2-bytes and verify that they're equal to 1140, then read 1-byte and determine the "packet_type"
[00:12:22] bradland: which is really the *package* type
[00:12:38] bradland: i don't like that they used the term packet, because this is net code.
[00:12:51] bradland: so that means the game is transmitting \0 on the wire?
[00:13:09] bradland: i have the game running
[00:13:11] bradland: let me see what i get
[00:13:37] bradland: i used tcpdump too
[00:13:42] bradland: come to think of it
[00:13:50] bradland: i compared the two and both showed the \0 stream
[00:16:23] bradland: yeah, i'm getting the streams of \0
[00:16:40] bradland: so it has to be the game
[00:16:59] bradland: yeah, "feature"
[00:17:06] bradland: there are quite a few clients for this
[00:17:23] bradland: so if you're receiving the \0 stream, you can say with some confidence that you're "connected"
[00:17:32] bradland: makes the stream processing a bit more challenging though
[00:17:36] bradland: because you can't simply wait for data
[00:17:44] bradland: read 1 byte at a time?
[00:17:50] bradland: i thought about that, but was worried about performance
[00:18:32] bradland: was thinking of reading in a chunk of 1024, scanning for the first non \0, then appending chunks until we reach the right size for the packet type (byte 3)
[00:19:57] bradland: mg^: what do you mean "in whole"?
[00:20:06] bradland: i get a byte stream
[00:21:44] bradland: yeah, packet issues are handled by UDPSocket
[00:21:48] bradland: i'm not concerned with that
[00:21:58] bradland: i have to process the payload, which may be split across UDP packets
[00:22:37] bradland: i used to work in VoIP, so i'm painfully familiar with the loose nature of UDP
[00:23:00] bradland: this is all LAN based, so it's generally pretty reliable
[00:23:06] bradland: there's no checksum in this netcode though
[00:23:55] bradland: have a look at this:
[00:24:06] bradland: those are the structs that come across the wire
[00:24:13] bradland: the largest is 1367 bytes
[00:24:29] bradland: no checksum
[00:25:08] bradland: mg^: hahaha... no
[00:25:19] bradland: see the first two fields
[00:25:29] bradland: sBuildVersionNumber and sPacketType?
[00:25:45] bradland: the first is a fixed value
[00:26:00] bradland: the second you run mod 4 against it and it gets you your packet type
[00:26:04] bradland: you pick a struct and run with it
[00:26:30] bradland: since they have defined sizes, you can read 1367 bytes and attempt to process it
[00:26:49] bradland: *normally*, you'd get a checksum field early on so you could find out if you got al lthe data
[00:26:54] bradland: right on, thanks adaedra
[00:27:33] bradland: mg^ this code uses the bindata gem to pull apart a blob once i've identified it: 1367
[00:27:38] bradland:
[00:28:04] bradland: bindata could be thought of as a DSL built around String#unpack
[00:28:55] bradland: FCS won't guarantee that UDP arives in order
[00:29:02] bradland: it will only ensure the frame is intact
[00:29:25] bradland: at the app layer, a checksum helps you discard data you can't use
[00:29:44] bradland: yeah, i have bounds, so i could bounds check
[00:29:55] bradland: toss cookies if it fails
[00:31:33] bradland: "Pit Boss" app


[23:59:52] bradland: looking for some guidance in the form of libraries i could look at or names of patterns/search terms i could use to learn a little more about processing network streams


[15:42:56] bradland: So I think I ended up answering some of my own questions while writing this up, but I'd love any feedback/comments on this gist:
[15:43:28] bradland: I'm trying to write a library for reading and parsing a UDP stream that is broadcast by the racing simulator ProjectCARS.
[16:07:12] bradland: ddubs: i used tcpdump to grab the data, and it's identical to the pcars.dump content.
[16:09:11] bradland: so the game setting is a time interval, so i'm thinking it just dumps the contets of this C storage structure to the wire at the specified interval
[16:10:03] bradland: that means i should be able to detect any non-zero value and read the length of the number of bytes that constitute SharedMemory.h
[16:11:08] bradland: ddubs: added the SharedMemory.h file to the gist.
[16:11:26] bradland: there's no clear licensing on that code (and it is copyright), so I may not be able to levae it up indefinitely.
[17:03:59] bradland: linocisco: what's your background. is this your first language?
[17:07:17] bradland: linocisco: i'd second the recommendation of "Programming Ruby" by Dave Thomas
[17:07:57] bradland: linocisco: I would also recommend "Design Patterns in Ruby" by Obie Fernandez
[17:08:10] bradland: that book will help you once you've learned the basics of the Ruby language
[17:11:17] bradland: most of the problems you encounter will already have solutions
[17:11:33] bradland: a "design pattern" is an approach to solving a common problem
[17:11:44] bradland: it's a higher level concept than simply writing code
[17:11:49] bradland: it has to do with how you structure your code
[17:14:03] bradland: linocisco: yes, that would be a sensible approach.
[17:16:29] bradland: if your target is web programming, i would suggest starting out with something like Sinatra, which is a simple library for authoring websites that are powered by ruby
[17:16:32] bradland: Sinatra is very, very simple. it maps routes (URL paths) to code, but that's all.
[17:16:33] bradland: that will allow you to focus on writing Ruby code, rather than learning a framework.
[17:16:38] bradland: depending upon your final application goals, you may choose to stay with Sianatra, or move to a larger framework, such as Ruby on Rails.
[17:19:44] bradland: sometimes business requirements dictate that data must be encrypted while at rest
[17:21:34] bradland: alvarolm: pinning this on a MitM attack as a result of a TLS break is a pretty far stretch, unless you mean to say that someone has lost control of their CA.
[17:22:03] bradland: in that case, it's "not your problem", but I know how this goes. sometimes it's your problem, regardless.
[17:22:25] bradland: if the business needs dictate that the data needs to be encrypted while at rest, then that's the requirement.
[17:22:38] bradland: *however* I would not drop TLS.
[17:23:14] bradland: there's nothing silly about using TLS for transport, even if you're sending encrypted data.
[17:24:03] bradland: well, i'm still adequately impressed, havenwood
[17:24:25] bradland: i'll star anything with Array#pack in it anyway
[17:25:29] bradland: which isn't a terrible suggestion
[17:25:29] bradland: linocisco: i think he meant instead of the design patterns book.
[17:26:25] bradland: linocisco: Practical Object Oriented Design in Ruby is more general, which would help you understand the Design Patterns book better.
[18:22:51] bradland: shevy: you sure that was ever Stdlib?
[18:54:46] bradland: Papierkorb: the reason you're having difficulty getting an answer is because the question isn't entirely clear. internally, you can't affect the byte order of Fixnum within Ruby. you can only affect the byte order of buffers (strings in Ruby).
[18:55:26] bradland: so you read from the wire in to a string, then use pack/unpack to change the byte order for what will be sent back out over the wire, or persisted somewhere.
[18:56:21] bradland: Papierkorb: you'll need to post some code to get more help than what you've got so far.
[18:56:57] bradland: and parts[7] is a string value, I presume?
[18:57:20] bradland: highly recommended that you get familiar with String#unpack and Array#pack if you're going to work with bytes
[18:58:21] bradland: also, recommend that you set your string encoding to BINARY when creating/modifying strings you intend to use as binary buffers
[19:00:36] bradland: Papierkorb: i'll see if i can wire up a method def for you, but i need a few to get through my current work item.
[19:02:21] bradland: Papierkorb: Array#pack and String#unpack are the fastest (performance wise)
[19:02:46] bradland: i've benchmarked them
[19:03:55] bradland: reversing the byte order of a string buffer
[19:05:11] bradland: Array#pack and String#unpack are highly optimized
[19:05:29] bradland: i wouldn't be surprised if they beat the overhead of FFI for such a basic operation
[19:09:31] bradland: to be clear, i haven't benchmarked them against FFI, personally
[19:11:37] bradland: and here i am frantically copy/pasting in to a spreadsheet so i can deconstruct this SharedMemory.h file :)
[19:23:40] bradland: Papierkorb: did you work out a solutoin?
[19:23:47] bradland: I just wrapped up my other work.
[19:24:43] bradland: looks like FFI wins!
[19:26:37] bradland: a = 0x11223344
[19:27:44] bradland: Papierkorb: eh, i'm not sure this benchmark is valid
[19:28:13] bradland: for bits and ntoh, you're operating on a Fixnum, but for pack/unpack you're operating on a string
[19:28:28] bradland: so, presumably, this value comes from somewhere over a wire, right?
[19:28:34] bradland: which is going to be IO, which will result in a string
[19:28:38] bradland: not a Fixnum
[19:28:55] bradland: so, you get a Fixnum?
[19:29:40] bradland: in the pack benchmark, you're performing extra work
[19:32:26] bradland: Papierkorb: the bits and ntoh bencmarks are starting with a Fixnum, while pack has to start from a String
[19:32:35] bradland: which results in extra pack/unpack calls
[19:32:48] bradland: not that i think it's going to get down to 0.09s, but it's worth noting
[19:32:51] bradland: this is not apples-to-apples
[19:33:39] bradland: if you start with a string, your benchmarks for bits and ntoh need to do the work of converting from String to Fixnum within the benchmark


[15:37:31] bradland: *.net *.split
[15:41:40] bradland: has joined #ruby
[15:51:23] bradland: *.net *.split
[16:47:55] bradland: has joined #ruby


[22:44:30] bradland: has joined #ruby


[03:45:04] bradland: *.net *.split


[13:27:04] bradland: Ping timeout: 250 seconds
[13:27:17] bradland: has joined #ruby


[20:35:12] bradland: don't let the notion of a good salary draw you in to the SF startup land
[20:35:18] bradland: be sure a startup is what you want to do first


[20:34:08] bradland: has joined #ruby
[21:16:21] bradland: when using alternation in Regex, how does the regex engine scope the alternation characters.
[21:16:41] bradland: for example, in this code, is the alternation between 4|5 safe? as in, will the engine expand to include additional chars?
[21:16:44] bradland:
[21:18:07] bradland: Ox0dea: so group around the alternation, like this?
[21:18:11] bradland: that makes it more explicit
[21:18:21] bradland: also, would you prefer the ALTERNATION option or the CLASS syntax
[21:19:15] bradland: so /Version: 4|5/ is "Version: 4" or "5" ?
[21:19:58] bradland: gotcha, the only reason i was looking at alternation was because i felt like it communicated intent better
[21:20:15] bradland: with the group and alternation, i create a capture group that i won't use
[21:20:26] bradland: not sure how horrible that is
[21:22:19] bradland: Ox0dea: I see, so that means either really communicates what I want, provided the reader understands regex.
[21:22:22] bradland: Thanks for the feedback!


[03:42:29] bradland: Remote host closed the connection