biox

Activity Graph

Page 1 of 1

2018-07-02

[02:12:44] biox: Quit: probably server crash

2018-05-30

[20:51:18] biox: Quit: probably server crash
[20:52:20] biox: has joined #ruby

2018-05-29

[02:47:05] biox: *.net *.split
[02:48:12] biox: has joined #ruby

2018-04-25

[12:25:41] biox: *.net *.split
[12:27:18] biox: has joined #ruby
[15:01:12] biox: *.net *.split
[15:03:18] biox: has joined #ruby

2018-04-13

[00:05:59] biox: has joined #ruby

2018-04-12

[13:58:47] biox: Quit: probably server crash

2018-04-05

[08:12:22] biox: Ping timeout: 264 seconds
[08:12:51] biox: has joined #ruby

2018-03-19

[21:47:55] biox: Quit: probably server crash
[21:48:26] biox: has joined #ruby

2018-03-16

[16:03:03] biox: *.net *.split
[16:03:59] biox: has joined #ruby
[22:07:19] biox: *.net *.split
[22:07:54] biox: has joined #ruby

2018-02-27

[01:05:46] biox: Quit: probably server crash
[01:06:04] biox: has joined #ruby

2017-12-21

[05:13:09] biox: Ping timeout: 255 seconds
[05:14:28] biox: has joined #ruby

2017-12-20

[20:19:46] biox: Quit: probably server crash
[20:21:07] biox: has joined #ruby

2017-12-07

[09:39:39] biox: Quit: Ping timeout (120 seconds)
[09:40:05] biox: has joined #ruby

2017-12-03

[17:35:28] biox: Quit: probably server crash
[17:35:47] biox: has joined #ruby
[17:36:32] biox: Client Quit
[17:37:06] biox: has joined #ruby
[17:38:30] biox: Client Quit
[17:43:07] biox: has joined #ruby

2017-12-01

[02:28:53] biox: has joined #ruby

2017-11-13

[18:52:09] biox: Quit: Ping timeout (120 seconds)
[18:52:33] biox: has joined #ruby

2017-11-11

[23:13:58] biox: Ping timeout: 252 seconds
[23:35:08] biox: has joined #ruby

2017-11-08

[09:43:30] biox: *.net *.split
[09:49:05] biox: has joined #ruby

2017-10-31

[10:01:40] biox: Read error: Connection reset by peer
[10:02:01] biox: has joined #ruby

2017-10-21

[20:51:33] biox: Quit: probably server crash
[21:43:23] biox: has joined #ruby

2017-10-20

[05:31:49] biox: has joined #ruby
[05:41:20] biox: latemus what're you trying to do? are you reading a file into memory and trying to manipulate it?
[05:42:24] biox: latemus this is a repo of my various attempts to do exactly that most efficiently, maybe it'll give you some ideas https://gitlab.com/biox/more-example-scripts
[05:44:23] biox: latemus is this a practice in coding / is there some reason you're not using bash for this use-case? those tools are way faster at parsing log files and the like
[05:46:15] biox: yea fair enough, i do stuff in ruby all the time that'd be better in bash because it's fun :)
[05:46:47] biox: latemus if you're parsing a log file and outputting to another file i'm wondering why you can't just parse one line at a time and then drop it to reduce memory consumptipon?
[05:50:57] biox: tbh i have no idea. but i do know that you can use a lazy enumerator so you don't have to load the whole file into memory - you can do it on the fly discarding old data
[05:51:09] biox: unless you need to sort it, in which case you'll need to do some extra hackery
[05:54:07] biox: latemus sure, hope it helps! lazy enum is what you're looking for i think
[06:03:09] biox: Quit: The Lounge - https://thelounge.github.io
[06:35:08] biox: has joined #ruby

2017-10-09

[19:51:29] biox: has left #RubyOnRails: ("WeeChat 1.5")

2017-10-01

[05:00:02] biox: Quit: brb servers are ded
[05:00:18] biox: has joined #ruby
[05:00:18] biox: has joined #RubyOnRails

2017-09-23

[20:29:52] biox: Quit: brb servers are ded
[20:30:18] biox: has joined #ruby
[20:30:18] biox: has joined #RubyOnRails

2017-09-11

[20:24:59] biox: Ping timeout: 240 seconds
[20:27:42] biox: has joined #ruby
[20:27:42] biox: has joined #RubyOnRails

2017-09-01

[05:00:01] biox: Quit: brb servers are ded
[05:02:06] biox: has joined #ruby
[05:02:06] biox: has joined #RubyOnRails

2017-08-14

[23:51:27] biox: *.net *.split
[23:55:17] biox: has joined #ruby
[23:55:18] biox: has joined #RubyOnRails
[23:57:29] biox: *.net *.split
[23:59:44] biox: has joined #ruby
[23:59:45] biox: has joined #RubyOnRails

2017-08-01

[05:00:01] biox: Quit: brb servers are ded
[05:00:20] biox: has joined #ruby
[05:00:20] biox: has joined #RubyOnRails

2017-07-27

[22:52:29] biox: Hello rubyists! I'm struggling to come up with a data transform solution currently. Detail here: https://hastebin.com/xahohoboci.rb
[22:53:07] biox: More or less I'm bad at thinking about how to associate the data to its' job. I always tend to fall back on map/each loops, and I think that's limiting me.
[22:55:38] biox: it's not, just sorta pretty output.
[22:56:11] biox: But I will look into group_by now. It sounds like I'd need to split my initial strings into arrays and group them by [0] or something?
[22:58:08] biox: Alright, I'll hack something together, group_by seems to be what I was searching for, but I had no idea how to word it

2017-07-18

[03:03:51] biox: Ping timeout: 255 seconds
[03:06:54] biox: has joined #ruby
[03:06:54] biox: has joined #RubyOnRails

2017-07-04

[22:03:20] biox: Quit: brb servers are ded
[22:05:21] biox: has joined #RubyOnRails
[22:05:22] biox: has joined #ruby
[22:16:47] biox: Quit: brb servers are ded
[22:17:33] biox: has joined #RubyOnRails
[22:17:34] biox: has joined #ruby
[22:17:43] biox: Client Quit
[22:18:41] biox: has joined #RubyOnRails
[22:18:42] biox: has joined #ruby

2017-07-01

[05:00:02] biox: Quit: brb servers are ded
[05:00:33] biox: has joined #ruby
[05:00:33] biox: has joined #RubyOnRails

2017-06-28

[02:35:00] biox: Ping timeout: 260 seconds
[02:38:09] biox: has joined #ruby
[02:38:09] biox: has joined #RubyOnRails

2017-06-27

[05:39:12] biox: Ping timeout: 260 seconds
[05:40:23] biox: has joined #ruby
[05:40:23] biox: has joined #RubyOnRails

2017-06-20

[00:17:28] biox: *.net *.split
[00:22:37] biox: has joined #RubyOnRails
[00:23:01] biox: has joined #ruby

2017-06-17

[15:35:29] biox: Ping timeout: 268 seconds
[15:44:29] biox: has joined #ruby
[15:44:29] biox: has joined #RubyOnRails

2017-06-15

[01:24:59] biox: Hi all! I have a homework assignment: Create a ruby script that takes a huge file (20GB) that contains a number on each line. Supply two arguments: filename and N. Output should be the N largest numbers as specified by the user. Here is what I've come up with so far: https://hastebin.com/basefojoji.vbs - I think it works, but it seems to be unbearably slow. Any advice or direction would be appreciated.
[01:28:03] biox: It's been hanging for awhile, I'm not sure if the excessive array sorting is causing it to process slowly or if there is a mistake somewhere in my code.
[02:42:31] biox: lostapathy: thank you. I got the code working eventually as you can see here: https://github.com/jessebolson/examplescripts
[02:42:57] biox: But the problem is that it takes 30 minutes to process a 16GB file, which seems absurd to me.
[02:44:51] biox: It looks like I may be using readlines, and switching to a different kind of loop might make processing faster. But I cannot understand why a simple sort | tail in bash is orders of magnitude faster than my method
[02:58:17] biox: darix: I think that's what I'm doing? f.each_line do |line|
[02:58:55] biox: but I think my problem is that I'm making an array, doing a size comparison against it, and dropping the last value on every single line in the file, which I assume makes it slow.
[02:59:40] biox: https://github.com/jessebolson/examplescripts/blob/master/file-reader.rb line 10 is causing my problem I think, but I can't think of a better way to chunk through a file.
[15:17:53] biox: Hello everyone. I have an assignment that requires me to: provide two inputs to a ruby file (filename and N). Assuming the file contains newline-separated integers, parse the file and print the largest N numbers. The filesize I'm operating against is very large (16GB) so I can't load it all in memory at once. To speed things up I'm trying to use the Parallel gem, but once I run the code all of the memory on my
[15:17:55] biox: machine (10+GB) is consumed. This is true even with a smaller filesize (100MB) so there must be something wrong with my code. Am I missing something? https://hastebin.com/wexipuhixi.vbs
[15:24:33] biox: ropeney: it is. In fact it's the same if all I do nothing inside of the parallel block
[15:24:55] biox: so my guess is that Parallel creates a thread from each line first, then fires off the workers or something.
[15:25:38] biox: lostapathy: I got it working without parallel operations, I'm trying to make it faster now - running into this new problem
[15:26:28] biox: ropeney: How could I test something like that? The extent of my debugging currently is running stuff in IRB/puts'ing variables, but those methods don't apply here
[15:28:01] biox: ropeney: in my case it sounds like that would be array.reference_count or line.reference_count
[15:33:37] biox: ropeney: Okay great, I will give this a test and get back to you.
[15:56:34] biox: matthewd: While I can't load the huge file into memory all at once, I figured that I could chop it up using parallel threads. The original version of my program used maybe 1GB of memory for a 16GB file. With parallelism, 16GB of memory is consumed completely by a 100MB file. To me this is indicative of a different problem.
[15:57:01] biox: (The original version of my program being single-threaded, and ran at 100% of 1 core)
[15:58:33] biox: that's right
[15:58:48] biox: it consumed all 16GB of memory on my local system trying to run against a 100MB file. :/
[15:59:27] biox: matthewd: What path would you recommend if I wanted to improve performance?
[15:59:32] biox: I mean without using GNU sort ;)
[16:03:34] biox: matthewd: Inside of the loop I take each line and toss it into an array. Since I don't have the memory available to toss everything into the array at once my thought was I would periodically sort and trim the array if it grew over a certain size. I found that increasing this size could improve performance slightly, but not greatly (degrading performance at around 10,000,000 array items). Otherwise I can't think
[16:03:36] biox: of a step I can trim out of the process.
[16:04:31] biox: Oh wait, what if I just threw out numbers lesser than array[9] instead of having to sort them
[16:08:56] biox: Huh, I overlooked the heck out of that. One sec, I'll get back to you guys - I'll put a hold on the parallel idea.
[16:21:04] biox: So I came up with this, which looks to process about 500MB in 30s: https://hastebin.com/okokezamuf.vbs
[16:21:52] biox: I don't think that I can make the comparison any more efficient than if (number is smaller than the smallest number in the array) then do nothing
[16:40:26] biox: matthewd: given that this is already quite fast, wouldn't it make sense to make the process parallel at this point? the memory would still be bounded (I think) and four-cores could result in a significant improvement over a single core if I'm thinking about this correctly.
[16:41:10] biox: I mean clearly Ruby isn't the right tool for doing this quickly, I just want to see what it can do. :)
[16:45:16] biox: matthewd: Sure, and I don't have that target - but I'll give it a shot for fun. The Parallel library is having a hayday with my memory still, so I'll have to work through that. I'll give concurrent-ruby a try as suggested by ropeney as well.
[16:45:55] biox: ropeney: `sort #{filename} | head -n #{selection}`
[16:46:37] biox: I have trouble following C code :/
[16:46:44] biox: I tried yesterday and couldn't figure it out
[16:47:52] biox: The people in #bash say that sort loads the file into memory (preserving maybe 100MB and never dipping into swap intentionally) - then it makes files in /tmp for the rest of the data, so it definitely loads everything at once, does a mergesort (i think) and then prints the output
[16:49:01] biox: I have no idea
[16:49:22] biox: Hah, good point
[16:49:58] biox: Yeah, a mergesort would probably be good - I looked at a few ruby implementations but they all use arrays, and I can't hold the whole file in an array at once sadly
[16:50:40] biox: It is not. Does Ruby automatically compress stuff before it enters memory?
[16:50:47] biox: That question is probably way too vague
[16:51:42] biox: Oh interesting
[16:51:56] biox: I'll give that a shot next and compare it to my current code
[16:53:06] biox: sure, it sounds like that'd require a lot of computation since it'd have to happen on every number
[16:54:06] biox: it does, but it compares against a var that's already set versus the mergesort method that would have to look for a matching number on each iteration, unless I'm understanding my code wrong
[16:57:01] biox: Yea, I'm also not accounting for negative numbers, really making a lot of assumptions about the input
[16:58:24] biox: I have never done those things before, so it'd probably be a good practice for me to try. I'll give it a shot and let you know how it turns out.
[16:59:04] biox: And also, thanks for all of your patience! (and you too matthewd)

2017-06-12

[22:34:44] biox: has joined #ruby
[22:37:00] biox: hi all! what's the most elegant way to include something in a string if (some logic) is true, and exclude it if that logic is false? ruby noob here just trying to make some little things more elegant
[22:37:44] biox: for example 'puts "hello my name is biox" if X = true, otherwise puts "hello my is biox" if X = false. Is the most elegant way using if-statements?
[22:38:08] biox: the include or exclude bit is two space-separated words
[22:39:44] biox: like "something #{blah} something" and the logic would set blah equal to nil or something else?
[22:41:01] biox: ACTION googles the word ternary
[22:42:42] biox: Jayson_Virissimo: I like that method, I'll play around with it, thanks!

2017-06-01

[05:00:01] biox: Quit: brb servers are ded
[05:00:21] biox: has joined #RubyOnRails

2017-05-08

[02:07:19] biox: has joined #RubyOnRails
[02:15:11] biox: hi everyone! rails noobie here going through a rails book, and my controller test keeps failing. hopefully all relevant informations: https://hastebin.com/anoqetoxol.rb - let me know if there's anything else I can throw in
[02:15:34] biox: it has to do with this line: assert response.body.include?(stories(:one).name)
[02:15:47] biox: I'm wondering if the format is wrong
[03:10:03] biox: creature: in test/fixtures/stories.yml i define names for the stories. my impression is that those fixtures will load in parallel and be use-able as well?
[03:11:20] biox: https://hastebin.com/wubadolico.pas
[03:12:12] biox: pretty much yea.
[03:12:32] biox: to answer directly, yes.
[03:13:17] biox: creature: oh.
[03:14:28] biox: creature: you are 100% correct, I overlooked where I was placing these tests, ack!
[03:14:38] biox: Will correct the placement and report back, thanks for the direction! <3
[03:16:12] biox: hopefully! i got it in my head that because I did all of my work on the VoteController I only needed to write tests for it. :) I forgot that I did some mapping between the votes/story controllers
[03:31:00] biox: creature: you were correct, the problem hasn't resolved by moving the test. Where does the 'response' object come from? as in `assert response.body.include?(stories(:one).name)`
[03:31:14] biox: also is there any log file or troubleshooting technique i should be using?
[03:34:06] biox: creature: oh perfect, that's exactly what i was looking for.
[03:37:23] biox: creature: wow, what a useful tool!
[03:37:30] biox: i think I have it figured out
[03:37:32] biox: https://hastebin.com/ezaqiyomoh.tex
[03:38:07] biox: the 'el blog' string isn't in the body verbatim, since i'm replacing whitespaces with hyphens when i generate uris
[03:40:55] biox: creature:in the HTML it appears like this: action=\"/stories/980190962-el-blog/votes\"
[03:41:21] biox: So I changed my assert to assert response.body.include?(stories(:one).name.gsub(/ /, '-')) which made the test pass
[03:42:41] biox: feels a little flakey though
[03:46:45] biox: creature: this is a good point. :)
[03:48:56] biox: yaay, it works!
[04:36:30] biox: creature: kind of a complicated question, but I figured I'd ask it here. I'm considering diving deep into Rails and aiming to become a ror developer because I like the language and the framework a lot. Do you think that it's a good idea considering that microservices are the new hotness?
[04:39:53] biox: my background is mainly in the operations world - linux/chef/jenkins and the like, but no pure programming. i initially started playing with ror because i picked up a book, but i find myself reading and playing with it in my free time
[04:42:46] biox: isn't it kind of monolithic by design though? with the tight adherance to mvc
[04:46:51] biox: good point, it reminds me of chef in a lot of ways in that you can do a lot of things that it's not necessarily designed for but still winds up being pretty good at - that way all teams can speak the same language anyway
[04:53:31] biox: how long have you been using rails creature?
[04:56:01] biox: dopie: hai!

2017-03-15

[16:04:52] biox: has left #ruby: ("WeeChat 1.5")

2016-07-27

[01:20:23] biox: has left #ruby: ("WeeChat 1.4")

2016-07-23

[00:48:45] biox: Quit: brb servers are ded
[00:50:24] biox: has joined #ruby

2016-07-10

[05:00:00] biox: *.net *.split
[08:54:03] biox: has joined #ruby

2016-06-20

[12:19:40] biox: Ping timeout: 264 seconds
[12:29:03] biox: has joined #ruby

2016-06-16

[19:23:08] biox: has left #ruby: ("Leaving")
[22:23:10] biox: has joined #ruby

2016-06-10

[10:53:45] biox: Ping timeout: 260 seconds
[10:56:00] biox: has joined #ruby

2016-05-31

[14:48:01] biox: Quit: brb servers are ded
[14:49:12] biox: has joined #ruby

2016-05-30

[08:12:13] biox: has joined #ruby