Page 1 of 2 | Next »
[00:27:19] regedit: so i put it in the app/assets folder, and it will be copied over to the public/assets folder?
[00:27:41] regedit: also i need this .js file to be served independently, not combined with other .js etc.
[01:29:59] regedit: i see. i think this is possibly an exceptional case (greasemonky/tampermonkey script public download & update URL) and in the case of this heroku app it is committed to the git repo and thus it will always be part of the deployed web files so hopefully it shouldnt disappear for whatever reasons
[15:49:52] regedit: i have a HUGE string of text i'm trying to process/work with, it's giving me some problems because well, it's huge
[15:51:56] regedit: it's basically like 9M of phone numbers i'm trying to compare with other existing phone numbers in my database
[15:53:00] regedit: but first i fix up the huge string a bit... split it, remove some characters from it (area codes & whitespace etc), all in all it looks like the runtime piles up a lot of memory very quickly. even trying to call GC.start a few times but still memory complaints
[15:54:08] regedit: if it helps clarify the requirements here: this is a Do Not Call list from the federal gov
[15:55:04] regedit: ye eventually i try making a Hash of phone-number => nil, so that i can use it as a lookup
[15:56:04] regedit: i was wondering if i need to somehow make some sort of clever compact lookup structure out of it (btree or something?) so it uses as little ram as possible
[15:56:58] regedit: but it seems all the gsub and split etc steps before might also be "conflagrating" the ram copies of the data...
[16:01:02] regedit: some line numbers of possible interest: :20 fetch_file.file.read might load the entire zip compressed file into memory
[16:01:46] regedit: :30 dnc_raw = entry.get_input_stream.read copies the entire extracted string into ram?
[16:02:33] regedit: :33 trying to regex through the entire string to make sure it conforms to expected format - copies it again?
[16:07:49] regedit: im actually finding some instances of these extracted data files to contain 16M+ of data
[16:08:37] regedit: so how can i sanely process this much data within limited resources, effectively cleaning up after everything so RAM doesn't keep leaking?
[16:12:39] regedit: ineb: ah ye i was having problems giving Zip::File.open() something IO-ish, nothing seemed to work except open_buffer(fetch_file.file.read) which i think might indeed be the full string...
[16:20:12] regedit: TypeError: no implicit conversion of CarrierWave::Storage::Fog::File into String
[16:23:10] regedit: RuntimeError: Zip::File.open_buffer expects a String or IO-like argument (responds to tell, seek, read, close). Found: CarrierWave::Storage::Fog::File
[16:29:20] regedit: in this case it's the path where it's located within the storage back-end, in this case AWS S3 via fog-aws...
[16:29:45] regedit: ineb: won't StringIO.new(fetch_file.read) also cause another round of copying the string into memory?..
[16:34:46] regedit: ineb: ok so this pertains to reading & extracting the zip file, any idea if there's more to solve in this huge string processing code?..
[16:36:12] regedit: let's say we didnt even have to extract a zip file, but rather the data fetched from S3 was directly the uncompressed 16M file
[16:43:37] regedit: ineb: it causes memory complaints which in some cases triggers a killing of the process. It seems to happen "over time" after executing this process several times (not the first times it executes. memory leak-ish...)
[19:17:40] regedit: how are ruby Hash lookups implemented? is it anything special to make look ups very fast?
[19:18:29] regedit: i'm trying to find a cood collection/container to use in order to lookup simply whether (boolean true/false) a value exists in the collection or not, with super fastest lookup time possible
[19:53:30] regedit: i've been doing Hash with key => nil, because all i care for are the existence of the keys, dont care for the values
[19:53:51] regedit: so i was wondering if there's an even more optimized way of only using the keys part of it without values
[19:57:45] regedit: also, since i'm adding a huge amount of elements, which if i understand correctly the underlying mechanisms of Hash how it increases in number of underlying "bins", i wish there was a way for me to specify number of bins to begin with, so it doesnt have to keep re-dividing and re-creating Hash bins...
[19:59:00] regedit: "At some point you pay an exponentially increased time penalty while Ruby resizes the bin pool"
[19:03:17] regedit: anyone use carrierwave + aws/s3? why is calling uploader.remove! not actually deleting the file from s3?
[19:03:35] regedit: anyone use carrierwave + aws/s3? why is calling uploader.remove! not actually deleting the file from s3?
[18:31:43] regedit: (how) can i reference the request parameters within routes.rb? ex. if params['foo'] == 'bar'; match.... end
[18:32:15] regedit: NameError: undefined local variable or method `params' for #<ActionDispatch::Routing::Mapper:0x00000006112780>
[01:03:36] regedit: anyone? how can i do this the Sidekiq way https://github.com/httprb/http/wiki/Thread-Safety
[01:04:07] regedit: trying to persist connection (keep-alive) across many executions of a sidekiq worker
[01:05:03] regedit: ex. imagine queueing up a bunch of Sidekiq tasks each of which performs an API call to the same host
[04:57:36] regedit: i once came asking for a recommendation for an HTTP library, and was advised to use Typhoeus
[04:58:11] regedit: problem is - i now realize i want to make these HTTP requests from a Sidekiq worker....
[05:00:56] regedit: the sidekiq worker performs its task on 1 record at a time, rather than having all records to process and queue up for typhoeus http requests... so it seems i might be losing out on running many Typhoeus+hydra requests in parallel
[05:02:58] regedit: is there another way of queuing up many many requests for typhoeus in parallel from within workers that operate on only 1 record at a time?
[05:19:24] regedit: havenwood: so persistent http connection is not an applicable notion for me if my records are being processed 1 at a time in sidekiq workers?
[13:04:40] regedit: havenwood: still there? do i need to do anything to expressly use such a connection pool? or it happens behind the scenes
[13:06:16] regedit: ah nvm i think my answer should be here https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling
[13:08:24] regedit: havenwood: is your recommendation to use connection_pool with HTTP.rb or Typhoeus?
[13:16:22] regedit: Silex: any idea how do use HTTP.rb in conjunction with (sidekiq with) connection_pool ?
[13:22:24] regedit: the last mystery to solve would be how to do this https://github.com/httprb/http/wiki/Thread-Safety in a sidekiq worker so that the http connection is persisted across hundreds of distinct sidekiq worker executions...
[13:22:54] regedit: and then somehow close the connection pool at the right time after the batch of said sidekiq jobs have completed
[22:59:04] regedit: is anyone successfully using Sidekiq + connection_pool + HTTP.rb to make a bunch of keep-alive requests across many distinct executions of a Sidekiq Worker?
[18:04:08] regedit: where do i put a misc/util method that i need to call from various different places in the codebase?
[18:04:48] regedit: in this case: several different Sidekiq workers need to clean up & fix some CSV files before being able to do with them whatever each worker processes
[18:12:53] regedit: at the beginning of these workers they need to open spreadsheet files of course. Using the 'roo' gem, sometimes it's a xls file, xlsx, csv
[18:13:43] regedit: CSVs are a pain because they often come malformed, including various bad characters, and other problems needing cleaning - otherwise they dont open with roo...
[18:15:43] regedit: right now i have some 10 lines of CSV cleaning code that is repeated identically in 3 distinct worker classes...
[18:17:11] regedit: since they're not really serving the MVC side of the rails app, i guess i should put such generic/utility methods under lib/ ?
[04:07:51] regedit: How can i query an Author's related Posts where the Posts' related Comments exist where some criteria applies?
[04:17:21] regedit: kind of like in SQL: select * from authors a join posts p on a.id = p.author_id where exists (select * from comments where post_id = p.id and not trashed)
[04:17:46] regedit: kind of like in SQL: select * from authors a join posts p on a.id = p.author_id where exists (select * from comments where post_id = p.id and not trashed)
[22:04:36] regedit: I have a Messages model and ContactNumber model. I'd like to add ContactNumber.opted_out_message_id column which should refer to a single Messages record
[22:04:48] regedit: The idea being i need to record whether a ContactNumber is considered "opted out", and if so which Message record is the one that "opted out" the ContactNumber
[22:05:55] regedit: i'm fumbling around not sure how to achieve this, messing with belongs_to / has_one, add_reference with custom column name, no foreign keys etc...
[17:01:27] regedit: if i'm using a gem like Unirest and i know i'm gonna be placing hundreds of HTTP requests to a given 3rd party, do i have to do anything to care about persistent connection / keep alive?
[17:10:54] regedit: i'm gonna be interacting with an API, processing hundreds of records, each one its own API call
[20:23:09] regedit: i have 2 Sidekiq::Worker classes, i'd like both of them to perform some interactions with a 3rd party API, rather new to Ruby & RoR, how should i structure this? Create some sort of DoTheApiThing class which both workers will use?
[22:32:51] regedit: app/workers? isn't that where actual worker classes should go, the kinds that are supposed to be perfom()ed as scheduled jobs?
[16:51:10] regedit: but this is in middle of a regex, you'd think that would be picked up by the compiler..