On Friday, September 2, 2016 at 2:31:29 PM UTC-5, Jeremy Evans wrote:
>
> I'm guessing streaming would preform better, but you would still need to
> use a separate thread or sharding when using streaming.
>
That's great. I'll give streaming a try, but I believe the bottleneck isn't
going to be retrieving the rows, but with the work I'm doing before
updating them.
> I have to say, if you are really just updating updated_at and are
> concerned about performance, I would not use your approach unless you
> absolutely had to have a model hook executed. The only time you really
> need to do query per row is if you need to execute ruby code on a per row
> basis to determine what the new column values should be.
>
This is fantastic advice, and I'll bookmark this for future use. I'm sorry
I didn't make it clear, but I meant that updated_at example to be a
simplified example of what I'll be doing. I'll be calculating values on a
per record basis and then issuing an update_all call per record which will
include the updated_at to remove it from the set of records to be processed.
Person.where(updated_at: nil).paged_each { |p| Thread.new {
p.update_all(some_calculated_value: CalculateValue(p), updated_at:
Time.now.utc) }.join }
Apologies, I should have made the example a bit more clear. In any case,
your suggestion will be useful to future Googlers, I'm sure.
Thank you again for all of your help. I'm off to the races using paged_each
and threads. I'll report back if I run into anything of interest relevant
to this thread, and I'll try sequel_pg and streaming if time permits in the
next few weeks. It sounds like we'd get a number of performance benefits
beyond streaming vs cursors, so it's worth looking into for sure.
- Trevor
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sequel-talk+unsubscr...@googlegroups.com.
To post to this group, send email to sequel-talk@googlegroups.com.
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
