Great advice (again), Fred. I had found some other tips that didn't
suit my situation, but this is great! Thanks.
@Lionel: Yes, I was using threads for every host, so I should return
these connections to the pool, because some BRB processes can overlap
each other and have 5 or more active connecti
On 20 Nov 2009, at 12:32, Lionel Bouton wrote:
>
> BTW, if you use threads, you should join each thread (or try to) to have
> at least some control over the number of threads running concurrently.
>
> If not, each time you loop over such code you create new threads (which
> probably will use ne
Lionel Bouton a écrit, le 11/20/2009 01:27 PM :
> jhaagmans a écrit, le 11/20/2009 12:41 PM :
>
>> I haven't had a chance to test Conrad's suggestion, but I've also seen
>> that I reach the connection limit for ActiveRecord. Is there a way to
>> do, for example:
>>
>> hosts.each do |host|
>> #
jhaagmans a écrit, le 11/20/2009 12:41 PM :
> I haven't had a chance to test Conrad's suggestion, but I've also seen
> that I reach the connection limit for ActiveRecord. Is there a way to
> do, for example:
>
> hosts.each do |host|
> #Do all kinds of stuff to host
> end
> hosts.save
>
> So I don
On Fri, Nov 20, 2009 at 3:41 AM, jhaagmans wrote:
> I haven't had a chance to test Conrad's suggestion, but I've also seen
> that I reach the connection limit for ActiveRecord. Is there a way to
> do, for example:
>
> hosts.each do |host|
> #Do all kinds of stuff to host
> end
> hosts.save
>
> S
I haven't had a chance to test Conrad's suggestion, but I've also seen
that I reach the connection limit for ActiveRecord. Is there a way to
do, for example:
hosts.each do |host|
#Do all kinds of stuff to host
end
hosts.save
So I don't have alot of concurrent connections to my database?
--
Yo
Thank you both.
Conrad, do you want me to execute this from within BackgroundRB? I'll
be back tonight and test it.
--
You received this message because you are subscribed to the Google Groups "Ruby
on Rails: Talk" group.
To post to this group, send email to rubyonrails-t...@googlegroups.com.
To
On Thu, Nov 19, 2009 at 6:34 AM, jhaagmans wrote:
> Hi Fred,
>
> On 19 nov, 14:59, Frederick Cheung wrote:
> > On Nov 19, 1:22 pm, jhaagmans wrote:> Hi Fred,
> >
> > > Thanks for your response. I hope you can answer three questions about
> > > this.
> >
> > > - Why is it that pharrington's exam
jhaagmans a écrit, le 11/19/2009 03:34 PM :
> And yes, I'm still using BRB, I really can't think of a way to avoid
> using BRB. I need to query a few thousand hosts every hour, every day.
>
>
This translates to a few hosts each second. If I had this kind of load I
wouldn't use a background job
Hi Fred,
On 19 nov, 14:59, Frederick Cheung wrote:
> On Nov 19, 1:22 pm, jhaagmans wrote:> Hi Fred,
>
> > Thanks for your response. I hope you can answer three questions about
> > this.
>
> > - Why is it that pharrington's example did work, even though the hosts
> > he used (I copied them) were
On Thu, Nov 19, 2009 at 5:22 AM, jhaagmans wrote:
> Hi Fred,
>
> Thanks for your response. I hope you can answer three questions about
> this.
>
> - Why is it that pharrington's example did work, even though the hosts
> he used (I copied them) were never resolved before on my server? I
> should h
On Nov 19, 1:22 pm, jhaagmans wrote:
> Hi Fred,
>
> Thanks for your response. I hope you can answer three questions about
> this.
>
> - Why is it that pharrington's example did work, even though the hosts
> he used (I copied them) were never resolved before on my server? I
> should have had the
Hi Fred,
Thanks for your response. I hope you can answer three questions about
this.
- Why is it that pharrington's example did work, even though the hosts
he used (I copied them) were never resolved before on my server? I
should have had the same problem, right?
- I've used net-dns before becau
On Nov 19, 11:00 am, jhaagmans wrote:
> I'm a little further on this. I've started logging the process instead
> of writing to ActiveRecord. This is my code:
> As you can see, it does do -some- threading, but it finishes requests
> only once every 10 seconds or so. What am I doing wrong? pharri
I'm a little further on this. I've started logging the process instead
of writing to ActiveRecord. This is my code:
def schedule_queries
t = Time.now
hosts = get_hosts(30)
logger.info "Starting request for #{hosts.count} domains at #{t}"
domains.each do |domain|
Thread.new do
beg
On Wed, Nov 18, 2009 at 12:20 PM, pharrington wrote:
> On Nov 18, 1:43 pm, Conrad Taylor wrote:
> > Each thread must acquire the lock before it can execute. Thus, it
> operates
> > similar to a queue data structure (i.e. first in first out (FIFO)) and
> this
> > is how
> > it work today in rega
Well, BackgroundRB does have a problem because workers can't overlap,
so a worker is put in the queue when it's started, which is a bit of a
pain because some HTTP-requests take longer than others. But look at
this:
def schedule_queries
i =
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
>
> > The reason why we didn't choose jRuby was because it uses too much
> > memory to be able to run this on a VPS. Is there any documentation
> > available on using jRuby on a low-memory (<256MB) system? I've looked
> > for it, but couldn't find it. Maybe there's an alternative workaround
> > for
On Nov 18, 1:43 pm, Conrad Taylor wrote:
> Each thread must acquire the lock before it can execute. Thus, it operates
> similar to a queue data structure (i.e. first in first out (FIFO)) and this
> is how
> it work today in regards to Ruby 1.8.6, 1.8.7, and 1.9.1. I know the C
> implementation
>
Wow, thanks for all your help, greatly appreciated.
> The Global Interpreter Lock (GIL) prevents threads from executing in
> parallel when using Ruby 1.8.6 aka MRI, 1.8.7, and 1.9.1 aka YARV.
> However, JRuby 1.3.x/1.4.x, MacRuby 0.5 Beta 2, Maglev and several
> other upcoming Ruby VMs are not con
On Wed, Nov 18, 2009 at 10:23 AM, pharrington wrote:
> On Nov 18, 12:32 pm, Conrad Taylor wrote:
> > On Tue, Nov 17, 2009 at 1:07 PM, jhaagmans
> wrote:
> >
> > > Hi,
> >
> > > I have a backgroundrb worker that gets triggered every second. When
> > > it's triggered, it's supposed to make 2 - 15
pharrington a écrit, le 11/18/2009 07:23 PM :
> Emm just because the threads aren't all executing *simultaneously*
> doesn't that they aren't running in parallel (due to all the thread
> switching etc).
>
> Regardless, I can't seem to reproduce the OPs behviour:
>
>
I'm too lazy to check the de
On Nov 17, 4:07 pm, jhaagmans wrote:
> Hi,
>
> I have a backgroundrb worker that gets triggered every second. When
> it's triggered, it's supposed to make 2 - 15 http-requests using
> Net::HTTP. My idea was to put every execution into a thread so the
> next execution doesn't have to wait for the l
On Nov 18, 12:32 pm, Conrad Taylor wrote:
> On Tue, Nov 17, 2009 at 1:07 PM, jhaagmans wrote:
>
> > Hi,
>
> > I have a backgroundrb worker that gets triggered every second. When
> > it's triggered, it's supposed to make 2 - 15 http-requests using
> > Net::HTTP. My idea was to put every execution
Update: I've tried doing it using EventMachine, which won't work
either:
EM.run do
make_request(host)
end
def make_request
begin
client = EventMachine::HttpRequest.new(host).get
host.set_active
rescue
host.set_inactive
ensure
EM.stop
end
end
Now it's only executing the
25 matches
Mail list logo
