Re: DJ Workers Crashing Without Specific Information
On Sep 30, 2010, at 3:58 PM, WilliamF wrote: > RPM Monitoring DJ worker host:railgun64.29642 pid:29790 > *** Starting job worker host:railgun64.29642 pid:29790 > /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation > fault > ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, > Ruby Enterprise Edition 2010.02 Oof, bummer, William -- I imagine you'd see the same error in the console -- perhaps you could confirm? $ heroku console 'Delayed::Worker.new.work_off(1)' And I have hardly any experience tracking down these segfaults, but if you feel up for a wild goose shave, I'm wondering if it might have something to do with data passed to URI.split (which gets compared with a regex on line 385 of uri/common.rb). Cheers, -- Matthew smime.p7s Description: S/MIME cryptographic signature
Re: DJ Workers Crashing Without Specific Information
ActiveRecord batches won't work for you? http://rdoc.info/docs/rails/2.3.8/ActiveRecord/Batches/ClassMethods Both methods default to 1000 results per batch, but can be configured with the :batch_size option. Mike On Mon, Oct 4, 2010 at 10:56 AM, William Flanagan wrote: > Nope, we love empty queues. > I think it comes down to memory, that we aren't cursoring the output from > the database. We are rewriting our code to implement our own cursor support, > since AR doesn't support it. > The hard part is that we get no information from the process when it > crashes. We were finally smart enough to write custom code that tells us > which job rows are locked, so we can start the troubleshooting process that > way, since the lock is not cleared. > > But, the bigger issue is this: I have had this ticket open 5 days and this > is the first bit of communication we have had. We are writing our methods to > support heroku specifically at this point, and for it to take 5 days to get > any response seems like too long. > I have opened 2 tickets in my history with heroku, both took almost 1 week > to get a response. What does someone do if their app is in production and > completely down? Wait a week? Is this typical? If so, I need to know that. > Any thoughts anyone? > > Sent from my iPad > On Oct 4, 2010, at 13:28, Brandon Casci wrote: > > William...this may sound trivial, but to your DJ workers crash if the work > queues is empty? > > On Mon, Oct 4, 2010 at 1:08 PM, Dan Croak wrote: >> >> There are some troubleshooting suggestions on this page: >> >> http://docs.heroku.com/ps >> >> On Mon, Oct 4, 2010 at 1:04 PM, Brandon Casci >> wrote: >> > I can only offer commiseration because I'm in the same position as you >> > ;) >> > >> > Please let me know what they say. >> > >> > What do your jobs do? >> > >> > On Thu, Sep 30, 2010 at 8:58 AM, WilliamF >> > wrote: >> >> >> >> Hi all, >> >> >> >> I'm looking for suggestions, as I'm not getting a response from >> >> Heroku. >> >> >> >> My background DJ workers are crashing. I can't seem to make them stay >> >> up for more than 15 or so seconds. If I look at the logs, the error >> >> message I'm getting is: >> >> >> >> >> >> RPM Monitoring DJ worker host:railgun64.29642 pid:29790 >> >> *** Starting job worker host:railgun64.29642 pid:29790 >> >> /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation >> >> fault >> >> ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, >> >> Ruby Enterprise Edition 2010.02 >> >> >> >> NewRelic doesn't show the DJ worker in its logs that I can >> >> troubleshoot. NewRelic states: RPM recorded no DelayedJob background >> >> task executions in the last 30 minutes. >> >> >> >> Which, needless to say, isn't the most helpful. >> >> >> >> I've taken every job that I can find that is outstanding and locked in >> >> the database, duplicated the data down to my local machine, and run my >> >> machine in production mode. The workers are able to process these just >> >> fine on my local machine. >> >> >> >> Given this, and the lack of response by Heroku, I'm stuck. I'd love >> >> some suggestions as to the best/next way to go to troubleshoot this >> >> problem. Background tasks are CORE to my application. If the >> >> background tasks don't work, my app is dead in the water. >> >> >> >> Help! >> >> >> >> William >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "Heroku" group. >> >> To post to this group, send email to her...@googlegroups.com. >> >> To unsubscribe from this group, send email to >> >> heroku+unsubscr...@googlegroups.com. >> >> For more options, visit this group at >> >> http://groups.google.com/group/heroku?hl=en. >> >> >> > >> > >> > >> > -- >> > = >> > Brandon Casci >> > Loudcaster >> > http://loudcaster.com >> > = >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Heroku" group. >> > To post to this group, send email to her...@googlegroups.com. >> > To unsubscribe from this group, send email to >> > heroku+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/heroku?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Heroku" group. >> To post to this group, send email to her...@googlegroups.com. >> To unsubscribe from this group, send email to >> heroku+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en. >> > > > > -- > = > Brandon Casci > Loudcaster > http://loudcaster.com > = > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this
Re: DJ Workers Crashing Without Specific Information
Nope, we love empty queues. I think it comes down to memory, that we aren't cursoring the output from the database. We are rewriting our code to implement our own cursor support, since AR doesn't support it. The hard part is that we get no information from the process when it crashes. We were finally smart enough to write custom code that tells us which job rows are locked, so we can start the troubleshooting process that way, since the lock is not cleared. But, the bigger issue is this: I have had this ticket open 5 days and this is the first bit of communication we have had. We are writing our methods to support heroku specifically at this point, and for it to take 5 days to get any response seems like too long. I have opened 2 tickets in my history with heroku, both took almost 1 week to get a response. What does someone do if their app is in production and completely down? Wait a week? Is this typical? If so, I need to know that. Any thoughts anyone? Sent from my iPad On Oct 4, 2010, at 13:28, Brandon Casci wrote: > William...this may sound trivial, but to your DJ workers crash if the work > queues is empty? > > On Mon, Oct 4, 2010 at 1:08 PM, Dan Croak wrote: > There are some troubleshooting suggestions on this page: > > http://docs.heroku.com/ps > > On Mon, Oct 4, 2010 at 1:04 PM, Brandon Casci wrote: > > I can only offer commiseration because I'm in the same position as you ;) > > > > Please let me know what they say. > > > > What do your jobs do? > > > > On Thu, Sep 30, 2010 at 8:58 AM, WilliamF wrote: > >> > >> Hi all, > >> > >> I'm looking for suggestions, as I'm not getting a response from > >> Heroku. > >> > >> My background DJ workers are crashing. I can't seem to make them stay > >> up for more than 15 or so seconds. If I look at the logs, the error > >> message I'm getting is: > >> > >> > >> RPM Monitoring DJ worker host:railgun64.29642 pid:29790 > >> *** Starting job worker host:railgun64.29642 pid:29790 > >> /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation > >> fault > >> ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, > >> Ruby Enterprise Edition 2010.02 > >> > >> NewRelic doesn't show the DJ worker in its logs that I can > >> troubleshoot. NewRelic states: RPM recorded no DelayedJob background > >> task executions in the last 30 minutes. > >> > >> Which, needless to say, isn't the most helpful. > >> > >> I've taken every job that I can find that is outstanding and locked in > >> the database, duplicated the data down to my local machine, and run my > >> machine in production mode. The workers are able to process these just > >> fine on my local machine. > >> > >> Given this, and the lack of response by Heroku, I'm stuck. I'd love > >> some suggestions as to the best/next way to go to troubleshoot this > >> problem. Background tasks are CORE to my application. If the > >> background tasks don't work, my app is dead in the water. > >> > >> Help! > >> > >> William > >> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Heroku" group. > >> To post to this group, send email to her...@googlegroups.com. > >> To unsubscribe from this group, send email to > >> heroku+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/heroku?hl=en. > >> > > > > > > > > -- > > = > > Brandon Casci > > Loudcaster > > http://loudcaster.com > > = > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Heroku" group. > > To post to this group, send email to her...@googlegroups.com. > > To unsubscribe from this group, send email to > > heroku+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > > > > -- > = > Brandon Casci > Loudcaster > http://loudcaster.com > = > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For mo
Re: DJ Workers Crashing Without Specific Information
William...this may sound trivial, but to your DJ workers crash if the work queues is empty? On Mon, Oct 4, 2010 at 1:08 PM, Dan Croak wrote: > There are some troubleshooting suggestions on this page: > > http://docs.heroku.com/ps > > On Mon, Oct 4, 2010 at 1:04 PM, Brandon Casci > wrote: > > I can only offer commiseration because I'm in the same position as you ;) > > > > Please let me know what they say. > > > > What do your jobs do? > > > > On Thu, Sep 30, 2010 at 8:58 AM, WilliamF > wrote: > >> > >> Hi all, > >> > >> I'm looking for suggestions, as I'm not getting a response from > >> Heroku. > >> > >> My background DJ workers are crashing. I can't seem to make them stay > >> up for more than 15 or so seconds. If I look at the logs, the error > >> message I'm getting is: > >> > >> > >> RPM Monitoring DJ worker host:railgun64.29642 pid:29790 > >> *** Starting job worker host:railgun64.29642 pid:29790 > >> /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation > >> fault > >> ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, > >> Ruby Enterprise Edition 2010.02 > >> > >> NewRelic doesn't show the DJ worker in its logs that I can > >> troubleshoot. NewRelic states: RPM recorded no DelayedJob background > >> task executions in the last 30 minutes. > >> > >> Which, needless to say, isn't the most helpful. > >> > >> I've taken every job that I can find that is outstanding and locked in > >> the database, duplicated the data down to my local machine, and run my > >> machine in production mode. The workers are able to process these just > >> fine on my local machine. > >> > >> Given this, and the lack of response by Heroku, I'm stuck. I'd love > >> some suggestions as to the best/next way to go to troubleshoot this > >> problem. Background tasks are CORE to my application. If the > >> background tasks don't work, my app is dead in the water. > >> > >> Help! > >> > >> William > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Heroku" group. > >> To post to this group, send email to her...@googlegroups.com. > >> To unsubscribe from this group, send email to > >> heroku+unsubscr...@googlegroups.com > . > >> For more options, visit this group at > >> http://groups.google.com/group/heroku?hl=en. > >> > > > > > > > > -- > > = > > Brandon Casci > > Loudcaster > > http://loudcaster.com > > = > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Heroku" group. > > To post to this group, send email to her...@googlegroups.com. > > To unsubscribe from this group, send email to > > heroku+unsubscr...@googlegroups.com > . > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- = Brandon Casci Loudcaster http://loudcaster.com = -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: DJ Workers Crashing Without Specific Information
There are some troubleshooting suggestions on this page: http://docs.heroku.com/ps On Mon, Oct 4, 2010 at 1:04 PM, Brandon Casci wrote: > I can only offer commiseration because I'm in the same position as you ;) > > Please let me know what they say. > > What do your jobs do? > > On Thu, Sep 30, 2010 at 8:58 AM, WilliamF wrote: >> >> Hi all, >> >> I'm looking for suggestions, as I'm not getting a response from >> Heroku. >> >> My background DJ workers are crashing. I can't seem to make them stay >> up for more than 15 or so seconds. If I look at the logs, the error >> message I'm getting is: >> >> >> RPM Monitoring DJ worker host:railgun64.29642 pid:29790 >> *** Starting job worker host:railgun64.29642 pid:29790 >> /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation >> fault >> ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, >> Ruby Enterprise Edition 2010.02 >> >> NewRelic doesn't show the DJ worker in its logs that I can >> troubleshoot. NewRelic states: RPM recorded no DelayedJob background >> task executions in the last 30 minutes. >> >> Which, needless to say, isn't the most helpful. >> >> I've taken every job that I can find that is outstanding and locked in >> the database, duplicated the data down to my local machine, and run my >> machine in production mode. The workers are able to process these just >> fine on my local machine. >> >> Given this, and the lack of response by Heroku, I'm stuck. I'd love >> some suggestions as to the best/next way to go to troubleshoot this >> problem. Background tasks are CORE to my application. If the >> background tasks don't work, my app is dead in the water. >> >> Help! >> >> William >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Heroku" group. >> To post to this group, send email to her...@googlegroups.com. >> To unsubscribe from this group, send email to >> heroku+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en. >> > > > > -- > = > Brandon Casci > Loudcaster > http://loudcaster.com > = > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: DJ Workers Crashing Without Specific Information
I can only offer commiseration because I'm in the same position as you ;) Please let me know what they say. What do your jobs do? On Thu, Sep 30, 2010 at 8:58 AM, WilliamF wrote: > Hi all, > > I'm looking for suggestions, as I'm not getting a response from > Heroku. > > My background DJ workers are crashing. I can't seem to make them stay > up for more than 15 or so seconds. If I look at the logs, the error > message I'm getting is: > > > RPM Monitoring DJ worker host:railgun64.29642 pid:29790 > *** Starting job worker host:railgun64.29642 pid:29790 > /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation > fault > ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, > Ruby Enterprise Edition 2010.02 > > NewRelic doesn't show the DJ worker in its logs that I can > troubleshoot. NewRelic states: RPM recorded no DelayedJob background > task executions in the last 30 minutes. > > Which, needless to say, isn't the most helpful. > > I've taken every job that I can find that is outstanding and locked in > the database, duplicated the data down to my local machine, and run my > machine in production mode. The workers are able to process these just > fine on my local machine. > > Given this, and the lack of response by Heroku, I'm stuck. I'd love > some suggestions as to the best/next way to go to troubleshoot this > problem. Background tasks are CORE to my application. If the > background tasks don't work, my app is dead in the water. > > Help! > > William > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- = Brandon Casci Loudcaster http://loudcaster.com = -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
DJ Workers Crashing Without Specific Information
Hi all, I'm looking for suggestions, as I'm not getting a response from Heroku. My background DJ workers are crashing. I can't seem to make them stay up for more than 15 or so seconds. If I look at the logs, the error message I'm getting is: RPM Monitoring DJ worker host:railgun64.29642 pid:29790 *** Starting job worker host:railgun64.29642 pid:29790 /usr/ruby1.8.7/lib/ruby/1.8/uri/common.rb:385: [BUG] Segmentation fault ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2010.02 NewRelic doesn't show the DJ worker in its logs that I can troubleshoot. NewRelic states: RPM recorded no DelayedJob background task executions in the last 30 minutes. Which, needless to say, isn't the most helpful. I've taken every job that I can find that is outstanding and locked in the database, duplicated the data down to my local machine, and run my machine in production mode. The workers are able to process these just fine on my local machine. Given this, and the lack of response by Heroku, I'm stuck. I'd love some suggestions as to the best/next way to go to troubleshoot this problem. Background tasks are CORE to my application. If the background tasks don't work, my app is dead in the water. Help! William -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers crashing....
Try heroku workers 0; heroku workers 1 to force a worker restart. btw I upgraded to the 1.9 stack and haven't had a worker crash since. Alex -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers crashing....
Thats what I'm saying. They aren't restarting at all. I guess I need to file a ticket? On Thu, 2010-08-12 at 18:46 +0200, Abel Tamayo wrote: > It's done automatically. You don't have to select or configure > anything. > > On Thu, Aug 12, 2010 at 6:33 PM, Chris Kalaboukis > wrote: > Hi Abel: > > For some reason, its not restarting the job. Is there a way to > change > that setting - its not re-starting the job its just hanging > there. i > need it to restart when it fails, can you tell me what I need > to edit in > order to make it do that? > > Thanks...Chris > > > On Thu, 2010-08-12 at 18:15 +0200, Abel Tamayo wrote: > > Heroku is supposed to restart a worker everytime it crashes. > Whatever > > is in your jobs:work task will be executed as soon as it > fails. > > > > > > However, I run a very dependent on background jobs webapp in > Heroku > > and this is not enough for me. What I did was create a rake > task that > > finds the last record to be updated and issue a background > job to > > update the next after that one and automate the task to find > the rest > > of the records that need to be updated. > > > > > > The only thing you'll be able to do to interact with your > app once > > it's deployed are Rake tasks. > > > > > > On Thu, Aug 12, 2010 at 5:54 PM, Chris Kalaboukis > > wrote: > > Possibly. Problem is I can't seem to catch it...do > you know > > how I can, > > other than running some monitor to restart. > > > > BTW, how do I restart a crashed job? Is there a way > to do it > > from the > > heroku command line? > > > > Thanks...Chris > > > > > > On Thu, 2010-08-12 at 17:47 +0200, Abel Tamayo > wrote: > > > Segmentation faults are the most mysterious of > faults... > > could it be > > > some incompatibility between Ruby 1.8.6 and the > PostgreSQL > > adapter? > > > > > > On Thu, Aug 12, 2010 at 5:38 PM, Chris Kalaboukis > > > wrote: > > > I think I found the culprit: > > > > > > ==> dj-1943630-crashlog.log <== > > > > (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) > > > RPM Monitoring DJ worker > host:railgun64.29102 > > pid:9157 > > > ** [NewRelic] Connected to NewRelic > Service at > > > collector5.newrelic.com:80 > > > ** [NewRelic] New Relic RPM Agent 2.12.3 > > Initialized: pid = > > > 9157 > > > ** [NewRelic] Agent Log found > > > > > > in /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log > > > *** Starting job worker > host:railgun64.29102 > > pid:9157 > > > > > > > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: > [BUG] Segmentation fault > > > ruby 1.8.6 (2008-08-11) [x86_64-linux] > > > > > > Any ideas on how to fix? > > > > > > > > > > > > On Thu, 2010-08-12 at 01:28 -0700, Chris > Kalaboukis > > wrote: > > > > Hi all: I've got workers crashing for no > apparent > > reason. > > > Have the > > > > exception code in there but it just > stops running > > and ps > > > reports > > > > "crashed&q
Re: workers crashing....
It's done automatically. You don't have to select or configure anything. On Thu, Aug 12, 2010 at 6:33 PM, Chris Kalaboukis wrote: > Hi Abel: > > For some reason, its not restarting the job. Is there a way to change > that setting - its not re-starting the job its just hanging there. i > need it to restart when it fails, can you tell me what I need to edit in > order to make it do that? > > Thanks...Chris > > On Thu, 2010-08-12 at 18:15 +0200, Abel Tamayo wrote: > > Heroku is supposed to restart a worker everytime it crashes. Whatever > > is in your jobs:work task will be executed as soon as it fails. > > > > > > However, I run a very dependent on background jobs webapp in Heroku > > and this is not enough for me. What I did was create a rake task that > > finds the last record to be updated and issue a background job to > > update the next after that one and automate the task to find the rest > > of the records that need to be updated. > > > > > > The only thing you'll be able to do to interact with your app once > > it's deployed are Rake tasks. > > > > > > On Thu, Aug 12, 2010 at 5:54 PM, Chris Kalaboukis > > wrote: > > Possibly. Problem is I can't seem to catch it...do you know > > how I can, > > other than running some monitor to restart. > > > > BTW, how do I restart a crashed job? Is there a way to do it > > from the > > heroku command line? > > > > Thanks...Chris > > > > > > On Thu, 2010-08-12 at 17:47 +0200, Abel Tamayo wrote: > > > Segmentation faults are the most mysterious of faults... > > could it be > > > some incompatibility between Ruby 1.8.6 and the PostgreSQL > > adapter? > > > > > > On Thu, Aug 12, 2010 at 5:38 PM, Chris Kalaboukis > > > wrote: > > > I think I found the culprit: > > > > > > ==> dj-1943630-crashlog.log <== > > > (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) > > > RPM Monitoring DJ worker host:railgun64.29102 > > pid:9157 > > > ** [NewRelic] Connected to NewRelic Service at > > > collector5.newrelic.com:80 > > > ** [NewRelic] New Relic RPM Agent 2.12.3 > > Initialized: pid = > > > 9157 > > > ** [NewRelic] Agent Log found > > > > > in > /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log > > > *** Starting job worker host:railgun64.29102 > > pid:9157 > > > > > > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: > [BUG] Segmentation fault > > > ruby 1.8.6 (2008-08-11) [x86_64-linux] > > > > > > Any ideas on how to fix? > > > > > > > > > > > > On Thu, 2010-08-12 at 01:28 -0700, Chris Kalaboukis > > wrote: > > > > Hi all: I've got workers crashing for no apparent > > reason. > > > Have the > > > > exception code in there but it just stops running > > and ps > > > reports > > > > "crashed". Neither heroku logs or New Relic have > > anything to > > > add. > > > > Where can I find out why a workers crashed like > > that? > > > > > > > > Thanks...Chris > > > > > > > > > > > > > -- > > > You received this message because you are subscribed > > to the > > > Google Groups "Heroku" group. > > > To post to this group, send email to > > her...@googlegroups.com. > > > To unsubscribe from this group, send email to heroku > > > +unsubscr...@googlegroups.com. > > > For more options, visit this group at > > > http://groups.google.com/group/heroku?hl=en. > > > > > > > > > > > > > > > > > >
Re: workers crashing....
Hi Abel: For some reason, its not restarting the job. Is there a way to change that setting - its not re-starting the job its just hanging there. i need it to restart when it fails, can you tell me what I need to edit in order to make it do that? Thanks...Chris On Thu, 2010-08-12 at 18:15 +0200, Abel Tamayo wrote: > Heroku is supposed to restart a worker everytime it crashes. Whatever > is in your jobs:work task will be executed as soon as it fails. > > > However, I run a very dependent on background jobs webapp in Heroku > and this is not enough for me. What I did was create a rake task that > finds the last record to be updated and issue a background job to > update the next after that one and automate the task to find the rest > of the records that need to be updated. > > > The only thing you'll be able to do to interact with your app once > it's deployed are Rake tasks. > > > On Thu, Aug 12, 2010 at 5:54 PM, Chris Kalaboukis > wrote: > Possibly. Problem is I can't seem to catch it...do you know > how I can, > other than running some monitor to restart. > > BTW, how do I restart a crashed job? Is there a way to do it > from the > heroku command line? > > Thanks...Chris > > > On Thu, 2010-08-12 at 17:47 +0200, Abel Tamayo wrote: > > Segmentation faults are the most mysterious of faults... > could it be > > some incompatibility between Ruby 1.8.6 and the PostgreSQL > adapter? > > > > On Thu, Aug 12, 2010 at 5:38 PM, Chris Kalaboukis > > wrote: > > I think I found the culprit: > > > > ==> dj-1943630-crashlog.log <== > > (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) > > RPM Monitoring DJ worker host:railgun64.29102 > pid:9157 > > ** [NewRelic] Connected to NewRelic Service at > > collector5.newrelic.com:80 > > ** [NewRelic] New Relic RPM Agent 2.12.3 > Initialized: pid = > > 9157 > > ** [NewRelic] Agent Log found > > > in /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log > > *** Starting job worker host:railgun64.29102 > pid:9157 > > > > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: > [BUG] Segmentation fault > > ruby 1.8.6 (2008-08-11) [x86_64-linux] > > > > Any ideas on how to fix? > > > > > > > > On Thu, 2010-08-12 at 01:28 -0700, Chris Kalaboukis > wrote: > > > Hi all: I've got workers crashing for no apparent > reason. > > Have the > > > exception code in there but it just stops running > and ps > > reports > > > "crashed". Neither heroku logs or New Relic have > anything to > > add. > > > Where can I find out why a workers crashed like > that? > > > > > > Thanks...Chris > > > > > > > > > -- > > You received this message because you are subscribed > to the > > Google Groups "Heroku" group. > > To post to this group, send email to > her...@googlegroups.com. > > To unsubscribe from this group, send email to heroku > > +unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the > Google > > Groups "Heroku" group. > > To post to this group, send email to > her...@googlegroups.com. > > To unsubscribe from this group, send email to heroku > > +unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > > -- > > You received this m
Re: workers crashing....
Heroku is supposed to restart a worker everytime it crashes. Whatever is in your jobs:work task will be executed as soon as it fails. However, I run a very dependent on background jobs webapp in Heroku and this is not enough for me. What I did was create a rake task that finds the last record to be updated and issue a background job to update the next after that one and automate the task to find the rest of the records that need to be updated. The only thing you'll be able to do to interact with your app once it's deployed are Rake tasks. On Thu, Aug 12, 2010 at 5:54 PM, Chris Kalaboukis wrote: > Possibly. Problem is I can't seem to catch it...do you know how I can, > other than running some monitor to restart. > > BTW, how do I restart a crashed job? Is there a way to do it from the > heroku command line? > > Thanks...Chris > > On Thu, 2010-08-12 at 17:47 +0200, Abel Tamayo wrote: > > Segmentation faults are the most mysterious of faults... could it be > > some incompatibility between Ruby 1.8.6 and the PostgreSQL adapter? > > > > On Thu, Aug 12, 2010 at 5:38 PM, Chris Kalaboukis > > wrote: > > I think I found the culprit: > > > > ==> dj-1943630-crashlog.log <== > > (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) > > RPM Monitoring DJ worker host:railgun64.29102 pid:9157 > > ** [NewRelic] Connected to NewRelic Service at > > collector5.newrelic.com:80 > > ** [NewRelic] New Relic RPM Agent 2.12.3 Initialized: pid = > > 9157 > > ** [NewRelic] Agent Log found > > in > /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log > > *** Starting job worker host:railgun64.29102 pid:9157 > > > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: > [BUG] Segmentation fault > > ruby 1.8.6 (2008-08-11) [x86_64-linux] > > > > Any ideas on how to fix? > > > > > > > > On Thu, 2010-08-12 at 01:28 -0700, Chris Kalaboukis wrote: > > > Hi all: I've got workers crashing for no apparent reason. > > Have the > > > exception code in there but it just stops running and ps > > reports > > > "crashed". Neither heroku logs or New Relic have anything to > > add. > > > Where can I find out why a workers crashed like that? > > > > > > Thanks...Chris > > > > > > > > > -- > > You received this message because you are subscribed to the > > Google Groups "Heroku" group. > > To post to this group, send email to her...@googlegroups.com. > > To unsubscribe from this group, send email to heroku > > +unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Heroku" group. > > To post to this group, send email to her...@googlegroups.com. > > To unsubscribe from this group, send email to heroku > > +unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers crashing....
Possibly. Problem is I can't seem to catch it...do you know how I can, other than running some monitor to restart. BTW, how do I restart a crashed job? Is there a way to do it from the heroku command line? Thanks...Chris On Thu, 2010-08-12 at 17:47 +0200, Abel Tamayo wrote: > Segmentation faults are the most mysterious of faults... could it be > some incompatibility between Ruby 1.8.6 and the PostgreSQL adapter? > > On Thu, Aug 12, 2010 at 5:38 PM, Chris Kalaboukis > wrote: > I think I found the culprit: > > ==> dj-1943630-crashlog.log <== > (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) > RPM Monitoring DJ worker host:railgun64.29102 pid:9157 > ** [NewRelic] Connected to NewRelic Service at > collector5.newrelic.com:80 > ** [NewRelic] New Relic RPM Agent 2.12.3 Initialized: pid = > 9157 > ** [NewRelic] Agent Log found > in /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log > *** Starting job worker host:railgun64.29102 pid:9157 > > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: > [BUG] Segmentation fault > ruby 1.8.6 (2008-08-11) [x86_64-linux] > > Any ideas on how to fix? > > > > On Thu, 2010-08-12 at 01:28 -0700, Chris Kalaboukis wrote: > > Hi all: I've got workers crashing for no apparent reason. > Have the > > exception code in there but it just stops running and ps > reports > > "crashed". Neither heroku logs or New Relic have anything to > add. > > Where can I find out why a workers crashed like that? > > > > Thanks...Chris > > > > > -- > You received this message because you are subscribed to the > Google Groups "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to heroku > +unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > > > > > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to heroku > +unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers crashing....
Segmentation faults are the most mysterious of faults... could it be some incompatibility between Ruby 1.8.6 and the PostgreSQL adapter? On Thu, Aug 12, 2010 at 5:38 PM, Chris Kalaboukis wrote: > I think I found the culprit: > > ==> dj-1943630-crashlog.log <== > (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) > RPM Monitoring DJ worker host:railgun64.29102 pid:9157 > ** [NewRelic] Connected to NewRelic Service at > collector5.newrelic.com:80 > ** [NewRelic] New Relic RPM Agent 2.12.3 Initialized: pid = 9157 > ** [NewRelic] Agent Log found > in /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log > *** Starting job worker host:railgun64.29102 pid:9157 > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: > [BUG] Segmentation fault > ruby 1.8.6 (2008-08-11) [x86_64-linux] > > Any ideas on how to fix? > > > On Thu, 2010-08-12 at 01:28 -0700, Chris Kalaboukis wrote: > > Hi all: I've got workers crashing for no apparent reason. Have the > > exception code in there but it just stops running and ps reports > > "crashed". Neither heroku logs or New Relic have anything to add. > > Where can I find out why a workers crashed like that? > > > > Thanks...Chris > > > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers crashing....
I think I found the culprit: ==> dj-1943630-crashlog.log <== (in /disk1/home/slugs/152245_4b6a640_e6b3/mnt) RPM Monitoring DJ worker host:railgun64.29102 pid:9157 ** [NewRelic] Connected to NewRelic Service at collector5.newrelic.com:80 ** [NewRelic] New Relic RPM Agent 2.12.3 Initialized: pid = 9157 ** [NewRelic] Agent Log found in /disk1/home/slugs/152245_4b6a640_e6b3/mnt/log/newrelic_agent.log *** Starting job worker host:railgun64.29102 pid:9157 /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1102: [BUG] Segmentation fault ruby 1.8.6 (2008-08-11) [x86_64-linux] Any ideas on how to fix? On Thu, 2010-08-12 at 01:28 -0700, Chris Kalaboukis wrote: > Hi all: I've got workers crashing for no apparent reason. Have the > exception code in there but it just stops running and ps reports > "crashed". Neither heroku logs or New Relic have anything to add. > Where can I find out why a workers crashed like that? > > Thanks...Chris > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
workers crashing....
Hi all: I've got workers crashing for no apparent reason. Have the exception code in there but it just stops running and ps reports "crashed". Neither heroku logs or New Relic have anything to add. Where can I find out why a workers crashed like that? Thanks...Chris -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
