The code used to automatically fire of DelayedJob workers work trigger every time Rails was loaded, including when a DelayedJob worker initialized. This had the obvious fork-bomb effect...
On inspection it turns out that the different process models of Webrick, Passenger, and Unicorn, make doing this in a robust and effective way difficult. Instead, we back out that code and document the need to run a worker process, along with the ways to have that running. This puts responsibility on the user to get a worker in place. Paired-With: Matt Robinson <[email protected]> --- README.markdown | 31 ++++++++++++++++++--- config/initializers/delayed_job.rb | 25 ----------------- config/settings.yml.example | 3 -- db/migrate/20110614234202_create_delayed_jobs.rb | 14 ++++++++++ 4 files changed, 40 insertions(+), 33 deletions(-) diff --git a/README.markdown b/README.markdown index f199272..66cf139 100644 --- a/README.markdown +++ b/README.markdown @@ -132,7 +132,20 @@ Installation rake db:migrate db:test:prepare -6. Adjust the `delayed_job_workers` setting in `settings.yml`. You should set this to the number of worker processes you want for background jobs. We recommend one per CPU core in your system, but will work correctly with one or more. +6. Start the `delayed_job` workers for your deployment. You need at least one worker for each environment you run, and we recommend one worker per CPU core in production. + + 1. For typical use with the `production` environment, one worker per CPU: + + env CPUS=4 RAILS_ENV=production /.../script/delayed_job \ + -p dashboard -n $CPUS -m start + + 2. Using the rake task to start a single worker (not recommended for production): + + rake RAILS_ENV=production jobs:work + + 3. Running a development worker for testing: + + rake RAILS_ENV=development jobs:work Ownership and permission requirements @@ -189,6 +202,12 @@ Regardless of how you installed the code, you need to run the database migration After upgrading the code and running the migrations, you'll need to restart your Puppet Dashboard server for these changes to take effect, which may require restarting your webserver. +### Delayed Job Background Tasks + +When you upgrade from Puppet Dashboard 1.1.1 or earlier, you will need to configure worker processes for the `delayed_job` system. This is used to significantly improve responsiveness of the dashboard, especially to improve throughput from a Puppet Master when reports are sent to the Puppet Dashboard. + +Please see the notes at the end of `Running` below, or `Background Processing`, for the ways to get this running. Without at least one running worker, reports will no longer finish importing into the Puppet Dashboard database. + Running ------- @@ -356,18 +375,20 @@ Background Processing The Puppet Dashboard performs a number of tasks, such as report import, that can consume significant system resources. To ensure that performance remains snappy under load, we use the `delayed_job` background processing system to manage these tasks without tying up a web front end thread. -The Puppet Dashboard code will automatically spawn a manager and worker in the background, which will run these tasks without tying up the resources of the web server. This will share the same credentials and access as the web front-end, so should introduce no additional security risk. +You will need to configure at least one background worker process to run these tasks. We recommend one worker per core in a production environment, for better performance. More workers than CPU cores will not improve throughput. + +You can use `script/delayed_job` to act as a process supervisor, forking the desired number of workers and restarting them if they abort. We recommend this approach, compared to using the individual rake task to start a worker. -You should set the `delayed_job_workers` configuration value to the number of workers you want running; we recommend one per core. If this is greater than zero we will automatically start a worker monitor in the background to perform background tasks. +For additional reliability, you might want to use a service monitoring tool like [god][http://god.rubyforge.org/], [monit][http://mmonit.com/monit/], or [runit][http://smarden.org/runit/]. We generally recommend you supervise the `script/delayed_job` monitor, but provided the workers keep running things should be good. -This is not entirely efficient or easy to monitor, however, so you might instead want to set that to zero and fire up the worker monitor yourself. See `scripts/delayed_job --help` for details on firing up the worker daemon manually. +There is also plenty of great documentation available about different ways to get `delayed_job` workers running, such as http://stackoverflow.com/questions/1226302/how-to-monitor-delayed-job-with-monit Performance ----------- The Puppet Dashboard slows down as it manages more data. Here are ways to make it run faster, from easiest to hardest: -* Tune `delayed_job_workers` to have one worker per CPU core on your system. +* Run one, and only one, `delayed_job` worker per CPU core. * Optimize your database by running `rake RAILS_ENV=production db:raw:optimize` from your Puppet Dashboard directory, this will reorganize and reanalyze your database for faster queries. * Run the application in `production` mode, e.g. by running `./script/server -e production`. The default `development` mode is significantly slower because it doesn't cache and logs more details. * Run the application using multiple processes to handle more concurrent requests. You can use Phusion Passenger, or clusters of Thin or Unicorn servers to serve multiple concurrent requests. diff --git a/config/initializers/delayed_job.rb b/config/initializers/delayed_job.rb index 2346782..5dce260 100644 --- a/config/initializers/delayed_job.rb +++ b/config/initializers/delayed_job.rb @@ -1,27 +1,2 @@ -DELAYED_JOB_PID_PATH = Pathname.new "#{Rails.root}/tmp/pids/delayed_job_#{Rails.env}" -DELAYED_JOB_PID_PATH.mkpath - Delayed::Worker.destroy_failed_jobs = false Delayed::Worker.max_attempts = 3 - -def start_delayed_job - return if Rails.env == 'test' - return if SETTINGS.delayed_job_workers <= 0 - Thread.new do - `#{Rails.root}/script/delayed_job --pid-dir=#{DELAYED_JOB_PID_PATH} -p dashboard -n #{SETTINGS.delayed_job_workers || 2} -m start` - end -end - -def process_is_dead? - begin - pid = File.read(DELAYED_JOB_PID_PATH).strip - Process.kill(0, pid.to_i) - false - rescue - true - end -end - -if !File.exist?(DELAYED_JOB_PID_PATH + 'delayed_job.pid') && process_is_dead? - start_delayed_job -end diff --git a/config/settings.yml.example b/config/settings.yml.example index 1f8945a..2cf416c 100644 --- a/config/settings.yml.example +++ b/config/settings.yml.example @@ -77,7 +77,4 @@ disable_legacy_report_upload_url: false # Disables the UI and controller actions for editing nodes, classes, groups and reports. Report submission is still allowed enable_read_only_mode: false -# Delayed job worker process count - we recommend one per core on your system -delayed_job_workers: 2 - #===[ fin ]============================================================= diff --git a/db/migrate/20110614234202_create_delayed_jobs.rb b/db/migrate/20110614234202_create_delayed_jobs.rb index ecb7db9..cd9b62b 100644 --- a/db/migrate/20110614234202_create_delayed_jobs.rb +++ b/db/migrate/20110614234202_create_delayed_jobs.rb @@ -23,6 +23,20 @@ class CreateDelayedJobs < ActiveRecord::Migration end add_index :delayed_jobs, [:priority, :run_at], :name => 'delayed_jobs_priority' + + STDERR.puts <<EOT +======================================================================== +You MUST run at least one `delayed_job` worker on your system, so that +background jobs are processed. Without this various parts of dashboard, +especially things like report import, will not work. + +Please see `README.markdown` for details of how to run the workers, or +use one of: + +] rake RAILS_ENV=production jobs:work +] ./script/delayed_job -p dashboard -n $CPUS -m start +======================================================================== +EOT end def self.down -- 1.7.5.4 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
