Shoryuken is probably the best option when it comes to processing background jobs using SQS with ActiveJob [1], however I couldn’t find any reliable documentation online to get it working with ElasticBeanstalk worker tiers. After some trial and error, I was able to implement a reliable setup.

Configure Shoryuken

If you haven’t already, create a shoryuken.yml file located in the config directory of your Rails project to specify what queues you want to consume:

# config/shoryuken.yml
queues:
  - my_queue1
  - my_queue2

This is a bare minimum config, so I recommend taking a look at the documentation for more options.

Setting up the ebextension

We’re going to use an ebextension to create a post-deploy script that will restart Shoryuken after deployment to ensure the latest changes are loaded.

Before we do that though, you’ll need to create a new environment variable on your worker tier. Unfortunately, ebextensions do not let you specify tier-specifc deploy scripts so this variable will be used to make sure the script only runs on your worker tiers.

The following command will create a WORKER variable with the value set to true:

eb setenv WORKER=true -e <worker tier environment name>

Now we’re ready to create the extension. Create a directory at the root of your project titled .ebextensions if it doesn’t already exist. Create a file named shoryuken.config under your .ebextensions directory with the following:

# .ebextensions/shoryuken.config
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_shoryuken":
    mode: "000777"
    content: |
      # Load environment variables and your app's ruby version
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      # Exit immediately if not on the worker tier
      if [ $WORKER = true ]; then
        exit 0
      fi

      cd /var/app/current

      # Kill the currently running shoryuken process if the pid file exists
      if [ -f /var/app/support/pids/shoryuken.pid ]
      then
        kill -TERM `cat /var/app/support/pids/shoryuken.pid`
        rm -rf /var/app/support/pids/shoryuken.pid
      fi

      sleep 5

      # Start shoryuken specifying the pid, log, and config paths
      bundle exec shoryuken \
        -R \
        -P /var/app/support/pids/shoryuken.pid \
        -C /var/app/current/config/shoryuken.yml \
        -L /var/app/support/logs/shoryuken.log \
        -d

At a high level, this script kills the currently running shoryuken process, waits five seconds for jobs to cleaned up, then starts it again.

Scheduling Jobs

Scheduling jobs with Shoryuken is a little more tricky. The easiest method I’ve found is to set up a CloudWatch Event to send a message to a specified SQS queue at a certain interval.

One Caveat…

Unfortunately you can’t use ActiveJob for CloudWatch scheduled jobs. This is because Shoryuken relies on specific SQS message attributes when working on jobs triggered with ActiveJob and CloudWatch doesn’t allow you to define these.

The workaround is to use a plain Ruby class instead of an ActiveJob subclass. The minor downside is that you will have to call Shoryuken-specific perform methods instead of ActiveJob’s perform methods (e.g.: perform_later) to enqueue your job if you plan on enqueueing it within your app in addition to scheduling it.

Here’s an example of a “plain Ruby” Shoryuken job:

class MyJob
  include Shoryuken::Worker

  shoryuken_options queue: :my_queue, auto_delete: true

  def perform(*args)
    # job logic
  end
end

Now head on over to the CloudWatch Management Console > Rules > Create Rule. Here you can specify a schedule and a target. Select your desired schedule and select “SQS Queue” from the target dropdown. Select your queue, then select “Constant (JSON text)” and put in the following:

{
	"job_class": "MyJob",
	"job_id": "00000000-0000-0000-0000-000000000000",
	"queue_name": "my_queue",
	"priority": null,
	"arguments": [],
	"locale": "en"
}

Change job_class to be the same name as the job you just created, and queue_name to the name of your SQS queue.

Now Shoryuken will be able to process scheduled jobs!

[1] Besides rolling your own SQS consumer, your two main options are either Shoryuken or active_elastic_job. Even though there’s a bit more setup, I decided to go with Shoryuken because active_elastic_job is very limited (for example, it doesn’t support multiple queues) and at the time of posting it appears to be unmaintained.