Helioqueuer

From Helioviewer Wiki

Overview

Helioqueuer is the background event queue for Helioviewer.org and the Helioviewer iPhone app. Its goal is to manage tasks such as: warming image caches, pre-generating event movies, intelligently clearing old movies and caches.

Installing and Configuring Helioqueuer

Helioqueuer is written in Ruby, and makes use of several existing Ruby libraries, known as "gems". Similar to Perl's CPAN, Ruby also has it's own repository of gems which can be easily installed using the "rubygem" tool. The below instructions describe how to install all of the prerequisites needed for Helioqueuer using a combination of software from the Ubuntu repositories and RubyGems.

Download Helioqueuer

Using Bazaar, download a copy of the latest version of the Helioqueuer code from Launchpad:

bzr branch lp:helioqueuer

Install Prerequisites

Next you will need to install some prerequisites that are not available through rubygem, including the rubygem tool itself. On Ubuntu, this can be done using the command:

sudo apt-get install ruby ruby-dev rake rubygems redis-server libmysqlclient-dev

and on Arch Linux:

sudo pacman -S ruby redis mysql-ruby

This will install the Ruby and MySQL Client development headers, rake (a ruby build system similar to Make), RubyGems, and Redis.

In order for the rubygems to be located by Helioqueuer, you need to adjust your system's PATH environmental variable. If you installed the gems at the system level using sudo, you can adjust the PATH variable by editing /etc/environment, e.g.:

#Ubuntu
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/var/lib/gems/1.8/bin"

#Arch
PATH="$PATH:/usr/lib/ruby/gems/1.9.1/gems/"

If you installed the gems at the user level, then you can adjust the PATH variable in .profile or .bashrc, e.g.:

PATH="$PATH:~/.gem/ruby/1.8/bin/"

Finally, using rake, install the remaining dependencies available through RubyGems:

sudo rake install:gems

On Arch Linux you may additionally need to install rack:

sudo gem install rack

Configure Helioqueuer

To configure Helioqueuer, enter the "config" directory and create a copy of the example configuration file, config.example.rb, provided the the Helioqueuer source code. Adjust any necessary parameters so that the values reflect your own system configuration. Once you are done making any changes rename the new file to config.rb. Finally, don't forget to adjust your Helioviewer.org configuration file to enable it to work with Helioqueuer.

Running Helioqueuer

Helioqueuer includes several different components, each of which needs to be started separately. These components are:

1. Helioqueuer Web API
2. Resque Web
3. Resque Scheduler
4. Resque Workers

These command can be run manually from the command line, or using a simple daemon and Cron.

Running Helioqueuer Manually

To start Helioqueuer manually, enter the Helioqueuer root directory and run the following commands:

rackup -Ilib boot.ru
rake resque:scheduler
QUEUE=* COUNT=3 rake resque:workers 

Note that some commands will take over standard output and need to be either backgrounded using "&", or run separately using something like Screen.

Using a daemon

In order to simplify the process of running Helioqueuer, a simple shell script is provided with the Helioqueuer code (start-hq.example.sh) which can be used to launch all of the required components. The file provided initially looks like:

#!/bin/bash
###############################################################################
#
# Helioqueuer Service Daemon
#
###############################################################################
 
# set environment variables including ruby library path
source /etc/environment
 
# Check for logging directories
if [ ! -d "log" ]; then
    mkdir log
fi
 
# Launch Helioqueuer services
export USER="user" && export PASS="password" && rackup -Ilib boot.ru 2>log/web_server &
rake resque:scheduler 2>log/resque_scheduler &
 
# Setup queues
QUEUE=on_demand_movie COUNT=1 rake resque:workers 2>log/resque_workers &
QUEUE=alternate_format_movie COUNT=1 rake resque:workers 2>log/resque_workers &
QUEUE=image_loader rake resque:work 2>log/resque_workers &

Save the file as a copy (e.g. start-hq.sh), and edit it to meet your server's needs. In particular, a unique username and password should be chosen which will be used to secure the Resque web-interface (optional), if that is desired.

Next create a cronjob to call the script each time the system is restarted, run "crontab -e" and add the following:

@reboot /bin/bash /home/username/helioqueuer/start-hq.sh

Done! Helioqueuer will now be automatically started when your system is booted.

Advanced

Enable Remote Access to Resque Web

When running the Resque web monitor tool, by default the service resides on port 5678. Aside from allowing the outside world access to that port or reassigning resque-web to another externally-accessible port, another option is to allow access through a reverse proxy using mod_proxy so that it can be accessed from port 80, e.g. http://server.org/resque.

To configure your server to forward requests to http://<server name>/resque, begin by enabling mod_proxy:

sudo a2enmod proxy
sudo a2enmod proxy_http

Next, edit mod_proxy's configuration and add the below information inside the <IfModule> block:

<Proxy /var/www/resque>
   AddDefaultCharset off
   Order deny,allow
   Allow from all
</Proxy>

ProxyPass /resque/ http://localhost:4567/resque/

That's it! Make sure you restart Apache in order for the changes to take effect:

sudo service apache restart

Managing workers from the command-line

You can use Linux signals to manager works from the command-line. For example, to gracefully stop a worker, you can use:

kill -s STOP <pid of worker>

For a list of other signals that are supported, see this mailing list thread.

Development Information

Rake Tasks

rake -T

rake install:check              # Check system dependencies
rake install:dep:ubuntu         # Custom task for installing on Ubuntu
rake install:gems               # Install all gems: ["redis", "resque", "resque-scheduler", "god"]
rake q:reload_scheduler         # Reload the scdeduler, use if you changed the scheduling
rake q:restart_workers          # Restart the workers (will also kill them if god isn't watching)
rake q:start                    # Start the scheduler and the workers
rake q:status                   # Show the status of everything monitored
rake q:stop                     # Stop scheduler and workers
rake redis:drop_events          # Forget all events seen -- depends on redis-cli
rake redis:empty                # Empty the database -- depends on redis-cli
rake resque:scheduler           # Start Resque Scheduler
rake resque:work                # Start a Resque worker
rake resque:workers             # Start multiple Resque workers.

Redis Notes

• For a list of commands: Redis Commands
• To remove all items in all databases, use "redis-cli FLUSHALL"
• Change databases with SELECT (DBNUM)
• Get a random key from the current DB with RANDOMKEY
• For simplicity, redis uses a numbered system for databases. Below is the current map of what database holds what information:

0 = Resque
1 = Event IDs 
2 = Event JSON 

Resque Usage

resque list                                                    # Show Resque worker statuses
resque-web                                                     # Start Resque status monitor (localhost:5678)
VERBOSE=true QUEUE=* rake resque:work                          # Start a single worker with verbose logging for all queues 
COUNT=5 QUEUE=* rake resque:workers                            # Start five workers for all queues
COUNT=5 QUEUES=on_demand_movie,hek_poll rake resque:workers    # Start five workers which look for on_demand_movie tasks first, then hek_poll tasks

HEK Event customization

To define new events, see: HelioqueuerEvents