Phagescan Master¶
- The Phagescan Master is the front-end and includes the following services:
- Django, Celery, RabbitMQ, PostgreSQL, Nginx, Gunicorn, and Supervisord.
- Django
- Runs the Phagescan UI and Admin UI on localhost.
- Celery
- The tasking system that communicates with the workers.
- RabbitMQ
- The broker between the master and worker celery tasks.
- PostgreSQL
- The database used by Django.
- Nginx
- The external-facing service that proxies requests to Django.
- Gunicorn
- Used to start Django automatically.
- Supervisord
- Used to start CeleryCAM automatically.
Starting/Stopping All Django and Celery Services¶
The script dev/ps_services.sh
can be used to start, stop, restart all of the Master Django and Celery services at one time.
Start¶
Assuming the RabbitMQ service is running and configured, you can start Django and Celery as follows:
sudo rabbitmqctl start_app
sudo dev/ps_services.sh start
Stop¶
You can stop the Django and Celery services and the RabbitMQ app as follows:
sudo dev/ps_services.sh stop
sudo rabbitmqctl stop_app
Restart¶
You can restart the Django and Celery services and reset the RabbitMQ app as follows:
sudo dev/ps_services.sh stop
sudo rabbitmqctl stop_app
sudo rabbitmqctl start_app
sudo dev/ps_services.sh start
This can be helpful to reset the Celery queues.
Django¶
Django can be started manually or with startup scripts. During development it is best to start it manually. A production environment will use the startup scripts.
Either way, Django should only listen on localhost. To access Phagescan from outside the host, Nginx is used as a proxy.
Manual Start/Stop¶
Run as same user that you used to start the 3 celeryd processes:
[avuser@host]$ python manage.py runserver -v 3 127.0.0.1:9000 --settings=scaggr.settings
Stop it with CTRL+C.
Automated Start/Stop¶
In Phagescan, we use gunicorn
to automatically start Django on boot.
Start with:
$ sudo service gunicorn start
Stop with:
$ sudo service gunicorn stop
See installation/salt-masterless/salt/gunicorn for sample configuration files and startup script.
Accessing¶
Access the Phagescan Web User Interfaces at:
http://127.0.0.1:9000
http://127.0.0.1:9000/admin
Celery¶
The Scan Master has 4 different Celery services running at the same time.
- CeleryCAM - this is only used in production
- Periodic Runner - run periodic tasks
- Result Collector - collect results from workers
- Master Task - manage tasks and send tasks to workers
Services 2-4 require a celery config file to be located in the [Project_root_dir]. The CeleryCAM uses a config file withing the /etc/supervisor/conf.d/ directory.
Manual Start/Stop¶
Start the 3 celery processes each in separate terminals:
[avuser@host]$ DJANGO_SETTINGS_MODULE=scaggr.settings celeryd --config=masterceleryconfig -E -B -l info --hostname=master.master
[avuser@host]$ DJANGO_SETTINGS_MODULE=scaggr.settings celeryd --config=resultsceleryconfig -E -B -l info --hostname=master.results
[avuser@host]$ DJANGO_SETTINGS_MODULE=scaggr.settings celeryd --config=periodicceleryconfig -E -B -l info --hostname=master.periodic
Stop them with CTRL+C.
If you want debug output, change the -l info
to -l debug
.
Automated Start/Stop¶
In Phagescan, we use Celeryd startup scripts and supervisord
to start the 4 Celery services automatically.
Start with:
$ sudo service celeryd-master start
$ sudo service celeryd-periodic start
$ sudo service celeryd-result start
$ sudo service supervisord start
Stop with:
$ sudo service celeryd-master stop
$ sudo service celeryd-periodic stop
$ sudo service celeryd-result stop
$ sudo service supervisord stop
See installation/salt-masterless/salt/[celery/master | supervisord] for sample configuration files and startup script.
RabbitMQ¶
RabbitMQ is installed as an Ubuntu package and will start automatically on boot, by default. For Phagescan, all we need to do is add users, permissions, and vhosts for Celery to use.
Adding Users, Permissions, and Vhosts¶
Replace the username and password with the credentials that you would like to use and run the following commands:
$ sudo rabbitmqctl add_user phagemasteruser longmasterpassword
$ sudo rabbitmqctl add_user phageworkeruser longworkerpassword
$ sudo rabbitmqctl add_vhost phage
$ sudo rabbitmqctl set_permissions -p phage phagemasteruser ".*" ".*" ".*"
$ sudo rabbitmqctl set_permissions -p phage phageworkeruser ".*" ".*" ".*"
If the master and worker hosts use different user/pass combinations to communicate with the broker, you must use the commands as written above. However, if the master and worker hosts use the same user/pass, you may add one user and set_permissions on that one user. In that case, there is no need for the second user. A production system should use separate credentials for the master and workers.
Note: These credentials are the BROKER_CONF for the master and workers. So, make sure the username and password you created here are set in both the master and worker BROKER_CONFs:
For the master it is in scaggr/settings.py. For the worker it is in workerceleryconfig.py.
Deleting Users, Permissions, and Vhosts¶
You delete the user and vhost and the permissions are automatically deleted.
$ sudo rabbitmqctl del_user phagemasteruser
$ sudo rabbitmqctl del_user phageworkeruser
$ sudo rabbitmqctl del_vhost phage
Starting the RabbitMQ Application¶
You only have to do this if you've manually stopped the application. Starting the RabbitMQ service does this automatically.
$ sudo rabbitmqctl start_app
Stopping the RabbitMQ Application¶
This is how you would clear/delete the Celery Queues from RabbitMQ after you stop the Celery services:
$ sudo rabbitmqctl stop_app
Examining the Queues¶
RabbitMQ is the broker for the Celery queues, so you can examine many details about the Celery queues using rabbitmqctl
.
See the rabbitmqctl documentation for guidance.
One of the more useful commands is:
$ sudo rabbitmqctl -p phage list_queues name consumers messages
This will list all of the queues on the phage vhost; specifically the name, number of consumers, and number of messages for each queue.
PostgreSQL¶
Configure¶
Replace the username and password with the credentials that you would like to use and run the following commands:
$ sudo su postgres
$ psql
postgres=# create user citestsuper createdb superuser password 'sup3rdup3r';
postgres=# create database phage owner citestsuper;
postgres=# \q
$ psql -d phage
phage=# create extension hstore;
phage=# \q
$ exit
Starting/Stopping¶
This is a standard service, so start and stop by doing:
$ sudo service postgresql start
$ sudo service postgresql stop
Nginx¶
A default Nginx config file is provided in installation/salt-masterless/salt/nginx/. Once it is configured, you can start or stop the nginx service as follows:
$ sudo service nginx start
$ sudo service nginx stop
Note: after changing the configuration file, you should restart the service.