Python version: 2.7
Python modules: multiprocessing, argparse, time, datetime, sys, os
** if possible, [ import multiprocessing as mp ]
External Python modules:
Pika ([login to view URL])
Pika documentation ([login to view URL])
Psycopg2 ([login to view URL])
Server OS: Fedora 14
For the sake of the specification we'll call the app "vnetserver"
The app should be controlled by an "init" run script called "vnetctrl"
service vnetctrl {start|stop|restart|status}
....status: display main parent and any running children
"vnetctrl" starts "vnetserver"
vnetserver is called with a single command line option "-f" which points to the config file
example: vnetserver -f /etc/vnet/[login to view URL]
The config file contains the following:
rabbitserver: [login to view URL]
pgdsn: host=[login to view URL] dbname=mydbname user=mydnuser
App starts and creates a pool of workers, 2x workers per cpu
no. of cpus determined with multiprocessing.cpu_count() .... [ mp.cpu_count() ]
App reads a message from the queue and passes it to a worker
worker connects to postgres and inserts the message
if the insert is successful, worker then acknowledges consumption (channel.basic_ack...)
worker returns to the pool
Additional thoughts:
Is it possible to dynamically adjust the "worker pool size" based on system load?
At this stage the message format is a simple test message which is the "epoch. milliseconds"
Python Pika comes with various examples
The objective is to test that Python will run on all cpus when given a heavy load.