librelist archives

« back to archive

Broker, workers, tasks and the fun of managing all of them

Broker, workers, tasks and the fun of managing all of them

Loic d'Anterroches
2011-12-13 @ 13:08

just a short notice to inform you about what I am up to at the moment.
Basically I keep developing Bareku to support Python for my scientific
requirements. The requirements are to provide an array of services to
the web applications coded with Photon.

What you get is that a single service, let say "recoengine", can be made
available by 4 processes on 2 differents VMs. This allows scaling. Each
recoengineX is basically the exact same code, like
 $ php myproject.phar task recoengine
but started on different VMs and possibly with different parameters

So, you get:

Photon web app -+---> VM1 -+---> recoengine1
                |          +---> recoengine2
                +---> VM2 -+---> recoengine3
                           +---> recoengine4

How do we get up to this point without the need to have each Photon web
app process connecting directly to the N process tasks is my current work.

The simplest approach is to create a ZMQ QUEUE device for each task,
this way the recoengine processes connect to the queue at the back and
the web app processes to the queue at the front. The queue can be very
optimized (just a bit of C code with just two binds).

Or we can create a broker. I started to work on the broker but now, I am
not totally sure because the broker adds a level of complexity and
requires a protocol.

So, I will be doing some real life testing to figure out all that and
figure out what should be the simplest most common pattern, then I will
simply add it to Photon.


Indefero - Project management and code hosting -
Photon - High Performance PHP Framework -
Céondo Ltd - Web + Science = Fun -