librelist archives

« back to archive

Introducing the Majordomo pattern to handle task messaging

Introducing the Majordomo pattern to handle task messaging

From:
Loic d'Anterroches
Date:
2011-10-25 @ 12:56
Hello,

as I need to do something like:

1. request comes in.
2. contact service 1 for computation
3. contact service 2 for computation independently of service 1
4. aggregate the results
5. send an answer

I need to do a bit more than the current task runner. The current task
runner is nice but you need to push the requests and then read them
afterwards without a general timeout. That is, I want to say that the
total of steps 2 and 3 is "50 ms max" and discard the computation 1 or 2
or both if they cannot make the cut.

To do that, I will implement an extended Majordomo service:
http://zguide.zeromq.org/page%3aall#toc72

It will be based on the one shown above but will add the support of
multiple workers per service (only one in the given implementation) and
timeout at the client side to discard results above the cut.

This means that on this image:
https://github.com/imatix/zguide/raw/master/images/fig59.png

A client is a Photon handler, but you can have many "Water" or many
"Tea" workers. That is, you can scale each worker (a Photon task or task
built in any language) independently while keeping on the client side a
single point of connection (a client does not need to connect to all the
workers). Also, the broker can be smart in the scheduling of the
requests to the workers.

I will ensure that the tasks are still easy to write but this means that
you will need a broker process. But the code is 200 lines of PHP and
nothing more, so pretty simple. It will just be a run of:

$ hnu broker

Have a nice day,
loïc

--
Indefero - Project management and code hosting - http://www.indefero.net
Photon - High Performance PHP Framework - http://photon-project.com
Céondo Ltd - Web + Science = Fun - http://www.ceondo.com

Re: [photon.users] Introducing the Majordomo pattern to handle task messaging

From:
Loic d'Anterroches
Date:
2011-10-27 @ 07:10
Hello,

oh, a very nice thing about this pattern: both the workers and the
clients are connecting to the same ZeroMQ port. That is, you need only
one "port" to manage all the work. This is really good, this means we
will be able to have a relatively "sane" default port on localhost for
the default configuration and nothing else to have a system working out
of the box.

I will also some small Python examples for you to create workers in Python.

loïc


On 2011-10-25 14:56, Loic d'Anterroches wrote:
> Hello,
> 
> as I need to do something like:
> 
> 1. request comes in.
> 2. contact service 1 for computation
> 3. contact service 2 for computation independently of service 1
> 4. aggregate the results
> 5. send an answer
> 
> I need to do a bit more than the current task runner. The current task
> runner is nice but you need to push the requests and then read them
> afterwards without a general timeout. That is, I want to say that the
> total of steps 2 and 3 is "50 ms max" and discard the computation 1 or 2
> or both if they cannot make the cut.
> 
> To do that, I will implement an extended Majordomo service:
> http://zguide.zeromq.org/page%3aall#toc72
> 
> It will be based on the one shown above but will add the support of
> multiple workers per service (only one in the given implementation) and
> timeout at the client side to discard results above the cut.
> 
> This means that on this image:
> https://github.com/imatix/zguide/raw/master/images/fig59.png
> 
> A client is a Photon handler, but you can have many "Water" or many
> "Tea" workers. That is, you can scale each worker (a Photon task or task
> built in any language) independently while keeping on the client side a
> single point of connection (a client does not need to connect to all the
> workers). Also, the broker can be smart in the scheduling of the
> requests to the workers.
> 
> I will ensure that the tasks are still easy to write but this means that
> you will need a broker process. But the code is 200 lines of PHP and
> nothing more, so pretty simple. It will just be a run of:
> 
> $ hnu broker
> 
> Have a nice day,
> loïc
> 
> --
> Indefero - Project management and code hosting - http://www.indefero.net
> Photon - High Performance PHP Framework - http://photon-project.com
> Céondo Ltd - Web + Science = Fun - http://www.ceondo.com