librelist archives

« back to archive

Refactoring workers

Refactoring workers

From:
William Martin
Date:
2014-06-25 @ 12:13
Hi,

I am working on projects with 10 to 20 photon workers.
Deploying multiple instances of the application is a nightmare about ports
assignment.

First, i have looks to Majordomo protocol to reduce the number of port to 1
by application instance.
Majordomo is a job broker, and can be a single point of failure.
http://rfc.zeromq.org/spec:18

Then, i find the ZMQ discover protocol named ZRE.
It's a UDP broadcast protocol to connect worker each to others, and
transfers message in TCP.
They have no single point of failure, and workers can go away and come back.
http://rfc.zeromq.org/spec:36

I will upgrade photon to use ZRE.

Any questions are welcome.

William MARTIN

Re: [photon.users] Refactoring workers

From:
Loic d'Anterroches
Date:
2014-06-25 @ 12:35
Hello,


> I am working on projects with 10 to 20 photon workers.
> Deploying multiple instances of the application is a nightmare about
> ports assignment.
>
> First, i have looks to Majordomo protocol to reduce the number of port
> to 1 by application instance.
> Majordomo is a job broker, and can be a single point of failure.
> http://rfc.zeromq.org/spec:18
>
> Then, i find the ZMQ discover protocol named ZRE.
> It's a UDP broadcast protocol to connect worker each to others, and
> transfers message in TCP.
> They have no single point of failure, and workers can go away and come back.
> http://rfc.zeromq.org/spec:36
>
> I will upgrade photon to use ZRE.
>
> Any questions are welcome.

Would it be possible to use it on demand and keep the fixed port 
approach available? Not everybody can broadcast on UDP and sometimes, 
people are using software one level "upper" to manage all the ports etc.

loïc

Re: [photon.users] Refactoring workers

From:
William Martin
Date:
2014-06-25 @ 12:39
Of course, i will do the maximum to not break the existing


On Wed, Jun 25, 2014 at 2:35 PM, Loic d'Anterroches <loic@ceondo.com> wrote:

> Hello,
>
>
> > I am working on projects with 10 to 20 photon workers.
> > Deploying multiple instances of the application is a nightmare about
> > ports assignment.
> >
> > First, i have looks to Majordomo protocol to reduce the number of port
> > to 1 by application instance.
> > Majordomo is a job broker, and can be a single point of failure.
> > http://rfc.zeromq.org/spec:18
> >
> > Then, i find the ZMQ discover protocol named ZRE.
> > It's a UDP broadcast protocol to connect worker each to others, and
> > transfers message in TCP.
> > They have no single point of failure, and workers can go away and come
> back.
> > http://rfc.zeromq.org/spec:36
> >
> > I will upgrade photon to use ZRE.
> >
> > Any questions are welcome.
>
> Would it be possible to use it on demand and keep the fixed port
> approach available? Not everybody can broadcast on UDP and sometimes,
> people are using software one level "upper" to manage all the ports etc.
>
> loïc
>



-- 
---------------------------------------------------------
William MARTIN
wysman @NoSpAm@ gmail @DoT@ com

Re: [photon.users] Refactoring workers

From:
Roger Qiu
Date:
2014-06-26 @ 04:43
Hey,

ZRE sounds pretty interesting. Did you take a look at zbeacon from the CZMQ?

Also why did you choose ZRE? There seems to be other kinds of multicast 
discovery algorithms. Did you try or investigate other ones? I'm curious 
because I'm trying to figure out a similar problem.

Thanks,
Roger

On 25/06/2014 10:35 PM, Loic d'Anterroches wrote:
> Hello,
>
>
>> I am working on projects with 10 to 20 photon workers.
>> Deploying multiple instances of the application is a nightmare about
>> ports assignment.
>>
>> First, i have looks to Majordomo protocol to reduce the number of port
>> to 1 by application instance.
>> Majordomo is a job broker, and can be a single point of failure.
>> http://rfc.zeromq.org/spec:18
>>
>> Then, i find the ZMQ discover protocol named ZRE.
>> It's a UDP broadcast protocol to connect worker each to others, and
>> transfers message in TCP.
>> They have no single point of failure, and workers can go away and come back.
>> http://rfc.zeromq.org/spec:36
>>
>> I will upgrade photon to use ZRE.
>>
>> Any questions are welcome.
> Would it be possible to use it on demand and keep the fixed port
> approach available? Not everybody can broadcast on UDP and sometimes,
> people are using software one level "upper" to manage all the ports etc.
>
> loïc

-- 
Founder of Polycademy & SnapSearch
http://polycademy.com
https://snapsearch.io
+61420925975

Re: [photon.users] Refactoring workers

From:
William Martin
Date:
2014-06-26 @ 05:54
Hi,

zbeacon is more low level. it do not connect peers one to others.
ZRE is based on zbeacon for the discover part, and other common ZMQ stuff
for the communication part, like dealer socket, hwm, timeout ...

Thanks,
William


On Thu, Jun 26, 2014 at 6:43 AM, Roger Qiu <roger.qiu@polycademy.com> wrote:

> Hey,
>
> ZRE sounds pretty interesting. Did you take a look at zbeacon from the
> CZMQ?
>
> Also why did you choose ZRE? There seems to be other kinds of multicast
> discovery algorithms. Did you try or investigate other ones? I'm curious
> because I'm trying to figure out a similar problem.
>
> Thanks,
> Roger
>
> On 25/06/2014 10:35 PM, Loic d'Anterroches wrote:
> > Hello,
> >
> >
> >> I am working on projects with 10 to 20 photon workers.
> >> Deploying multiple instances of the application is a nightmare about
> >> ports assignment.
> >>
> >> First, i have looks to Majordomo protocol to reduce the number of port
> >> to 1 by application instance.
> >> Majordomo is a job broker, and can be a single point of failure.
> >> http://rfc.zeromq.org/spec:18
> >>
> >> Then, i find the ZMQ discover protocol named ZRE.
> >> It's a UDP broadcast protocol to connect worker each to others, and
> >> transfers message in TCP.
> >> They have no single point of failure, and workers can go away and come
> back.
> >> http://rfc.zeromq.org/spec:36
> >>
> >> I will upgrade photon to use ZRE.
> >>
> >> Any questions are welcome.
> > Would it be possible to use it on demand and keep the fixed port
> > approach available? Not everybody can broadcast on UDP and sometimes,
> > people are using software one level "upper" to manage all the ports etc.
> >
> > loïc
>
> --
> Founder of Polycademy & SnapSearch
> http://polycademy.com
> https://snapsearch.io
> +61420925975
>
>


-- 
---------------------------------------------------------
William MARTIN
wysman @NoSpAm@ gmail @DoT@ com

Re: [photon.users] Refactoring workers

From:
William Martin
Date:
2014-07-21 @ 11:50
Hi,

I have send a pull request to the PHP-ZMQ extension maintainer to add a
class binding to libzyre.
As soon as this feature will be merge, i will add it photon.

William


On Thu, Jun 26, 2014 at 7:54 AM, William MARTIN <wysman@gmail.com> wrote:

> Hi,
>
> zbeacon is more low level. it do not connect peers one to others.
> ZRE is based on zbeacon for the discover part, and other common ZMQ stuff
> for the communication part, like dealer socket, hwm, timeout ...
>
> Thanks,
> William
>
>
> On Thu, Jun 26, 2014 at 6:43 AM, Roger Qiu <roger.qiu@polycademy.com>
> wrote:
>
>> Hey,
>>
>> ZRE sounds pretty interesting. Did you take a look at zbeacon from the
>> CZMQ?
>>
>> Also why did you choose ZRE? There seems to be other kinds of multicast
>> discovery algorithms. Did you try or investigate other ones? I'm curious
>> because I'm trying to figure out a similar problem.
>>
>> Thanks,
>> Roger
>>
>> On 25/06/2014 10:35 PM, Loic d'Anterroches wrote:
>> > Hello,
>> >
>> >
>> >> I am working on projects with 10 to 20 photon workers.
>> >> Deploying multiple instances of the application is a nightmare about
>> >> ports assignment.
>> >>
>> >> First, i have looks to Majordomo protocol to reduce the number of port
>> >> to 1 by application instance.
>> >> Majordomo is a job broker, and can be a single point of failure.
>> >> http://rfc.zeromq.org/spec:18
>> >>
>> >> Then, i find the ZMQ discover protocol named ZRE.
>> >> It's a UDP broadcast protocol to connect worker each to others, and
>> >> transfers message in TCP.
>> >> They have no single point of failure, and workers can go away and come
>> back.
>> >> http://rfc.zeromq.org/spec:36
>> >>
>> >> I will upgrade photon to use ZRE.
>> >>
>> >> Any questions are welcome.
>> > Would it be possible to use it on demand and keep the fixed port
>> > approach available? Not everybody can broadcast on UDP and sometimes,
>> > people are using software one level "upper" to manage all the ports etc.
>> >
>> > loïc
>>
>> --
>> Founder of Polycademy & SnapSearch
>> http://polycademy.com
>> https://snapsearch.io
>> +61420925975
>>
>>
>
>
> --
> ---------------------------------------------------------
> William MARTIN
> wysman @NoSpAm@ gmail @DoT@ com
>



-- 
---------------------------------------------------------
William MARTIN
wysman @NoSpAm@ gmail @DoT@ com