librelist archives

Change to ZeroMQ 2.1.x Caused Bug In Mongrel2 Handlers

From:

Zed A. Shaw

Date:

2011-05-26 @ 16:25

Hey everyone, this is a PSA for the latest Mongrel2 release regarding
ZeroMQ 2.1.x.  That release of ZeroMQ changed how identities work in a
way that breaks the ability to fire up as many handler processes as you want
using the same identity.  Now ZeroMQ requires that *each* process has
it's own unique identity or else it won't round-robin anymore.

The fix is, in your handler code (not in your mongrel2 config), use an
None (nil, or don't set it) send_ident so that zeromq gives you a random
one.  Or, work up a scheme where each process gets a slightly different
ident.

The issue with not setting a send_ident though is you don't get durable
messaging going out from the handler.

I'll be looking at how the zeromq sockets are working for 1.7 and
hopefully come up with a way to make this work that doesn't suck.  It
might be that each process has to tack a number on the end of the ident
so that it identifies properly.

Any suggestions on this would be great.

-- 
Zed A. Shaw
http://zedshaw.com/

On May 26, 2011 12:25:30 PM Zed A. Shaw wrote:
> Hey everyone, this is a PSA for the latest Mongrel2 release regarding
> ZeroMQ 2.1.x.  That release of ZeroMQ changed how identities work in a
> way that breaks the ability to fire up as many handler processes as you
> want using the same identity.  Now ZeroMQ requires that *each* process has
> it's own unique identity or else it won't round-robin anymore.
> 
> The fix is, in your handler code (not in your mongrel2 config), use an
> None (nil, or don't set it) send_ident so that zeromq gives you a random
> one.  Or, work up a scheme where each process gets a slightly different
> ident.
> 
> The issue with not setting a send_ident though is you don't get durable
> messaging going out from the handler.
> 
> I'll be looking at how the zeromq sockets are working for 1.7 and
> hopefully come up with a way to make this work that doesn't suck.  It
> might be that each process has to tack a number on the end of the ident
> so that it identifies properly.
> 
> Any suggestions on this would be great.

As far as I can tell, the only place that identity is needed is for the SUB 
socket on the mongrel2 server.

I can't think any reason why they are needed for the PUSH/PULL sockets: 
The PUSH socket (on mongrel2) will queue messages if there are no connected 
PULL sockets (on the application servers).  No messages will be lost in this 
situation (see http://api.zeromq.org/2-1:zmq-socket#toc12)
If an application server dies, then all the messages in its PULL queue will be 
lost.  Having an identity will not change this.  Mongrel2 will simply route 
the messages to other application servers, or queue them if there are none.  
Worse, the PUSH socket might queue messages (based on my interpretation of 
http://api.zeromq.org/2-1:zmq-setsockopt#toc6) for a PULL socket with identity 
of the dead server, instead of routing then to other instances of the 
application server, which could mean that some http queries never get a 
response.
If the mongrel2 server dies, then all the messages in the PUSH queue will be 
lost.  Once again, having an identity will not change this.
Put another way, I can't see how durability helps in this situation.
Similarly, the PUB sockets on the application servers don't need identity.  If 
the application server crashes, then all messages the messages on the PUB 
queue would be lost, and identity does not help.  
Finally, having identity on the SUB socket on mongrel2 might be required. 
Certainly, it is required if an application server is going to work on a 
request while mongrel2 is being restarted, otherwise the application server 
will discard any responses produced before the connection is re-established 
with the mongrel2 server.  But, I don't know what mongrel2 is going to do with 
the response because the connection with the web browser would have been lost 
during the restart.  So, if mongrel2 is going to be discarding the response, 
then why not have the application server drop it, instead of trying to deliver 
it to mongrel2.  This suggests setting identity on the SUB socket might also 
be counter-productive.
In conclusion, if my analysis is correct, identity and durable connections are 
not required, and might even slow things down.  (Not to mention that "the 
whole notion of durable sockets may be dropped in future implementations since 
it's extraordinarily complex in the code, and tends to create brittle 
architectures" according to a message on the zeromq mailing list)

Please let me know of any errors in my analysis.

Cheers,
Henry

-- 
Henry Baragar
Instantiated Software

On May 26, 2011 12:25:30 PM Zed A. Shaw wrote:
> Hey everyone, this is a PSA for the latest Mongrel2 release regarding
> ZeroMQ 2.1.x.  That release of ZeroMQ changed how identities work in a
> way that breaks the ability to fire up as many handler processes as you
> want using the same identity.  Now ZeroMQ requires that *each* process has
> it's own unique identity or else it won't round-robin anymore.
> 
> The fix is, in your handler code (not in your mongrel2 config), use an
> None (nil, or don't set it) send_ident so that zeromq gives you a random
> one.  Or, work up a scheme where each process gets a slightly different
> ident.
> 
> The issue with not setting a send_ident though is you don't get durable
> messaging going out from the handler.
> 
> I'll be looking at how the zeromq sockets are working for 1.7 and
> hopefully come up with a way to make this work that doesn't suck.  It
> might be that each process has to tack a number on the end of the ident
> so that it identifies properly.
> 
> Any suggestions on this would be great.

Based on what I have learned, both from experimenting and from discussions on 
the 0MQ mailing list, I believe that using 0MQ PUSH/PULL is not the right 
approach for mongrel2, even though it looks like streaming is the appropriate 
model for mongrel2.

I now believe that 0MQ REP/REQ sockets are the correct approach for mongrel2, 
where application handlers user REQ sockets to request work from the REP 
sockets on the mongrel2 web server (I know, it sounds backwards).  There are 
numerous advantages to this approach:
the number of messages that could get lost when an application server dies or 
is restarted is limited to the messages actually being worked on
adding an extra application server enables pending messages to be handled 
immediately, rather than waiting for the next incoming message (to mongrel2)
mongrel2 has more control over the queue of outstanding requests (for timeout 
purposes)
Similary, the application server would set up a REP socket to send partially 
processed messages downstream.

I know that this is a major change to the handler protocol, but given that 
mongrel2 needs a queuing solution and not just the message passing solution 
provided by 0MQ, I believe that it probably is the right direction to take.

Regards,
Henry

-- 
Henry Baragar
Instantiated Software

On Sun, Jul 03, 2011 at 03:53:00PM -0400, Henry Baragar wrote:
> I now believe that 0MQ REP/REQ sockets are the correct approach for mongrel2, 
> where application handlers user REQ sockets to request work from the REP 
> sockets on the mongrel2 web server (I know, it sounds backwards).  There are 
> numerous advantages to this approach:

Nope, the 0MQ REP/REQ sockets are broken in my opion for anything but
very simple tight work loads.  The second you send a REQ that never gets
replied to one of the sides gets out of sync and has a screwed up state,
which causes an exception deep in the C++ code, something no server can
handle.

Their answer to this is XREP/XREQ, but then you end up replicating all
the gear that push/pull and pub/sub give you, plus your own tracking,
and then timeouts...so then why bother using zmq at all if you have to
do all that anyway with a socket server.

There's nothing that prevents you from writing a bridge that gives you
req/rep, but in Mongrel2 it's a pain in the ass for very little benefit.

-- 
Zed A. Shaw
http://zedshaw.com/

could not decode message

On Sun, 2011-07-03 at 13:51 -0700, Zed A. Shaw wrote:
> On Sun, Jul 03, 2011 at 03:53:00PM -0400, Henry Baragar wrote:
> > I now believe that 0MQ REP/REQ sockets are the correct approach for mongrel2, 
> > where application handlers user REQ sockets to request work from the REP 
> > sockets on the mongrel2 web server (I know, it sounds backwards).  There are 
> > numerous advantages to this approach:
> 
> Nope, the 0MQ REP/REQ sockets are broken in my opion for anything but
> very simple tight work loads.  The second you send a REQ that never gets
> replied to one of the sides gets out of sync and has a screwed up state,
> which causes an exception deep in the C++ code, something no server can
> handle.

Plus, you get the latency of a round-trip from handler to mongrel2 and
back for each and every request.  Unless you put logic into m2 to send
requests in batches, at which point you might as well just:

> Their answer to this is XREP/XREQ, but then you end up replicating all
> the gear that push/pull and pub/sub give you, plus your own tracking,
> and then timeouts

For example, see my "dispatcher" device in m2wsgi:

   https://github.com/rfk/m2wsgi/blob/master/m2wsgi/device/dispatcher.py

This uses XREQ/XREP to, as Zed said, rebuild a kind of hybrid
PUSH/PULL/PUB/SUB socket.

It's not pretty, but it does avoid some of the disappearing-handler
issues that you can encounter with vanilla PUSH/PULL sockets.

Since you're doing your own handler tracking anyway, you can also build
fun things like sticky session routing.

*Most* of the ugliness in this code is to work around a single
limitation in the 0mq API: there is no way to get a list of all clients
currently connected to a socket.  So instead it has to track handlers as
they appear, handlers have to send explicit connect/disconnect messages,
and it gets ugly pretty fast.

If there was an API call that said "get the list of all idents connected
to this XREP socket" then this code would be a third of the size, much
cleaner, and I would be arguing for the approach as a compelling
replacement for PUSH/PULL sockets in m2 core.

C'est la vie.


> ...so then why bother using zmq at all if you have to
> do all that anyway with a socket server.
> 
> There's nothing that prevents you from writing a bridge that gives you
> req/rep, but in Mongrel2 it's a pain in the ass for very little benefit.

The above code started out as just such a REQ/REP bridge.


  Cheers,

     Ryan


-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan@rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

On Thu, May 26, 2011 at 05:47:35PM -0400, Henry Baragar wrote:
> As far as I can tell, the only place that identity is needed is for the SUB 
> socket on the mongrel2 server.

Well, yes and no.  Here's what I've gleaned from the code and the
manual, and it did change recently.

In the manual it has this sort of ambiguous statement:

http://zguide.zeromq.org/page:all#Transient-vs-Durable-Sockets

"What durable sockets give you is the promise that the ØMQ transmit
buffer is kept alive as long as the sender exists."

That implies that if you set the TRANSMIT side identity then the RECEIVE
side will be durable.  However when you read further down, they don't
really clarify, it's not until you get to:

http://zguide.zeromq.org/page:all#-Semi-Durable-Subscribers-and-High-Water-Marks

And the two code examples that you start to see what's *really*
happening:

* The SUB (receiver) side of the connection *sets the ident*.
* The PUB (transmit) side does *NOT* set it, and instead sets a HWM
  setting to constrain the pending messages.

This makes sense once you realize that the receiver is kind of using the
identity like a session cookie.  It connects to the PUB side and
declares its identity, the PUB side then looks in its list of pending
queues, finds the one for this identity, and sends any pending messages.

Which leads to the better description:

"If a *receiver* (SUB, PULL, REQ) side of a socket sets an identity then
the *sending* (PUB, PUSH, REP) side will buffer messages when they
aren't connected up to the HWM. The *sending* side does not need to set
an identity for this to work."

Ok, now to think up how to redesign this and reword it so that it will
be easier to use and work the way zeromq is now designed.

-- 
Zed A. Shaw
http://zedshaw.com/

On May 26, 2011 07:57:54 PM Zed A. Shaw wrote:
> "If a receiver (SUB, PULL, REQ) side of a socket sets an identity then
> the sending (PUB, PUSH, REP) side will buffer messages when they
> aren't connected up to the HWM. The sending side does not need to set
> an identity for this to work."

Instead of speculating further, I did some experiments this morning with 
PUSH/PULL sockets in 0MQ 2.17.

From my experimentation, it appears that the PUSH socket does no buffering.  
The PUSH socket will block if there are no connected PULL sockets, and it will 
drop messages if ZMQ_NOBLOCK is set and there are no connected PULL sockets.

Setting the identity made absolutely no difference.

Furthermore, setting the HWM for the PULL socket made no difference.  I test 
this using both the Ruby and Perl bindings.

I have posted a couple of questions on the 0MQ mailing list, but have yet to 
get a response.

Regards,
Henry


-- 
Henry Baragar
Instantiated Software

On Fri, May 27, 2011 at 03:58:11PM -0400, Henry Baragar wrote:
> On May 26, 2011 07:57:54 PM Zed A. Shaw wrote:
> > "If a receiver (SUB, PULL, REQ) side of a socket sets an identity then
> > the sending (PUB, PUSH, REP) side will buffer messages when they
> > aren't connected up to the HWM. The sending side does not need to set
> > an identity for this to work."

>From my experimentation, it appears that the PUSH socket does no buffering.  

Haha, man their docs are flat out wrong then.  So, this only applies to
PUB/SUB and REQ/REP? Dumb.  I'll play with this too now that you've
brought it up.

-- 
Zed A. Shaw
http://zedshaw.com/

On Thu, 2011-05-26 at 17:47 -0400, Henry Baragar wrote:
>
> I can't think any reason why they are needed for the PUSH/PULL
> sockets: 
> 
>       * The PUSH socket (on mongrel2) will queue messages if there are
>         no connected PULL sockets (on the application servers). No
>         messages will be lost in this situation (see
>         http://api.zeromq.org/2-1:zmq-socket#toc12)
>       * If an application server dies, then all the messages in its
>         PULL queue will be lost. Having an identity will not change
>         this. Mongrel2 will simply route the messages to other
>         application servers, or queue them if there are none. Worse,
>         the PUSH socket might queue messages (based on my
>         interpretation of
>         http://api.zeromq.org/2-1:zmq-setsockopt#toc6) for a PULL
>         socket with identity of the dead server, instead of routing
>         then to other instances of the application server, which could
>         mean that some http queries never get a response.

I believe this case is what Zed's talking about.  If your handler dies,
the PUSH socket might have queued messages for it that haven't seen sent
yet.

Having a persistent identity certainly does change things.

If it reconnects with the original identity, the PUSH socket will
recognise it and send through the queued messages.  The connected
clients will eventually get a response

If it reconnects with a new identify, those messages will just sit
around inside mongrel2 queued forever.  The connected clients will hang
forever (or until timed out).

To make matters worse, there is no way to cleanly "shut down" a zmq
socket.  Restarting your handler is handled exactly the same as if it
just died unexpectedly - meaning that if it doesn't have a persistent
identify, messages can easily go missing.

(My first message to this group was on exactly this topic, I've done a
*lot* of mucking around with various killing/shutdown sequences).

> Similarly, the PUB sockets on the application servers don't need
> identity. If the application server crashes, then all messages the
> messages on the PUB queue would be lost, and identity does not help. 

Yep.  PUB sockets don't do any queuing.  If the receiving identity is
not connected, the message is just dropped.

> In conclusion, if my analysis is correct, identity and durable
> connections are not required, and might even slow things down. (Not to
> mention that "the whole notion of durable sockets may be dropped in
> future implementations since it's extraordinarily complex in the code,
> and tends to create brittle architectures" according to a message on
> the zeromq mailing list)


They can help in certain cases, but I certainly wouldn't be sad to see
them go.


   Ryan


-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan@rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

On May 26, 2011 07:41:21 PM Ryan Kelly wrote:
> On Thu, 2011-05-26 at 17:47 -0400, Henry Baragar wrote:
> > I can't think any reason why they are needed for the PUSH/PULL
> > 
> > sockets:
> >       * The PUSH socket (on mongrel2) will queue messages if there are
> >       
> >         no connected PULL sockets (on the application servers). No
> >         messages will be lost in this situation (see
> >         http://api.zeromq.org/2-1:zmq-socket#toc12)
> >       
> >       * If an application server dies, then all the messages in its
> >       
> >         PULL queue will be lost. Having an identity will not change
> >         this. Mongrel2 will simply route the messages to other
> >         application servers, or queue them if there are none. Worse,
> >         the PUSH socket might queue messages (based on my
> >         interpretation of
> >         http://api.zeromq.org/2-1:zmq-setsockopt#toc6) for a PULL
> >         socket with identity of the dead server, instead of routing
> >         then to other instances of the application server, which could
> >         mean that some http queries never get a response.
> 
> I believe this case is what Zed's talking about.  If your handler dies,
> the PUSH socket might have queued messages for it that haven't seen sent
> yet.
> 
> Having a persistent identity certainly does change things.
> 
> If it reconnects with the original identity, the PUSH socket will
> recognise it and send through the queued messages.  The connected
> clients will eventually get a response
> 
> If it reconnects with a new identify, those messages will just sit
> around inside mongrel2 queued forever.  The connected clients will hang
> forever (or until timed out).
> 

The way to avoid this is to not use identities, because the messages in the 
PUSH queue will be delivered to the next handler that connects, or to the next 
handler in the pool.  Then, the only clients that hang or time out are the 
ones for which message were on the handler's PULL queue or PUB queue.

> To make matters worse, there is no way to cleanly "shut down" a zmq
> socket.  Restarting your handler is handled exactly the same as if it
> just died unexpectedly - meaning that if it doesn't have a persistent
> identify, messages can easily go missing.
> 

But this is no different than if they don't have a persistent identity!  The 
thing to note is that PUSH queue is different from the PULL queue and that 
once the message is in the PULL queue, it is out of the PUSH queue (and hence 
out of mongrel2).  The same messages go missing in both scenarios.

> (My first message to this group was on exactly this topic, I've done a
> *lot* of mucking around with various killing/shutdown sequences).
> 
> > Similarly, the PUB sockets on the application servers don't need
> > identity. If the application server crashes, then all messages the
> > messages on the PUB queue would be lost, and identity does not help.
> 
> Yep.  PUB sockets don't do any queuing.  If the receiving identity is
> not connected, the message is just dropped.
> 

I think you mean that if the receiving identity has not yet connected.  It 
will queue messages for a receiving identity that has connected but then died, 
which can lead to an out-of-memory situation if the receiving identity is not 
reconnected in time (which is unlikely for mongrel2 because the receving 
identity disappearing means that mongrel2 has disappeared meaning that the 
handler is not receving any new requests).

Henry

> > In conclusion, if my analysis is correct, identity and durable
> > connections are not required, and might even slow things down. (Not to
> > mention that "the whole notion of durable sockets may be dropped in
> > future implementations since it's extraordinarily complex in the code,
> > and tends to create brittle architectures" according to a message on
> > the zeromq mailing list)
> 
> They can help in certain cases, but I certainly wouldn't be sad to see
> them go.
> 
> 
>    Ryan

-- 
Henry Baragar
Instantiated Software

On Thu, 2011-05-26 at 23:14 -0400, Henry Baragar wrote:
> On May 26, 2011 07:41:21 PM Ryan Kelly wrote:

> > > * If an application server dies, then all the messages in its
> > > 
> > > PULL queue will be lost. Having an identity will not change
> > > this. Mongrel2 will simply route the messages to other
> > > application servers, or queue them if there are none. Worse,
> > > the PUSH socket might queue messages (based on my
> > > interpretation of
> > > http://api.zeromq.org/2-1:zmq-setsockopt#toc6) for a PULL
> > > socket with identity of the dead server, instead of routing
> > > then to other instances of the application server, which could
> > > mean that some http queries never get a response.
> > 
> 
> > I believe this case is what Zed's talking about. If your handler
> dies,
> > the PUSH socket might have queued messages for it that haven't seen
> sent yet.
>
> > Having a persistent identity certainly does change things.
>
> > If it reconnects with the original identity, the PUSH socket will
> > recognise it and send through the queued messages. The connected
> > clients will eventually get a response
>
> > If it reconnects with a new identify, those messages will just sit>
> around inside mongrel2 queued forever. The connected clients will hang
> forever (or until timed out).
>
> 
> The way to avoid this is to not use identities, because the messages
> in the PUSH queue will be delivered to the next handler that connects,
> or to the next handler in the pool. Then, the only clients that hang
> or time out are the ones for which message were on the handler's PULL
> queue or PUB queue.

That has not been my experience.

Perhaps it's changed in recent versions, but a few months ago I was
pawing through the PUSH socket source code trying to get a grip on
exactly this issue.

If you don't explicitly set an identity on the socket, then ZMQ will
make one up for you.

The PUSH socket code doesn't distinguish between "sockets with an
identity" and "sockets without an identity", because all sockets have an
identity internally.  It just manages a big mapping from socket ids to
queues of messages, and round-robins requests into these queues.

Think of its internal state as a list like this:

    connected_sockets = [(ID1,QUEUE1),(ID2,QUEUE2),...]

When you send a message to a PUSH socket, it does this:

    sock = connected_sockets.pop(0)
    sock[1].append(message)
    connected_sockets.append(sock)

As each connected socket asks for a new message, it gets one off its
relevant queue.

It's entirely possible for the PUSH socket to assign a message to a dead
PULL socket.  Unless they've added code to deal with this in the latest
version, which would make me very happy :-)


  Ryan

-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan@rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

Zed - is this issue bigger than using a uuid for each handler?

Brubeck does this today:

https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L163 #
generates id
https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L169 #
sets identity on socket to id


On Thu, May 26, 2011 at 12:25 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:

> Hey everyone, this is a PSA for the latest Mongrel2 release regarding
> ZeroMQ 2.1.x.  That release of ZeroMQ changed how identities work in a
> way that breaks the ability to fire up as many handler processes as you
> want
> using the same identity.  Now ZeroMQ requires that *each* process has
> it's own unique identity or else it won't round-robin anymore.
>
> The fix is, in your handler code (not in your mongrel2 config), use an
> None (nil, or don't set it) send_ident so that zeromq gives you a random
> one.  Or, work up a scheme where each process gets a slightly different
> ident.
>
> The issue with not setting a send_ident though is you don't get durable
> messaging going out from the handler.
>
> I'll be looking at how the zeromq sockets are working for 1.7 and
> hopefully come up with a way to make this work that doesn't suck.  It
> might be that each process has to tack a number on the end of the ident
> so that it identifies properly.
>
> Any suggestions on this would be great.
>
> --
> Zed A. Shaw
> http://zedshaw.com/
>

> The fix is, in your handler code (not in your mongrel2 config), use an
> None (nil, or don't set it) send_ident so that zeromq gives you a random
> one.  Or, work up a scheme where each process gets a slightly different
> ident.

None doesn't work with the stock handler.py and neither does an empty
string:

  File "build/bdist.macosx-10.6-universal/egg/mongrel2/handler.py", line 45,
in __init__
  File "socket.pyx", line 234, in zmq.core.socket.Socket.setsockopt
(zmq/core/socket.c:2300)
TypeError: expected str, got: None

File "build/bdist.macosx-10.6-universal/egg/mongrel2/handler.py", line 45,
in __init__
  File "socket.pyx", line 264, in zmq.core.socket.Socket.setsockopt
(zmq/core/socket.c:2618)
zmq.core.error.ZMQError: Invalid argument

That constructor's code's gotta change somewheres, at least to catch and
pass on the exception when it gets raised. 

On Thu, May 26, 2011 at 12:30:03PM -0400, James Dennis wrote:
> Zed - is this issue bigger than using a uuid for each handler?
> 
> Brubeck does this today:
> 
> https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L163 #
> generates id
> https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L169 #
> sets identity on socket to id

That's one way to do it, which will work with mongrel2 restarts since
each process will have a unique identity to work with.  But, if the
process dies then you lose any pending messages for it since it will
make a new uuid.

-- 
Zed A. Shaw
http://zedshaw.com/

Excellent point... thanks.

On Thu, May 26, 2011 at 2:31 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:

> On Thu, May 26, 2011 at 12:30:03PM -0400, James Dennis wrote:
> > Zed - is this issue bigger than using a uuid for each handler?
> >
> > Brubeck does this today:
> >
> > https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L163 #
> > generates id
> > https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L169 #
> > sets identity on socket to id
>
> That's one way to do it, which will work with mongrel2 restarts since
> each process will have a unique identity to work with.  But, if the
> process dies then you lose any pending messages for it since it will
> make a new uuid.
>
> --
> Zed A. Shaw
> http://zedshaw.com/
>

2011/5/26 Zed A. Shaw <zedshaw@zedshaw.com>:
> On Thu, May 26, 2011 at 12:30:03PM -0400, James Dennis wrote:
>> Zed - is this issue bigger than using a uuid for each handler?
>>
>> Brubeck does this today:
>>
>> https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L163 #
>> generates id
>> https://github.com/j2labs/brubeck/blob/master/brubeck/mongrel2.py#L169 #
>> sets identity on socket to id
>
> That's one way to do it, which will work with mongrel2 restarts since
> each process will have a unique identity to work with.  But, if the
> process dies then you lose any pending messages for it since it will
> make a new uuid.

I'm planning this for wsgid, but it will be smart and reuse uuids when
re-spawning a dead worker.
But if you kill the master proccess and start a new one, you will lose messages.


-- 
Dalton Barreto
http://daltonmatos.wordpress.com
http://wsgid.com