librelist archives

home // archives //

« back to archive

Subtle but deadly ZMQ 2.1.7 compatibility issue

Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Nathan Duran
Date:
2011-05-26 @ 06:54 
The M2 Python egg does this:

        resp = CTX.socket(zmq.PUB)
        resp.setsockopt(zmq.IDENTITY, sender_id)
        resp.connect(pub_addr)

It's always worked, and still does work... Once...

If you launch multiple instances of the same handler process under the
current ZMQ, they all connect to M2, get their messages, and send their
responses, but M2 will only receive the responses of the FIRST process that
connected. Every other request will timeout until it round-robins back to
that first one. No exceptions are raised, no errors are logged, the
responses just disappear, leaving the browser/client waiting.

Simply setting the socket identity AFTER the connection is established
solves this problem, like so:

        resp = CTX.socket(zmq.PUB)
        resp.connect(pub_addr)
        resp.setsockopt(zmq.IDENTITY, sender_id)

I don't know if this is a bug in the Python bindings or ZMQ itself, but it
was quite confounding, and I don't think backward compatibility would be
sacrificed by reordering these operations permanently.

Anyone else seen this or am I going crazy?

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Zed A. Shaw
Date:
2011-05-26 @ 15:42 
On Wed, May 25, 2011 at 11:54:33PM -0700, Nathan Duran wrote:
> Simply setting the socket identity AFTER the connection is established
> solves this problem, like so:
> 
>         resp = CTX.socket(zmq.PUB)
>         resp.connect(pub_addr)
>         resp.setsockopt(zmq.IDENTITY, sender_id)
> 
> I don't know if this is a bug in the Python bindings or ZMQ itself, but it
> was quite confounding, and I don't think backward compatibility would be
> sacrificed by reordering these operations permanently.

WTF, I distinctly remember having to go through *all* my code and do the
exact inverse because they said it had to go in that order (set ident,
then connect).

You got a repro process you do for this so I can make sure it's fixed?

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Zed A. Shaw
Date:
2011-05-26 @ 16:10 
On Wed, May 25, 2011 at 11:54:33PM -0700, Nathan Duran wrote:
> The M2 Python egg does this:
> 
>         resp = CTX.socket(zmq.PUB)
>         resp.setsockopt(zmq.IDENTITY, sender_id)
>         resp.connect(pub_addr)
> 
> It's always worked, and still does work... Once...

Alright, just confirmed this with sustrik in #zeromq.  They "fixed" a
few zeromq bugs related to identity that we just happened to be relying
on.  What they did is they changed the semantics of identity from:

"Identity XXXX is for this application (which can have N processes)."

To the more exact:

"Identity XXX is for process 1, YYY is for process 2, JJJ is for ..."

and then started enforcing them.  Effectively, for you to run 10
processes for the same handler, you'd need to give each one a different
identity.

The solution is to just not set the identity and then zeromq will pick a
random identity, but that means you don't get durability (because they
conflated having identity with durability).

I'll send another email telling people how to fix this until we come up
with a solution, but you basically don't need to change the python
handler code, just in the handler don't set a send_ident or recv_ident
and it will at least work (but will lose messages if a handler goes
down).


-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Javier Guerra Giraldez
Date:
2011-05-26 @ 19:50 
On Thu, May 26, 2011 at 11:10 AM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> Alright, just confirmed this with sustrik in #zeromq.  They "fixed" a
> few zeromq bugs related to identity that we just happened to be relying
> on.  What they did is they changed the semantics of identity from:

i think the issue i raised on hiredis (about not being able to start
two instances of a handler) is caused by this.  I'll check tonight the
ZeroMQ version i'm using and try not setting port identity

-- 
Javier

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Javier Guerra Giraldez
Date:
2011-05-27 @ 02:38 
On Thu, May 26, 2011 at 2:50 PM, Javier Guerra Giraldez
<javier@guerrag.com> wrote:
> On Thu, May 26, 2011 at 11:10 AM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
>> Alright, just confirmed this with sustrik in #zeromq.  They "fixed" a
>> few zeromq bugs related to identity that we just happened to be relying
>> on.  What they did is they changed the semantics of identity from:
>
> i think the issue i raised on hiredis (about not being able to start
> two instances of a handler) is caused by this.  I'll check tonight the
> ZeroMQ version i'm using and try not setting port identity


.... and of course, where i wrote hiredis i meant derdewey's mongrel2_c_handler

i've been using ZeroMQ 2.0.11, and my simptoms aren't exactly the
same: when set the identity, starting a second handler kills mongrel2
with an assertion fault in libzmq with the message "Assertion failed:
!engine (session.cpp:288)".  sounds related, no?




-- 
Javier

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Zed A. Shaw
Date:
2011-05-27 @ 04:59 
On Thu, May 26, 2011 at 09:38:32PM -0500, Javier Guerra Giraldez wrote:
> .... and of course, where i wrote hiredis i meant derdewey's mongrel2_c_handler
> 
> i've been using ZeroMQ 2.0.11, and my simptoms aren't exactly the
> same: when set the identity, starting a second handler kills mongrel2
> with an assertion fault in libzmq with the message "Assertion failed:
> !engine (session.cpp:288)".  sounds related, no?
> 

No I don't think that's related.  That sounds like the problem we had
with ZeroMQ 2.1.6.  Can you double check the version and make sure that
you've removed *all* other version plus done ldconfig.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Javier Guerra Giraldez
Date:
2011-05-27 @ 05:19 
On Thu, May 26, 2011 at 11:59 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> No I don't think that's related.  That sounds like the problem we had
> with ZeroMQ 2.1.6.  Can you double check the version and make sure that
> you've removed *all* other version plus done ldconfig.

i'm positive about version.  it's the only zeromq i've installed on
this machine, and the only libzmq.so so be seen anywhere on my drive.
it's the one reported by ldconfig -p, and matches a freshly compiled
library from source.

-- 
Javier

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Zed A. Shaw
Date:
2011-05-27 @ 05:54 
On Fri, May 27, 2011 at 12:19:11AM -0500, Javier Guerra Giraldez wrote:
> On Thu, May 26, 2011 at 11:59 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> > No I don't think that's related.  That sounds like the problem we had
> > with ZeroMQ 2.1.6.  Can you double check the version and make sure that
> > you've removed *all* other version plus done ldconfig.
> 
> i'm positive about version.  it's the only zeromq i've installed on
> this machine, and the only libzmq.so so be seen anywhere on my drive.
> it's the one reported by ldconfig -p, and matches a freshly compiled
> library from source.

Well I haven't used the mongrel2_c_handler, but first thing you should
do is try the python handler in examples/python/ and make sure that at
least that works.  If it does then it's something the C handler is doing
wrong.  If it doesn't then it's something about your particular setup
that is wrong since the Python works for me and others.

Here's how you test it:

cd examples/python/
sudo python setup.py install
cd ../http_0mq/
cp ../examples/mongrel2.conf .
m2sh config
m2sh start -every

Now in another window, go into examples/http_0mq again and run:

python http.py

Finally, in yet another window, do:

curl http://localhost:6767/handlertest

Try to get that working and then report back.  Also, try upgrading to
2.1.7 and see if that fails.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Ian Barber
Date:
2011-05-26 @ 14:06 
On Thu, May 26, 2011 at 7:54 AM, Nathan Duran <principal@khiltd.com> wrote:

>
> I don't know if this is a bug in the Python bindings or ZMQ itself, but it
> was quite confounding, and I don't think backward compatibility would be
> sacrificed by reordering these operations permanently.
>
> Anyone else seen this or am I going crazy?
>
>
This looks a little odd - the identity is sent on connect, so in the second
case you're not actually setting an identity at all - you're using an
automatically generated one. Any situation where you have multiple sockets
with the same identity is going to go wrong, but setting different
identities should work fine.

Ian

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Nathan Duran
Date:
2011-05-26 @ 21:28 
> Alright, just confirmed this with sustrik in #zeromq.  They "fixed" a
> few zeromq bugs related to identity that we just happened to be relying
> on.  

Gotta love those bug fixes! Thanks for the confirmation. I'll just drop the
IDs from my constructors for now.

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Nathan Duran
Date:
2011-05-26 @ 14:42 
> This looks a little odd - the identity is sent on connect, so in the second
> case you're not actually setting an identity at all - you're using an
> automatically generated one. Any situation where you have multiple sockets
> with the same identity is going to go wrong, but setting different identities
> should work fine.

Maybe I'm goofing up something incredibly basic then--how can I run multiple
instances of each type of handler I use with its own identity?

The "Handler" directive only accepts a single value for send_ident, and the
existing code used to work perfectly fine with 20, 30 or 40 instances of the
handler processes recycling the same ID. Now, only the first instance to
connect works.

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Ian Barber
Date:
2011-05-26 @ 14:51 
On Thu, May 26, 2011 at 3:42 PM, Nathan Duran <principal@khiltd.com> wrote:

>
> The "Handler" directive only accepts a single value for send_ident, and the
> existing code used to work perfectly fine with 20, 30 or 40 instances of
> the
> handler processes recycling the same ID. Now, only the first instance to
> connect works
>

That does sound like the binding is doing the wrong thing - perhaps try
passing null/just not calling send_ident? In absence of a specific Identity,
ZeroMQ will generate a random one, which will work absolutely fine unless
you need durable sockets (which I wouldn't imagine you do in this case -
essentially if the app is restarted it will appear to mongrel 2 as a new
handler). Else can you generate a different identity for each call? I
haven't used the python interface, so apologies if these suggestions are
dumb!

Ian

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Zed A. Shaw
Date:
2011-05-26 @ 15:44 
On Thu, May 26, 2011 at 07:42:14AM -0700, Nathan Duran wrote:
> > This looks a little odd - the identity is sent on connect, so in the second
> > case you're not actually setting an identity at all - you're using an
> > automatically generated one. Any situation where you have multiple sockets
> > with the same identity is going to go wrong, but setting different identities
> > should work fine.
> 
> Maybe I'm goofing up something incredibly basic then--how can I run multiple
> instances of each type of handler I use with its own identity?
> 
> The "Handler" directive only accepts a single value for send_ident, and the
> existing code used to work perfectly fine with 20, 30 or 40 instances of the
> handler processes recycling the same ID. Now, only the first instance to
> connect works. 

Hmm ok ignore my last email, it might be you're doing it wrong.  You
should be able to fire up many instances of a handler all with the same
ident.   Can you shoot me the handler config and how you're setting up
the multiple instances so I can see what you're doing?

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Nathan Duran
Date:
2011-05-26 @ 15:47 
> WTF, I distinctly remember having to go through *all* my code and do the
> exact inverse because they said it had to go in that order (set ident,
> then connect).
> 
> You got a repro process you do for this so I can make sure it's fixed?

I'm seeing the same thing in all their examples, too, so it may well be a
flaw in my understanding or a flat out bug in their library.

I can reproduce the problem anywhere with our source base, but I can't give
out that code. I'll rip out everything private and get something everyone
can see together later today.

Re: [mongrel2] Subtle but deadly ZMQ 2.1.7 compatibility issue

From:
Nathan Duran
Date:
2011-05-26 @ 15:13 
> 
> That does sound like the binding is doing the wrong thing - perhaps try
> passing null/just not calling send_ident? In absence of a specific Identity,
> ZeroMQ will generate a random one, which will work absolutely fine unless you
> need durable sockets (which I wouldn't imagine you do in this case -
> essentially if the app is restarted it will appear to mongrel 2 as a new
> handler). Else can you generate a different identity for each call? I haven't
> used the python interface, so apologies if these suggestions are dumb!

I'm not sure it's the Python interface, but it's very possible that I've
cocked something up when putting together my initial config a very long time
ago and just assumed that it had been working correctly.

What I'm doing is very much akin to what the documentation says I should be
doing by declaring my handlers as such:

auth_handler = Handler(  send_spec="tcp://*:9999",
send_ident="A4D8F847-B50D-41A9-8F11-A33B3D7D3052",
                                recv_spec="tcp://*:9998",
                                recv_ident="")

What I get from my reading of the docs is that I should be specifying a
recv_ident if I want the durable sockets, but that the send_ident is
expected regardless.

I am NOT providing a recv_ident to any handler declaration, and I'm
reasonably certain that ""if you want to fire up 10 more handlers, you just
start them" means that I *should* be able to do this somehow.

Archives based on code by Armin Ronacher //