librelist archives

Re: [mongrel2] Disconnects from mongrel.

From:

Anders Fugmann

Date:

2011-07-01 @ 18:52

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Thanks for the reply. You are correct in that the service receives and
responds to an "old" request. Let my describe simple steps to reproduce:

1. Start mongrel.
2. use curl to do a http request, but kill curl before a reply is received.
3. Start the service.

The service now receives the request and sends off a response. Now
mongrel starts sending disconnect messages, while logging

[ERROR] (src/register.c:214: errno: Resource temporarily unavailable)
Nothing registered under id 0.
[ERROR] (src/register.c:199: errno: None) Invalid FD given for exists check

in an endless stream. The service does not receive any request, nor does
it send any replies (to the disconnects).

So the message flow is:

mongrel2 <-> service
- -> request (old)
<- reply
- -> disconnect
- -> disconnect
- -> disconnect
- -> disconnect
...

Regards
Anders Fugmann


On 2011-07-01 02:54, Zed A. Shaw wrote:
> On Thu, Jun 30, 2011 at 09:34:19AM +0200, Anders Fugmann wrote:
>> Hi,
>>
>> Im using mongrel in a straight forward way to relay http request to my 
>> backend servers. However, sometimes the system sometimes enters an 
>> infinite loop, where mongrel keeps sending disconnect messages to the 
>> backend service.
>> ...
>> It seems to happen if the client has received a timeout as my backend 
>> service has died. When the backend service restarts it begins to receive 
>> these disconnects. Im not streaming data.
>>
> 
> Hmm, interesting, so you're saying this happens:
> 
> 1. Mongrel2 is serving requests just fine.
> 2. Backend handler goes down.
> 3. Mongrel2 is still receiving and queuing requests.
> 4. Mongrel2 times out the connection to the browser.
> 4. Backend handler comes up.
> 5. Mongrel2 sends a ton of disconnect messages.
> 
> Something like that?  Can you log, when the handler starts, that you
> aren't actually receiving an old previously queued message and trying to
> respond to it?  What can happen is, your handler goes down, but mongrel2
> still puts messages into zeromq.  When your handler comes back online,
> it then gets all of these messages that it "missed", but the browser is
> now gone.
> 
> Log when you get a regular message the disconnect.  If, after a restart,
> all you get are disconnects then I'll have to figure out what's going
> on.  If you get a request/reply/disconnect combination, then those are
> stale requests that your handler hasn't dealt with.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4OF1MACgkQ9GexfkaZ7hRlfACePIkuA6u1HCH7853EbpiACeZQ
+UoAmgJgSmjrVFaqniPTUirOa3R8r4ns
=OhFV
-----END PGP SIGNATURE-----

On Fri, Jul 01, 2011 at 08:52:03PM +0200, Anders Fugmann wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi,
> 
> Thanks for the reply. You are correct in that the service receives and
> responds to an "old" request. Let my describe simple steps to reproduce:

Alright, I think I know what's up, but can you do this for me:

1. make clean dev && sudo make install
2. rm logs/error.log
3. run your test.
4. email me the logs/error.log file.

With that I can fix it this weekend.

-- 
Zed A. Shaw
http://zedshaw.com/

Hi Zed,

Sorry for the late reply, but I have been out on vacation.

I have followed you instructions and recreated the problem with the 
'make dev' version of mongrel 1.7.5. However mongrel2 did not produce an 
error log (even though it was configured to do so).

I piped stdout/stderr to a file, and attached it (console.log), and 
access.log

If you need more, please do not hesitate to write.

I also tried the develop branch, and it exhibits the same behavior.

Regards
Anders


On 07/02/2011 12:01 AM, Zed A. Shaw wrote:
> On Fri, Jul 01, 2011 at 08:52:03PM +0200, Anders Fugmann wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi,
>>
>> Thanks for the reply. You are correct in that the service receives and
>> responds to an "old" request. Let my describe simple steps to reproduce:
>
> Alright, I think I know what's up, but can you do this for me:
>
> 1. make clean dev&&  sudo make install
> 2. rm logs/error.log
> 3. run your test.
> 4. email me the logs/error.log file.
>
> With that I can fix it this weekend.
>

On Mon, Aug 01, 2011 at 08:59:38AM +0200, Anders Fugmann wrote:
> Hi Zed,
> 
> Sorry for the late reply, but I have been out on vacation.

Hey, no problem.  It looks like you have a handler code that's not
dealing with disconnect messages.  Can you give me more information on
what language you're using, how you're handling requests, etc.?

Also, look at examples/http_0mq/http.py and specifically how it's doing
is_disconnect() to avoid sending a reply to that message.

I'll be looking at how this is documented and how to let people just
turn it off in the future.

-- 
Zed A. Shaw
http://zedshaw.com/

Hi Zed,

You are completly correct. I thouhg I had handled this case, but an 
error in our request parsing ocaml module resulted in sending back an 
error reply to mongrel when it received a disconnect, rather than 
ignoring the disconnect as expected.

Code has been fixed, and needless to say everything works as it should.

Many thanks
Anders


On 08/03/2011 02:10 AM, Zed A. Shaw wrote:
> On Mon, Aug 01, 2011 at 08:59:38AM +0200, Anders Fugmann wrote:
>> Hi Zed,
>>
>> Sorry for the late reply, but I have been out on vacation.
>
> Hey, no problem.  It looks like you have a handler code that's not
> dealing with disconnect messages.  Can you give me more information on
> what language you're using, how you're handling requests, etc.?
>
> Also, look at examples/http_0mq/http.py and specifically how it's doing
> is_disconnect() to avoid sending a reply to that message.
>
> I'll be looking at how this is documented and how to let people just
> turn it off in the future.
>