librelist archives

« back to archive

RFC: First cut of Timeout/Clearout code

RFC: First cut of Timeout/Clearout code

From:
Zed A. Shaw
Date:
2011-05-03 @ 21:00
I have a rough implementation of the timeout/clearout mechanism in
Mongrel2.  This also found a bug that was probably leaking out handler
connections that failed, so double win.

Right now the timeout and clearout are nasty and aggressive.  Basically,
if Mongrel2 can't accept, it tries to force any connection that has not
pinged or hasn't written/read data recently to die.  Then ever 5 seconds
or so tries to do it again just as aggressively.

DO NOT RUN TRUNK IN PRODUCTION RIGHT NOW.

It's like this so that I can thrash the server and see how it handles
this aggressive killing and timeout.  It works mostly, but I'll need to
fine tune it some more and clean it up.

Alright, now for the feature question:

1. How should this be configured?  I'm thinking settings maps like with
other stuff, and a setting for ping_timeout, min_read_rate,
min_write_rate, and then a kill_weight which would be a number 1-3 for
how many of the three it hits before killing it.  Like, you could set
kill_weight = 3 and it'll only kill sockets that haven't been pinged,
have a min_read_rate, and write wait.  kill_weight = 1 would kill it if
any of those go off.

2. Is there another metric to use?

3. Should it be different for "overloaded" vs. "timeout"?

Thanks, let me know what y'all think about the above.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] RFC: First cut of Timeout/Clearout code

From:
Hedge Hog
Date:
2011-05-03 @ 22:01
On Wed, May 4, 2011 at 7:00 AM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> I have a rough implementation of the timeout/clearout mechanism in
> Mongrel2.  This also found a bug that was probably leaking out handler
> connections that failed, so double win.
>
> Right now the timeout and clearout are nasty and aggressive.  Basically,
> if Mongrel2 can't accept, it tries to force any connection that has not
> pinged or hasn't written/read data recently to die.  Then ever 5 seconds
> or so tries to do it again just as aggressively.
>
> DO NOT RUN TRUNK IN PRODUCTION RIGHT NOW.
>
> It's like this so that I can thrash the server and see how it handles
> this aggressive killing and timeout.  It works mostly, but I'll need to
> fine tune it some more and clean it up.
>
> Alright, now for the feature question:
>
> 1. How should this be configured?  I'm thinking settings maps like with
> other stuff, and a setting for ping_timeout, min_read_rate,
> min_write_rate, and then a kill_weight which would be a number 1-3 for
> how many of the three it hits before killing it.  Like, you could set
> kill_weight = 3 and it'll only kill sockets that haven't been pinged,
> have a min_read_rate, and write wait.  kill_weight = 1 would kill it if
> any of those go off.
>
> 2. Is there another metric to use?

kill_recipe='prw'
so you can kill_recipe = 'p'
If this appeals, should  'rw' and 'wr' be regarded as different?

HTH


>
> 3. Should it be different for "overloaded" vs. "timeout"?
>
> Thanks, let me know what y'all think about the above.
>
> --
> Zed A. Shaw
> http://zedshaw.com/
>



-- 
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

Re: [mongrel2] RFC: First cut of Timeout/Clearout code

From:
Zed A. Shaw
Date:
2011-05-04 @ 01:09
On Wed, May 04, 2011 at 08:01:12AM +1000, Hedge Hog wrote:
> > 2. Is there another metric to use?
> 
> kill_recipe='prw'
> so you can kill_recipe = 'p'
> If this appeals, should  'rw' and 'wr' be regarded as different?

Hmmm, a string of options could work.  I'll try them out.  What would be
the expected default?  All three before a kill, so 'prw'?

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] RFC: First cut of Timeout/Clearout code

From:
Hedge Hog
Date:
2011-05-04 @ 01:19
On Wed, May 4, 2011 at 11:09 AM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> On Wed, May 04, 2011 at 08:01:12AM +1000, Hedge Hog wrote:
>> > 2. Is there another metric to use?
>>
>> kill_recipe='prw'
>> so you can kill_recipe = 'p'
>> If this appeals, should  'rw' and 'wr' be regarded as different?
>
> Hmmm, a string of options could work.  I'll try them out.  What would be
> the expected default?  All three before a kill, so 'prw'?

Yeah someone could get annoyed if mongrel2 was too sensitive out the box.
To my mind the real advantage of this is that it is suggestive, and
'pw' is more likely to trigger your memory than 2.
Not sure what the performance hit is.

TIA

>
> --
> Zed A. Shaw
> http://zedshaw.com/
>



-- 
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com