librelist archives

CPU load of inactive workers

From:

Matthias Wächter

Date:

2011-10-14 @ 07:57

Hi!

I recently hit Mongrel2, now I’m trying some simple tests with it. I’m 
currently going for Ruby.

I wrote a small little Rack app that does nothing more than just sending 
back a "hello world" and 
simple statistics: Timestamp of Rack start, Current Time and number of 
calls. I connected that via 
two IPC sockets with mongrel2 with no special features, a simple 
straight-forward configuration. I 
started two rackup workers to see them process my requests on a cyclic 
basis. Everything works.

Now I am looking at performance. Not in hits/second but in terms of CPU 
usage under low load.

I started the processes about 12 days ago, in that time the two rackup 
processes were hit by a total 
of less than 500 requests: Every now and then I hit reload in my browser a
couple of times to see 
whether the two rackups still run and return statistics properly.

Within these past 12 days, mongrel2 ate about 7 seconds CPU time. Fine. 
But in the same time, the 
two rackup processes ate about 6 minutes CPU each! Watching them on `top` 
without accessing the site 
I can see their CPU usage time grow and grow, every few seconds their 
numbers rise.

Before digging deeper – Ruby, Rack, rack-mongrel2, mongrel2, zeromq – let 
me ask here: is this CPU 
load of inactive workers an inherent effect of the use of zeromq in 
mongrel2? Or is it related to 
the IPC sockets I’m using instead of the TCP sockets? Would this load be 
the same for, say, a worker 
written in Python or C(++)?

– Matthias

The most common cause of this is not  ignoring messages about
disconnected clients.  If you send to a disconnected client, you get a
special message from mongrel2 letting you know it's not there anymore.

If you try to respond to that message as if it were a regular request,
you get into this loop:

Send To closed connection ID>-> Mongrel 2 responds with disconect>-+
            ^------------------------------------------------------+

I don't know about the rack-mongrel2 bindings, but with the default
python bindings, the request object has a method "is_disconnect" that if
it is true, you MUST not respond to the message.

-Jason

On 09:57 Fri 14 Oct     , Matthias Wächter wrote:
> Hi!
> 
> I recently hit Mongrel2, now I’m trying some simple tests with it. I’m 
currently going for Ruby.
> 
> I wrote a small little Rack app that does nothing more than just sending
back a "hello world" and 
> simple statistics: Timestamp of Rack start, Current Time and number of 
calls. I connected that via 
> two IPC sockets with mongrel2 with no special features, a simple 
straight-forward configuration. I 
> started two rackup workers to see them process my requests on a cyclic 
basis. Everything works.
> 
> Now I am looking at performance. Not in hits/second but in terms of CPU 
usage under low load.
> 
> I started the processes about 12 days ago, in that time the two rackup 
processes were hit by a total 
> of less than 500 requests: Every now and then I hit reload in my browser
a couple of times to see 
> whether the two rackups still run and return statistics properly.
> 
> Within these past 12 days, mongrel2 ate about 7 seconds CPU time. Fine. 
But in the same time, the 
> two rackup processes ate about 6 minutes CPU each! Watching them on 
`top` without accessing the site 
> I can see their CPU usage time grow and grow, every few seconds their 
numbers rise.
> 
> Before digging deeper – Ruby, Rack, rack-mongrel2, mongrel2, zeromq – 
let me ask here: is this CPU 
> load of inactive workers an inherent effect of the use of zeromq in 
mongrel2? Or is it related to 
> the IPC sockets I’m using instead of the TCP sockets? Would this load be
the same for, say, a worker 
> written in Python or C(++)?
> 
> – Matthias

On Oct 14, 2011, at 12:57 AM, Matthias Wächter wrote:
> 
> Within these past 12 days, mongrel2 ate about 7 seconds CPU time. Fine. 
But in the same time, the 
> two rackup processes ate about 6 minutes CPU each! Watching them on 
`top` without accessing the site 
> I can see their CPU usage time grow and grow, every few seconds their 
numbers rise.
> 
> Before digging deeper – Ruby, Rack, rack-mongrel2, mongrel2, zeromq – 
let me ask here: is this CPU 
> load of inactive workers an inherent effect of the use of zeromq in 
mongrel2? Or is it related to 
> the IPC sockets I’m using instead of the TCP sockets? Would this load be
the same for, say, a worker 
> written in Python or C(++)?

I don't think there's much chance of Mongrel2 causing this at all. I never
bothered profiling any processes running under it, but the last time I 
tried using it I found the rack-mongrel2 gem to be buggy and inefficient 
enough to fork my own and I never saw the behavior you're describing. 

Should be easy enough to copy and paste one of the Python handler examples
into a file and see what happens there.