librelist archives

home // archives //

« back to archive

handler design questions

handler design questions

From:
snacktime
Date:
2010-10-06 @ 19:50 
I'm writing a handler in jruby for the json protocol.   I'm looking at
a couple of ways to handle concurrency in the handler, and playing
around with the tools in java.util.concurrent.

So the first strategy which doesn't appear to work that well is to
fire up a pool of threads, and have a separate connection to mongrel2
in each thread.  0mq doesn't seem to like this, it seg faults making
the socket connection.  The docs say that the 0mq context is thread
safe, but the socket connections are not and must live in a single
thread.  So I'm not sure if I'm doing something wrong, it's the jruby
bindings, or it's the 0mq c library.  I'm using a fixed thread pool
and the threads are long lived, each one connects to 0mq and stays
connected forever or until killed.  So if I'm understanding things
correctly, it *should* be working.


Ideally I want to contain the code that needs to run in a thread to a
minumum, so the other approach I'm playing with is to make one socket
connection, run the application specific code in a thread, and use one
of the java.util.concurrent queue facilities to pass the response from
the worker thread back out to the main thread, which sends the
response to mongrel2.   This seems to work well with my simple test
code.

The third way is to just use Thread and write my own thread pool.  My
gut tells me using java.util.concurrent is the better way to go, but
that's just an educated guess nothing more.

I guess it's possible that a j2ee container could work with 0mq
connections instead of http.  But my knowledge of java containers is
rather limited, I'm not even sure if that's on the horizon anywhere.


Thoughts?

Chris

Re: [mongrel2] handler design questions

From:
Zed A. Shaw
Date:
2010-10-07 @ 00:51 
On Wed, Oct 06, 2010 at 12:50:48PM -0700, snacktime wrote:
> So the first strategy which doesn't appear to work that well is to
> fire up a pool of threads, and have a separate connection to mongrel2
> in each thread.

The thing to remember is that most of the 0mq I've done, heard about, or
seen is all process based.  Instead of firing up tons of threads, you'd
just do a bunch of processes and get the same effect.  I think what
screws this up is the insane amount of ram Java uses so where I'd have
no problem doing a bunch of C/C++ process, Java processes would be
difficult to manage.

So, I've had this idea for a while.  Start with a simple sequential
process and see how much you can get from that.  Just keep it dead simple
as a basic loop cranking on requests and sending them back and see
what's possible with just that.  The idea is see what you get in the
simplest case.

Then, put up a small set of processes, maybe 1 per core say.  Now this
gets you a bit more concurrency, and see what you get for a boost from
that.  Again you're keeping it simple, and just seeing what your
baseline is.

Now for the tricky part:  Come up with an internal architecture that
basically replicates this setup inside the JVM using threads.  With the
above simple sequential processes, you have this kind of configuration:

internet -> mongrel2 -> [p1, p2, p3, p4]

Mongrel2's job in this case is to "cook" and constrain all the things it
receives so that the backends are kept sane.  What I'm thinkg you could
do then is this:

mongrel2 -> [p1 -> (master) -> [t1, t2, t3, t4]]

What you've got is a master thread, just one, and it's job is to grab
requests and deal with the bullshit of checking them, figuring out their
routes, etc.  There's always something like this for your apps
internally.  The master's job is then to toss those onto an outgoing
queue, and read responses off an incoming queue.  The threads then just
run in a big loop and read off the queue same as they would of a 0mq.
Only difference is it's guaranteed to work right since it's already a
constrained locked queue.

The other advantage of this is that by spreading the work around a few
processes with threads in them, you have a lower risk of your jvm
exploding and taking the world with it.  In theory it should be way
easier to manage, distribute, and probably works faster too.

> I guess it's possible that a j2ee container could work with 0mq
> connections instead of http.  But my knowledge of java containers is
> rather limited, I'm not even sure if that's on the horizon anywhere.

Yeah, skip the containers unless you're doing web stuff.  It's easy to
use 0mq with django or rails because of WSGI and Rack, but the j2ee
stuff doesn't have the same thing going for it.  It's got the container
side, but not the other side.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] handler design questions

From:
Austin Wise
Date:
2010-10-07 @ 07:04 
> mongrel2 -> [p1 -> (master) -> [t1, t2, t3, t4]]
>
> What you've got is a master thread, just one, and it's job is to grab
> requests and deal with the bullshit of checking them, figuring out their
> routes, etc.  There's always something like this for your apps
> internally.  The master's job is then to toss those onto an outgoing
> queue, and read responses off an incoming queue.  The threads then just
> run in a big loop and read off the queue same as they would of a 0mq.
> Only difference is it's guaranteed to work right since it's already a
> constrained locked queue.


For what it's worth, something like this seems to work well in .NET,
which I'm assuming is similar to the JVM.  ASP.NET for example is fine
with having a single thread read in requests, so I let it access the
socket without any synchronization.  However it then puts every
request on its own thread in the thread pool.  To prevent multiple
threads writing to the response socket, I have response put on a
queue.  There is a dedicate thread reads out of this queue and pushes
the response down the ZMQ socket.

On the JVM you could probably use one of those BlockingQueue<E>s for
the purpose of queuing up responses.

Re: [mongrel2] handler design questions

From:
snacktime
Date:
2010-10-07 @ 18:18 
On Thu, Oct 7, 2010 at 12:04 AM, Austin Wise <austinwise@gmail.com> wrote:
>> mongrel2 -> [p1 -> (master) -> [t1, t2, t3, t4]]
>>
>> What you've got is a master thread, just one, and it's job is to grab
>> requests and deal with the bullshit of checking them, figuring out their
>> routes, etc.  There's always something like this for your apps
>> internally.  The master's job is then to toss those onto an outgoing
>> queue, and read responses off an incoming queue.  The threads then just
>> run in a big loop and read off the queue same as they would of a 0mq.
>> Only difference is it's guaranteed to work right since it's already a
>> constrained locked queue.
>
>
> For what it's worth, something like this seems to work well in .NET,
> which I'm assuming is similar to the JVM.  ASP.NET for example is fine
> with having a single thread read in requests, so I let it access the
> socket without any synchronization.  However it then puts every
> request on its own thread in the thread pool.  To prevent multiple
> threads writing to the response socket, I have response put on a
> queue.  There is a dedicate thread reads out of this queue and pushes
> the response down the ZMQ socket.
>
> On the JVM you could probably use one of those BlockingQueue<E>s for
> the purpose of queuing up responses.
>

So I ended up with something fairly similiar by the time I read all of
this.  The trick now turns out to be the actual implementation.  I've
only used the simpler blocking recv stuff in 0mq, but the main thread
needs to be able to poll 0mq and the queues so I'm playing around with
some approaches to that.

I've also run into an interesting bug where my 0mq context is getting
destroyed while it's still being referenced.  The following thread
shows exactly what I'm running into.

http://comments.gmane.org/gmane.network.zeromq.devel/2228

Chris

Re: [mongrel2] handler design questions

From:
snacktime
Date:
2010-10-07 @ 20:42 
Here is a rough working piece of code for this type of handler.  The
java.util.concurrent tools made this fairly easy to put together.

I added the req_waiting? method to the rack handler.  It polls your
0mq socket and returns true if there is anything ready to read.  Then
I made recv non blocking.

http://dpaste.de/2xoT/

Chris

Re: [mongrel2] handler design questions

From:
Zed A. Shaw
Date:
2010-10-07 @ 23:15 
On Thu, Oct 07, 2010 at 01:42:01PM -0700, snacktime wrote:
> Here is a rough working piece of code for this type of handler.  The
> java.util.concurrent tools made this fairly easy to put together.
> 
> I added the req_waiting? method to the rack handler.  It polls your
> 0mq socket and returns true if there is anything ready to read.  Then
> I made recv non blocking.

Cool, for your shutdown, look at doing a poison pill setup.  It's what
I'm going to with mongrel2 and is much more graceful and easy.
Basically threads know some internal magic message that if they receive
it they exit.  Probably just a setting on an empty request object and
you're good.

Then to shut everyone down, the main thread fills the queue with enough
poison pills for everyone and hangs out until they're all gone.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] handler design questions

From:
Peter Fagerlund
Date:
2010-10-07 @ 21:56 
i dont do java these days so i do not grep the :

> http://dpaste.de/2xoT/
>
altho i asked around in jruby group and head banana says :
"also, note that 0mq sockets are *NOT* thread safe; you must use a socket
from the *same* thread where you created it otherwise there are problems"

k

Re: [mongrel2] handler design questions

From:
snacktime
Date:
2010-10-07 @ 22:15 
On Thu, Oct 7, 2010 at 2:56 PM, Peter Fagerlund <admin@iprobot.com> wrote:
> i dont do java these days so i do not grep the :
>>
>> http://dpaste.de/2xoT/
>
> altho i asked around in jruby group and head banana says :
> "also, note that 0mq sockets are *NOT* thread safe; you must use a socket
> from the *same* thread where you created it otherwise there are problems"
> k

Ya that's true, a socket has to live in the same thread.

I'm working on an implementation now that gets rid of the java queues
and uses 0mq queues for communicating to the worker threads.

Chris

Re: [mongrel2] handler design questions

From:
Zed A. Shaw
Date:
2010-10-07 @ 23:16 
On Thu, Oct 07, 2010 at 03:15:02PM -0700, snacktime wrote:
> I'm working on an implementation now that gets rid of the java queues
> and uses 0mq queues for communicating to the worker threads.

Look at the inproc: stuff.  Basically make a 0mq queue that's
inproc:requests and inproc:responses and then you're done.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] handler design questions

From:
Peter Fagerlund
Date:
2010-10-07 @ 06:59 
How about you use plain ZMQ workers behind your M2Handler ...

<pre><code>

  +--------+  +--------+  +--------+
  | Client |  | Client |  | Client |
  +--------+  +--------+  +--------+
        |            |            |
       +-----------+-----------+
                     |
               +---+----+
               |   M2    |
               +--------+
                     |
               +--------+
              | Handler |
               +--------+
               |  ZMQ  |
               +---+----+
                     |
        +-----------+-----------+
       |              |             |
+---+----+  +---+----+  +---+----+
|  ZMQ  |  |  ZMQ  |  |  ZMQ  |
+--------+  +--------+  +--------+

</code></pre>

Re: [mongrel2] handler design questions

From:
Zed A. Shaw
Date:
2010-10-07 @ 15:29 
On Thu, Oct 07, 2010 at 08:59:19AM +0200, Peter Fagerlund wrote:
> How about you use plain ZMQ workers behind your M2Handler ...
> 

That works but the JVM is fairly massive when it runs, especially once
you put in web framework and ORM code.  The JVM typically solves this by
using threads instead of processes.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] handler design questions

From:
Peter Fagerlund
Date:
2010-10-07 @ 16:17 
yes . JVM is massive ram intense but once up it holds pretty good, also
quite fast with ruby syntax running as bytecode ...

the zmq workers can be c++ zmq's not living in the `container` where the req
is originating from ...

did i understand correct, we here have m2 infront of ruby(ffi?) handlers
executed in jruby ...

On Thu, Oct 7, 2010 at 5:29 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:

> On Thu, Oct 07, 2010 at 08:59:19AM +0200, Peter Fagerlund wrote:
> > How about you use plain ZMQ workers behind your M2Handler ...
> >
>
> That works but the JVM is fairly massive when it runs, especially once
> you put in web framework and ORM code.  The JVM typically solves this by
> using threads instead of processes.
>
> --
> Zed A. Shaw
> http://zedshaw.com/
>

Archives based on code by Armin Ronacher //