librelist archives

Idea for bulk transfers to/from handlers.

From:

Robert G. Jakabosky

Date:

2011-08-16 @ 01:02

After reading the thread about problems with large transfer between mongrel2 
and backend handler, I had an Idea for how to allow streaming of data between 
the backend handler and the client's socket.

In addition to the current PUSH/PULL & PUB/SUB sockets, handlers can use a 
XREQ/XREP sockets to handle bulk transfers.  Mongrel2 would bind a XREP socket 
to a third port.  Most handlers can continue to use the current sockets to 
handle requests, so no change to them.  New handlers that want to control the 
flow of data to/from the client socket can do so by sending requests to 
mongrel2 over a XREQ socket.

For large uploads from the client socket, mongrel2's config would mark a 
handler as wanting to control the transfer of the request body over the XREQ 
socket.  Mongrel2 would only send the http headers + connection id like it 
does now, but without the request body.  The handler would then send a request 
for a chunk of data to be read from the client socket, which mongrel2 would 
either try to read the whole requested chunk size or just respond with what 
was available (i.e. a partial read).  The handler would keep sending request 
until the whole request body was transferred.

For streaming data out to the client socket, the handler would send chunks of 
data in requests over the XREQ socket, mongrel2 would send back a response 
after it finished writing the chunk to the socket.  The handler can still 
close a client connection like it does now, if sending a chunk takes to long 
(maybe there can be a max. timeout value in the config, to automate the dis-
connect of slow clients).  This method might not work well for sending the 
same chunk out to many connections.

XREQ/XREP sockets would need to be used to allow multiple concurrent transfer.  
Also each mongrel2/handler instance would need to use a unique identity on so 
request/responses can be routed correctly.  Mongrel2 can re-use the same uuid 
it currently sends with each request it sends to the handler over the 
PUSH/PULL sockets.  Doing this should still allow many-to-many, many-to-one, 
one-to-many layouts of mongrel2 to backend handlers.

To help with the latency of each request/response pair, mongrel2 can buffer X 
Kbytes for each socket (i.e. send a response back to the handler when X Kbytes 
or less are in the buffer), so the handler has time to prepare the next chunk 
for sending.  I think this buffering in mongrel2 would only be needed for 
transfers that want to maintain a smooth constant transfer speed to the 
client.

For bi-directional connections (i.e. Websocket/flash/etc...) the backend 
handlers should be allowed to issue outstanding read request and still send 
write requests to mongrel2.

Basically this is like an async. RPC interface to the client socket.

-- 
Robert G. Jakabosky

On Mon, Aug 15, 2011 at 06:02:19PM -0700, Robert G. Jakabosky wrote:
> After reading the thread about problems with large transfer between mongrel2 
> and backend handler, I had an Idea for how to allow streaming of data between 
> the backend handler and the client's socket.

You could probably prototype this using the existing mongrel2 gear.
Wanna give it try?  What you'd do is this:

1. Write an upload handler that takes the upload requests and then
starts watching the file that's landing in your tmp_dir.
2. This handler also creates an XREQ/XREP socket like you describe.
3. Use the zmq_poll (it's a little different in Python) or a couple
threads to coordinate between the mongrel2 socket and the XREP socket.
4. Then, implement the API that'd be used to do this.

I think if can make it work then I can look at what it'd take to put it
in or do a similar little helper.

-- 
Zed A. Shaw
http://zedshaw.com/

On Tuesday 16, Zed A. Shaw wrote:
> On Mon, Aug 15, 2011 at 06:02:19PM -0700, Robert G. Jakabosky wrote:
> > After reading the thread about problems with large transfer between
> > mongrel2 and backend handler, I had an Idea for how to allow streaming
> > of data between the backend handler and the client's socket.
> 
> You could probably prototype this using the existing mongrel2 gear.
> Wanna give it try?  What you'd do is this:

Sorry about the long delay in responding, I had started working on this last 
week but then got interrupted with more important things.  I have made a 
prototype [1] written in Lua.  The prototype works as a gateway between 
mongrel2 with the current style handler interface and a backend handler that 
can use the proposed stream interface.

For large http file upload the gateway will allow a backend handler to send 
READ requests (over a XREQ socket) to get chunks of the uploaded file.  This 
shows how the stream interface can be used to allow the backend handler to 
control the reading of data from the client socket in mongrel2.  Right now it 
only handles large file uploads.  This feature could be used right now by 
handlers that need to handle file upload and run on a remote system from 
mongrel2, with the downside that the file must still be buffered to disk on 
the mongrel2 server.

The gateway also implements WRITE requests, but currently there is no flow-
control, the data is just re-formatted into a PUB message and sent to 
mongrel2.  I don't think this feature can be efficiently implemented in a 
handler.

Right now I don't have any use-case that need this streaming interface.  The 
main reason I sent the original e-mail is to get the idea out there for others 
who might need it and to get feedback on it.

Also note that the project [1] includes an async. handler interface for Lua 
that can be used with/without the streaming interface.  See the example [2] 
handler for how the async interface works.  The event loop uses zmq.poller, 
which can be used to listen for events on normal sockets or zmq sockets.  I 
will most likely move this code into it's own project later.

1. https://github.com/Neopallium/mongrel2_stream_prototype
2. 

https://github.com/Neopallium/mongrel2_stream_prototype/blob/master/handlers/async_test.lua

-- 
Robert G. Jakabosky

On Mon, Aug 15, 2011 at 9:02 PM, Robert G. Jakabosky
<bobby@sharedrealm.com> wrote:
> After reading the thread about problems with large transfer between mongrel2
> and backend handler, I had an Idea for how to allow streaming of data between
> the backend handler and the client's socket.
>
> In addition to the current PUSH/PULL & PUB/SUB sockets, handlers can use a
> XREQ/XREP sockets to handle bulk transfers.  Mongrel2 would bind a XREP socket
> to a third port.  Most handlers can continue to use the current sockets to
> handle requests, so no change to them.  New handlers that want to control the
> flow of data to/from the client socket can do so by sending requests to
> mongrel2 over a XREQ socket.
>
> For large uploads from the client socket, mongrel2's config would mark a
> handler as wanting to control the transfer of the request body over the XREQ
> socket.  Mongrel2 would only send the http headers + connection id like it
> does now, but without the request body.  The handler would then send a request
> for a chunk of data to be read from the client socket, which mongrel2 would
> either try to read the whole requested chunk size or just respond with what
> was available (i.e. a partial read).  The handler would keep sending request
> until the whole request body was transferred.
>
> For streaming data out to the client socket, the handler would send chunks of
> data in requests over the XREQ socket, mongrel2 would send back a response
> after it finished writing the chunk to the socket.  The handler can still
> close a client connection like it does now, if sending a chunk takes to long
> (maybe there can be a max. timeout value in the config, to automate the dis-
> connect of slow clients).  This method might not work well for sending the
> same chunk out to many connections.
>
> XREQ/XREP sockets would need to be used to allow multiple concurrent transfer.
> Also each mongrel2/handler instance would need to use a unique identity on so
> request/responses can be routed correctly.  Mongrel2 can re-use the same uuid
> it currently sends with each request it sends to the handler over the
> PUSH/PULL sockets.  Doing this should still allow many-to-many, many-to-one,
> one-to-many layouts of mongrel2 to backend handlers.
>
> To help with the latency of each request/response pair, mongrel2 can buffer X
> Kbytes for each socket (i.e. send a response back to the handler when X Kbytes
> or less are in the buffer), so the handler has time to prepare the next chunk
> for sending.  I think this buffering in mongrel2 would only be needed for
> transfers that want to maintain a smooth constant transfer speed to the
> client.
>
> For bi-directional connections (i.e. Websocket/flash/etc...) the backend
> handlers should be allowed to issue outstanding read request and still send
> write requests to mongrel2.
>
> Basically this is like an async. RPC interface to the client socket.

+1

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

On Tue, Aug 16, 2011 at 11:11:03AM -0400, Jim Fulton wrote:
> > Basically this is like an async. RPC interface to the client socket.
> 
> +1

Ok, you +1 but what would you *do* with it?  Typically, people come
asking for features like this, not because they have something new they
want to build, but because they have some old way they do things or some
old code they need to support.  If that's the case then describe the
code or let me see it so the feature can be done right.

If that's not the case, like you've got an idea for something new, then
describe what you're going to *do* with this feature.  Don't describe
what you want the feature to do.

I say this because usually what you want and what you really need are
very different things. :-)

-- 
Zed A. Shaw
http://zedshaw.com/

On 08/16/2011 07:19 PM, Zed A. Shaw wrote:
> On Tue, Aug 16, 2011 at 11:11:03AM -0400, Jim Fulton wrote:
>>> Basically this is like an async. RPC interface to the client socket.
>>
>> +1
>
> Ok, you +1 but what would you *do* with it?  Typically, people come
> asking for features like this, not because they have something new they
> want to build, but because they have some old way they do things or some
> old code they need to support.  If that's the case then describe the
> code or let me see it so the feature can be done right.
>
> If that's not the case, like you've got an idea for something new, then
> describe what you're going to *do* with this feature.  Don't describe
> what you want the feature to do.
>
> I say this because usually what you want and what you really need are
> very different things. :-)

In my case I have a setup a bit like Heroku, that is, each handler is a 
process running somewhere on my LAN in a VM with a read only file 
system. I get the uploads from Mongrel2 and stream them to MongoDB. I 
basically force a 15MB max size to not have to worry about the upload.

My next approach to hanlde larger files (not yet a need) will simply be 
to create a special "handler" working on the Mongrel2 host with a simple 
API to be able to HTTP GET a given file. As it will be on the LAN with a 
warm disc cache (just uploaded and touched by Mongrel2) it will be fast.

Dataflow:
- big upload from client
- headers to the handler -> accept or kill
- if accept, stream to disc
- end message with path on disc
- get the uploaded body to parse it and operate on it
- done

At the end on the Mongrel2 host I must have series of handler processes, 
this is the only drawback because it "pins" handlers on the same host. 
This is because I prefer not to go the shared drive/folder approach.

Not very complicated.

My question is, why not allow the sending of the body payload as a 
multipart zmq message at the end of the upload? So, instead of sending 
this last message with: "now you can grab the payload here" you directly 
send a multipart zmq message with the payload.

loïc

On Wed, Aug 17, 2011 at 10:18:01AM +0200, Loic d'Anterroches wrote:
> My question is, why not allow the sending of the body payload as a 
> multipart zmq message at the end of the upload? So, instead of sending 
> this last message with: "now you can grab the payload here" you directly 
> send a multipart zmq message with the payload.

Multipart isn't async.  You'd block whatever process was receiving that
file for each part until the whole file was done.

-- 
Zed A. Shaw
http://zedshaw.com/

On 08/17/2011 11:50 AM, Zed A. Shaw wrote:
> On Wed, Aug 17, 2011 at 10:18:01AM +0200, Loic d'Anterroches wrote:
>> My question is, why not allow the sending of the body payload as a
>> multipart zmq message at the end of the upload? So, instead of sending
>> this last message with: "now you can grab the payload here" you directly
>> send a multipart zmq message with the payload.
>
> Multipart isn't async.  You'd block whatever process was receiving that
> file for each part until the whole file was done.

Ok, but you block only for the transfer from Mongrel2 to the handler 
once the client has fully uploaded the body. Could be a configuration 
setting?

If you get a 500MB file, it would be pretty fast to push it over a 1Gbps 
connection (5 sec in the best case) and you could monitor the size of 
each part on the handler to push back notifications to the client. The 5 
sec internal transfer will anyway be nothing compared to the client to 
mongrel2 transfer time.

You still have the 5 sec blocking of the handler, if your job is to 
handle many big uploads, you would anyway have a lot of handlers ready 
to manage the slow uploads, this is just a scalling issue and zmq allow 
us to do that easily over the network (then you go from 1Gb to 10Gb 
interconnect).

Just an idea, my line of thinking is: "how to have the minimum of moving 
parts to handle large uploads?"

loïc

On Tue, Aug 16, 2011 at 1:19 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> On Tue, Aug 16, 2011 at 11:11:03AM -0400, Jim Fulton wrote:
>> > Basically this is like an async. RPC interface to the client socket.
>>
>> +1
>
> Ok, you +1 but what would you *do* with it?  Typically, people come
> asking for features like this, not because they have something new they
> want to build, but because they have some old way they do things or some
> old code they need to support.  If that's the case then describe the
> code or let me see it so the feature can be done right.
>
> If that's not the case, like you've got an idea for something new, then
> describe what you're going to *do* with this feature.  Don't describe
> what you want the feature to do.
>
> I say this because usually what you want and what you really need are
> very different things. :-)

Fair enough.

I have an existing CMS application running on a more traditional web
architecture.  Among the content our customers manage are often large
videos.  These are handled fairly well by our current architecture
which handles large request and responses effectively.  Mongrel2's
current behavior of wanting to store request and response bodies in
0mq messages or require the use of temporary files makes handling
large bodies challenging.

I'm interested in mongrel2 because, with 0mq, it looks like I can
implement the broker I mentioned in another message in a pretty
straightforward way.  I'm a bit concerned about how I'm going to
handle large bodies, in both requests and responses.  For requests, I
could have the broker run on the same machine as the Mongrel2 server
and do streaming in the broker, using some scheme like what Robert
proposed.  For output well, um, we were thinking of maybe serving
video out of S3. :) It would be great if I didn't have to handle input
streaming myself and if I wasn't forced to have a different delivery
scheme for large responses.  Robert's proposal looks like at least the
beginning of something that might work.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

On Tue, Aug 16, 2011 at 01:58:02PM -0400, Jim Fulton wrote:
> On Tue, Aug 16, 2011 at 1:19 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> I have an existing CMS application running on a more traditional web
> architecture.  Among the content our customers manage are often large
> videos.  These are handled fairly well by our current architecture
> which handles large request and responses effectively.  Mongrel2's
> current behavior of wanting to store request and response bodies in
> 0mq messages or require the use of temporary files makes handling
> large bodies challenging.

What's your current mechanism for handling this?  Are you doing
something like sending back notifications to the browser on every X
bytes received because the browser is idiotic and refuses to tell you
client-side how many bytes are sent?

In other words:

Mongrel2 handles uploads just fine, and does it really well.  File come
in, file go somewhere else.  What is it that your current CMS does that
means you need this "constant stream of 0mq messages for every X bytes
uploaded" scheme?

-- 
Zed A. Shaw
http://zedshaw.com/

On Wed, Aug 17, 2011 at 5:47 AM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> On Tue, Aug 16, 2011 at 01:58:02PM -0400, Jim Fulton wrote:
>> On Tue, Aug 16, 2011 at 1:19 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
>> I have an existing CMS application running on a more traditional web
>> architecture.  Among the content our customers manage are often large
>> videos.  These are handled fairly well by our current architecture
>> which handles large request and responses effectively.  Mongrel2's
>> current behavior of wanting to store request and response bodies in
>> 0mq messages or require the use of temporary files makes handling
>> large bodies challenging.
>
> What's your current mechanism for handling this?

We're using an asynchronous Python web server, zope.server,
which is based on Python's asyncore framework.  Each application
server runs in one of these.  Data from clients come in as fast as they
can send it and as fast as asyncore can receive it.  For large inputs,
the data are streamed to a temporary file and the application code
reads from that.  There's a load balancer between the clients and the
application servers that route requests to the application servers.
(There may also be other servers, like apache in the flow, but they
don't really change how this works in any significant way.)

For large outputs, we use ZODB blobs, which end up as files
on the application servers that are streamed to the clients
by asynchronous web server.  (The blobs are streamed from
a database server and cached locally.)

The load balancer is rather dumb, and I'm interested
in replacing it with Mongrel2 and a broker to rout
requests to application servers based on what they've
worked on before, to try to get better application-level
caching.

>  Are you doing
> something like sending back notifications to the browser on every X
> bytes received because the browser is idiotic and refuses to tell you
> client-side how many bytes are sent?

No.

> In other words:
>
> Mongrel2 handles uploads just fine, and does it really well.

It doesn't large uploads very well if you need handlers to reside
on different machines.  If you want that, then you currently have to
solve that yourself (or use a network file system).

> File come
> in, file go somewhere else.

Not sure what you mean.  As I understand it, with Mongrel2, a
large upload comes in and is written to a temporary file.  A handler
must be local, or you have to use a network file system for a
handler to get to the file.

>  What is it that your current CMS does that
> means you need this "constant stream of 0mq messages for every X bytes
> uploaded" scheme?

I must be communicating poorly.  Sorry.

I want to be able to handle large request and response bodies without
storing the in RAM. That's it.

Our current infrastructure doesn't have a problem in this area because the
each application server has it's own web server that can stream to and
from local files.  Distribution of requests to the application servers
is done via
a load balancer of some sort.

Maybe where we're not connecting is that I'm looking at using Mongrel2
to replace the load balancer and distribute requests to handlers on different
machines.  In this case, the application servers would no longer be embedded
in web servers, but would talk to a broker over 0mq.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

On Wed, Aug 17, 2011 at 10:53:09AM -0400, Jim Fulton wrote:
> > What's your current mechanism for handling this?
> 
> We're using an asynchronous Python web server, zope.server,
> which is based on Python's asyncore framework.  Each application
> server runs in one of these.  Data from clients come in as fast as they
> can send it and as fast as asyncore can receive it.  For large inputs,
> the data are streamed to a temporary file and the application code
> reads from that.  There's a load balancer between the clients and the
> application servers that route requests to the application servers.
> (There may also be other servers, like apache in the flow, but they
> don't really change how this works in any significant way.)

Ok, that's more like it.

> I want to be able to handle large request and response bodies without
> storing the in RAM. That's it.

Right, I get that, but the problem is if I want to send a 300M file to
you Mongrel2 has to put it all in ram to get it out over the request, or
use crazy schemes like multipart (which isn't async), or sending little
chunks.

Let me roll this in my head a bit, as I think probably what you need is
just something that takes files being uploaded and sends pieces
somewhere with 0mq.  Not really anything in Mongrel2 but a little
helper, so you don't have to write it.

Stay tuned.

-- 
Zed A. Shaw
http://zedshaw.com/

On Wed, Aug 17, 2011 at 12:04 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> On Wed, Aug 17, 2011 at 10:53:09AM -0400, Jim Fulton wrote:

...

>> I want to be able to handle large request and response bodies without
>> storing the in RAM. That's it.
>
> Right, I get that, but the problem is if I want to send a 300M file to
> you Mongrel2 has to put it all in ram to get it out over the request, or
> use crazy schemes like multipart (which isn't async), or sending little
> chunks.
>
> Let me roll this in my head a bit, as I think probably what you need is
> just something that takes files being uploaded and sends pieces
> somewhere with 0mq.  Not really anything in Mongrel2 but a little
> helper, so you don't have to write it.
>
> Stay tuned.

While you're rolling, I wonder if it would make sense
to do something for output, similar to what you already to
for input.  That is, allow a handler to stick a file somewhere
and ask mongrel to serve from it directly.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

On Mon, Aug 22, 2011 at 5:42 AM, Jim Fulton <jim@zope.com> wrote:
> On Wed, Aug 17, 2011 at 12:04 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
>> On Wed, Aug 17, 2011 at 10:53:09AM -0400, Jim Fulton wrote:
>
> ...
>
>>> I want to be able to handle large request and response bodies without
>>> storing the in RAM. That's it.
>>
>> Right, I get that, but the problem is if I want to send a 300M file to
>> you Mongrel2 has to put it all in ram to get it out over the request, or
>> use crazy schemes like multipart (which isn't async), or sending little
>> chunks.
>>
>> Let me roll this in my head a bit, as I think probably what you need is
>> just something that takes files being uploaded and sends pieces
>> somewhere with 0mq.  Not really anything in Mongrel2 but a little
>> helper, so you don't have to write it.
>>
>> Stay tuned.
>
> While you're rolling, I wonder if it would make sense
> to do something for output, similar to what you already to
> for input.  That is, allow a handler to stick a file somewhere
> and ask mongrel to serve from it directly.
>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
>

Supporting X-Sendfile?

On Sun, Aug 21, 2011 at 8:44 PM, Josh Simmons <simmons.44@gmail.com> wrote:
> On Mon, Aug 22, 2011 at 5:42 AM, Jim Fulton <jim@zope.com> wrote:
>> On Wed, Aug 17, 2011 at 12:04 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
>>> On Wed, Aug 17, 2011 at 10:53:09AM -0400, Jim Fulton wrote:
>>
>> ...
>>
>>>> I want to be able to handle large request and response bodies without
>>>> storing the in RAM. That's it.
>>>
>>> Right, I get that, but the problem is if I want to send a 300M file to
>>> you Mongrel2 has to put it all in ram to get it out over the request, or
>>> use crazy schemes like multipart (which isn't async), or sending little
>>> chunks.
>>>
>>> Let me roll this in my head a bit, as I think probably what you need is
>>> just something that takes files being uploaded and sends pieces
>>> somewhere with 0mq.  Not really anything in Mongrel2 but a little
>>> helper, so you don't have to write it.
>>>
>>> Stay tuned.
>>
>> While you're rolling, I wonder if it would make sense
>> to do something for output, similar to what you already to
>> for input.  That is, allow a handler to stick a file somewhere
>> and ask mongrel to serve from it directly.
>>
>> Jim
>>
>> --
>> Jim Fulton
>> http://www.linkedin.com/in/jimfulton
>>
>
> Supporting X-Sendfile?

Sounds like a good idea to me. :)

The only downside I can see is that I don't think Mongrel2 currently
parses response headers.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

On Mon, Aug 22, 2011 at 8:04 AM, Jim Fulton <jim@zope.com> wrote:
>> Supporting X-Sendfile?
>
> Sounds like a good idea to me. :)
>
> The only downside I can see is that I don't think Mongrel2 currently
> parses response headers.

better to do as a special ZMQ message instead of an HTTP header

-- 
Javier

On Mon, Aug 22, 2011 at 09:29:58AM -0500, Javier Guerra Giraldez wrote:
> On Mon, Aug 22, 2011 at 8:04 AM, Jim Fulton <jim@zope.com> wrote:
> >> Supporting X-Sendfile?
> >
> > Sounds like a good idea to me. :)
> >
> > The only downside I can see is that I don't think Mongrel2 currently
> > parses response headers.
> 
> better to do as a special ZMQ message instead of an HTTP header

That's what I'd think too, which would require tweaking the protocol a
bit to allow for out-of-band data on the response.

-- 
Zed A. Shaw
http://zedshaw.com/

I know I've mentioned this before, but the easiest way to add
out-of-band is to change the response from a netstring to a tnetstring.
If you get a string, it's just as before.  If you get a dictionary, you
have out-of-band communication.
On 09:35 Mon 22 Aug     , Zed A. Shaw wrote:
> On Mon, Aug 22, 2011 at 09:29:58AM -0500, Javier Guerra Giraldez wrote:
> > On Mon, Aug 22, 2011 at 8:04 AM, Jim Fulton <jim@zope.com> wrote:
> > >> Supporting X-Sendfile?
> > >
> > > Sounds like a good idea to me. :)
> > >
> > > The only downside I can see is that I don't think Mongrel2 currently
> > > parses response headers.
> > 
> > better to do as a special ZMQ message instead of an HTTP header
> 
> That's what I'd think too, which would require tweaking the protocol a
> bit to allow for out-of-band data on the response.
> 
> -- 
> Zed A. Shaw
> http://zedshaw.com/
> 

On Wed, Aug 24, 2011 at 10:38:09AM -0700, Jason Miller wrote:
> I know I've mentioned this before, but the easiest way to add
> out-of-band is to change the response from a netstring to a tnetstring.
> If you get a string, it's just as before.  If you get a dictionary, you
> have out-of-band communication.

Oh yeah....duh.  That'd work great actually.

-- 
Zed A. Shaw
http://zedshaw.com/

On 2011-08-24 20:38, Zed A. Shaw wrote:
> On Wed, Aug 24, 2011 at 10:38:09AM -0700, Jason Miller wrote:
>> I know I've mentioned this before, but the easiest way to add
>> out-of-band is to change the response from a netstring to a tnetstring.
>> If you get a string, it's just as before.  If you get a dictionary, you
>> have out-of-band communication.
> 
> Oh yeah....duh.  That'd work great actually.

And easy to generate from any language even without a tnetstring library
because it will most likely be very simple - that is, like now - in
99.9% of the cases.

loïc