Alright everyone, full tnetstring support is now merged into trunk from proxymustdie, along with all the other fixes I've made while working on it. Here's what's up: 1. There's a new column for Handlers, protocol. 2. You should m2sh config to rebuild your config file. 3. You can then set protocl='tnetstring' and you'll get the tnetstring protocol. 4. The examples/python/ code has a naive tnetstring library in mongrel2/tnetstrings.py that does them. 5. The Python mongrel2 library *dynamically* figures out if a payload it gets is a tnetstring or a JSON string for headers. 6. You should be able to just flip the protocol='tnetstring' in your config and it'll keep working with your existing Python applications. FOR HANDLER AUTHORS What I'd like now is for authors of Handler libraries to do what I've done in the Python example library: 1. Get a tnetstring implementation going. Look at the Python code as a naive minimal implementation is very easy. 2. Use the file in the mongrel2 source that gets created by the test suite: tests/request_payloads.txt as you test sample. 3. Update your handler library so that it can reliably parse the lines in tests/request_payloads.txt no matter what. That file has one line of json and one line of tnetstrings, and your library should be able to adapt to either and produce the same result. How you adapt to either tnetstring or json payloads is you do this: A. Use your tnetstring library to parse the payload. B. If you get a string, it's json, run your json loader. C. Otherwise, it's a tnetstring you're done. Easy, this works because tnetstrings are backward compatible with generic netstrings, so you don't need to do anything other than the above to handle both protocols. Let me know if you find bugs. -- Zed A. Shaw http://zedshaw.com/
2011/4/2 Zed A. Shaw <zedshaw@zedshaw.com>: > Alright everyone, full tnetstring support is now merged into trunk from > proxymustdie, along with all the other fixes I've made while working on > it. Here's what's up: > > 1. There's a new column for Handlers, protocol. > 2. You should m2sh config to rebuild your config file. Here is just a small SQL script to migrate an already existing mongrel2 config db. This can be useful for those already running mongrel2 and not using a config file to rebuild config db every time. alter table handler add column protocol TEXT default 'json'; Would be a good idea to have in a ChangeLog file all commands needed to migrate an existing config db, or at least all schema changes done for the upcoming release. -- Dalton Barreto http://daltonmatos.wordpress.com
Will m2 1.5 gracefully fail/ignore the new column in the schema or do they need to be upgraded concurrently? ✈ Matt On Apr 3, 2011, at 13:27 , Dalton Barreto wrote: > 2011/4/2 Zed A. Shaw <zedshaw@zedshaw.com>: >> Alright everyone, full tnetstring support is now merged into trunk from >> proxymustdie, along with all the other fixes I've made while working on >> it. Here's what's up: >> >> 1. There's a new column for Handlers, protocol. >> 2. You should m2sh config to rebuild your config file. > > > Here is just a small SQL script to migrate an already existing > mongrel2 config db. This can be useful for those already running > mongrel2 and not using a config file to rebuild config db every time. > > alter table handler add column protocol TEXT default 'json'; > > Would be a good idea to have in a ChangeLog file all commands needed > to migrate an existing config db, or at least all schema changes done > for the upcoming release. > > -- > Dalton Barreto > http://daltonmatos.wordpress.com
2011/4/3 Matt Towers <matt@ziplinegames.com>: > Will m2 1.5 gracefully fail/ignore the new column in the schema or do they > need to be upgraded concurrently? > In the tests that I did mongrel2 showed on the logs: [ERROR] (src/config/db.c:68: errno: No such file or directory) [SQL ERROR]: SELECT id, raw_payload, protocol FROM handler WHERE id=1 : 'no such column: protocol' [WARN] (src/config/config.c:108: errno: None) Couldn't get the Handler.raw_payload setting, you might need to rebuild your db. But the handler was loaded and I was able to send requests and receive successful responses. In this case this is probalby because the code checks for tnetstring protocol and in any case other than this, it assumes json. This is in config.c:78 I still think that is a good idea to always have the config db updated. Note: Even the error message showing a problem in handler.war_payload the problem in this case is with protocol, thats because the two columns are selected in the same SQL string. -- Dalton Barreto http://daltonmatos.wordpress.com
On Sat, 2011-04-02 at 11:14 -0700, Zed A. Shaw wrote: > Alright everyone, full tnetstring support is now merged into trunk from > proxymustdie, along with all the other fixes I've made while working on > it. Awesome. Excited for the upcoming release! I've also back-ported your tnetstring cleanups and fixes into my python module, and made a few more of my own. Would you be open to me trying to maintain a single "tns_core.c" that can be used unmodified in both projects? It may not be possible due to divergent customisations, but I'd be interested to try. > What I'd like now is for authors of Handler libraries to do what I've > done in the Python example library: > > 1. Get a tnetstring implementation going. Look at the Python code as a > naive minimal implementation is very easy. > 2. Use the file in the mongrel2 source that gets created by the test > suite: tests/request_payloads.txt as you test sample. > 3. Update your handler library so that it can reliably parse the lines > in tests/request_payloads.txt no matter what. That file has one line of > json and one line of tnetstrings, and your library should be able to > adapt to either and produce the same result. Done for m2wsgi, no bugs or even little hiccups along the way. Cheers, Ryan -- Ryan Kelly http://www.rfk.id.au | This message is digitally signed. Please visit ryan@rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details
On Sun, Apr 03, 2011 at 11:04:38AM +1000, Ryan Kelly wrote: > On Sat, 2011-04-02 at 11:14 -0700, Zed A. Shaw wrote: > > Alright everyone, full tnetstring support is now merged into trunk from > > proxymustdie, along with all the other fixes I've made while working on > > it. > > Awesome. Excited for the upcoming release! > > I've also back-ported your tnetstring cleanups and fixes into my python > module, and made a few more of my own. Would you be open to me trying > to maintain a single "tns_core.c" that can be used unmodified in both > projects? It may not be possible due to divergent customisations, but > I'd be interested to try. That should be possible. I'll try doing a Lua library similar to your Python library and then see what can be commong. I'm thinking putting the common C code out as a standalone "construction kit" might be good too. Also, I use the dbg.h file from mongrel2 to make C code tighter, and using those macros fixed a few bugs your code had while also shrinking the size down. Are you using it? -- Zed A. Shaw http://zedshaw.com/
On Sat, 2011-04-02 at 20:08 -0700, Zed A. Shaw wrote: > On Sun, Apr 03, 2011 at 11:04:38AM +1000, Ryan Kelly wrote: > > On Sat, 2011-04-02 at 11:14 -0700, Zed A. Shaw wrote: > > > Alright everyone, full tnetstring support is now merged into trunk from > > > proxymustdie, along with all the other fixes I've made while working on > > > it. > > > > Awesome. Excited for the upcoming release! > > > > I've also back-ported your tnetstring cleanups and fixes into my python > > module, and made a few more of my own. Would you be open to me trying > > to maintain a single "tns_core.c" that can be used unmodified in both > > projects? It may not be possible due to divergent customisations, but > > I'd be interested to try. > > That should be possible. I'll try doing a Lua library similar to your > Python library and then see what can be commong. I'm thinking putting > the common C code out as a standalone "construction kit" might be good > too. > > Also, I use the dbg.h file from mongrel2 to make C code tighter, and > using those macros fixed a few bugs your code had while also shrinking > the size down. Are you using it? Yes, after being thoroughly schooled by commit 300b3d9972 :-) Well, I'm using a version with an equivalent API that calls python's error handlers rather than logging to a file, but it should be easy to keep compatible. On a side note, here's some interesting results from the latest shootout of the python module: tnetstring 0.722779989243 cjson: 1.75372695923 speedup: 58.79 % yajl: 2.44137191772 speedup: 70.39 % ujson: 0.907096147537 speedup: 20.32 % marshal: 0.999366044998 speedup: 27.68 % Yes, you read that right, tnetstring can apparently round-trip data ~25% faster than python's own internal bytecode loader. Of course it supports a few less datatypes, but still, that's neat. Cheers, Ryan -- Ryan Kelly http://www.rfk.id.au | This message is digitally signed. Please visit ryan@rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details
On Sun, Apr 03, 2011 at 01:27:49PM +1000, Ryan Kelly wrote: > > Also, I use the dbg.h file from mongrel2 to make C code tighter, and > > using those macros fixed a few bugs your code had while also shrinking > > the size down. Are you using it? > > Yes, after being thoroughly schooled by commit 300b3d9972 :-) Awesome. That one little file is kind of "Zed's way to avoid bugs in C". > Well, I'm using a version with an equivalent API that calls python's > error handlers rather than logging to a file, but it should be easy to > keep compatible. I'll check out your version and see what it's doing. I'll probably need to do similar in lua. > Yes, you read that right, tnetstring can apparently round-trip data ~25% > faster than python's own internal bytecode loader. Of course it > supports a few less datatypes, but still, that's neat. Ha, now that's some evil right there. -- Zed A. Shaw http://zedshaw.com/
Are there any implications for Tir? ✈ Matt On Apr 2, 2011, at 11:14 , Zed A. Shaw wrote: > Alright everyone, full tnetstring support is now merged into trunk from > proxymustdie, along with all the other fixes I've made while working on > it. Here's what's up: > > 1. There's a new column for Handlers, protocol. > 2. You should m2sh config to rebuild your config file. > 3. You can then set protocl='tnetstring' and you'll get the tnetstring > protocol. > 4. The examples/python/ code has a naive tnetstring library in > mongrel2/tnetstrings.py that does them. > 5. The Python mongrel2 library *dynamically* figures out if a payload it > gets is a tnetstring or a JSON string for headers. > 6. You should be able to just flip the protocol='tnetstring' in your > config and it'll keep working with your existing Python applications. > > FOR HANDLER AUTHORS > > What I'd like now is for authors of Handler libraries to do what I've > done in the Python example library: > > 1. Get a tnetstring implementation going. Look at the Python code as a > naive minimal implementation is very easy. > 2. Use the file in the mongrel2 source that gets created by the test > suite: tests/request_payloads.txt as you test sample. > 3. Update your handler library so that it can reliably parse the lines > in tests/request_payloads.txt no matter what. That file has one line of > json and one line of tnetstrings, and your library should be able to > adapt to either and produce the same result. > > How you adapt to either tnetstring or json payloads is you do this: > > A. Use your tnetstring library to parse the payload. > B. If you get a string, it's json, run your json loader. > C. Otherwise, it's a tnetstring you're done. > > Easy, this works because tnetstrings are backward compatible with > generic netstrings, so you don't need to do anything other than the > above to handle both protocols. > > Let me know if you find bugs. > > -- > Zed A. Shaw > http://zedshaw.com/
Yeah I havn't updated m2-lua so it won't work with tnetstrings yet. I'll try to do this today. On Sun, Apr 3, 2011 at 5:38 AM, Matt Towers <matt@ziplinegames.com> wrote: > Are there any implications for Tir? > > ✈ Matt > > > > On Apr 2, 2011, at 11:14 , Zed A. Shaw wrote: > > Alright everyone, full tnetstring support is now merged into trunk from > proxymustdie, along with all the other fixes I've made while working on > it. Here's what's up: > > 1. There's a new column for Handlers, protocol. > 2. You should m2sh config to rebuild your config file. > 3. You can then set protocl='tnetstring' and you'll get the tnetstring > protocol. > 4. The examples/python/ code has a naive tnetstring library in > mongrel2/tnetstrings.py that does them. > 5. The Python mongrel2 library *dynamically* figures out if a payload it > gets is a tnetstring or a JSON string for headers. > 6. You should be able to just flip the protocol='tnetstring' in your > config and it'll keep working with your existing Python applications. > > FOR HANDLER AUTHORS > > What I'd like now is for authors of Handler libraries to do what I've > done in the Python example library: > > 1. Get a tnetstring implementation going. Look at the Python code as a > naive minimal implementation is very easy. > 2. Use the file in the mongrel2 source that gets created by the test > suite: tests/request_payloads.txt as you test sample. > 3. Update your handler library so that it can reliably parse the lines > in tests/request_payloads.txt no matter what. That file has one line of > json and one line of tnetstrings, and your library should be able to > adapt to either and produce the same result. > > How you adapt to either tnetstring or json payloads is you do this: > > A. Use your tnetstring library to parse the payload. > B. If you get a string, it's json, run your json loader. > C. Otherwise, it's a tnetstring you're done. > > Easy, this works because tnetstrings are backward compatible with > generic netstrings, so you don't need to do anything other than the > above to handle both protocols. > > Let me know if you find bugs. > > -- > Zed A. Shaw > http://zedshaw.com/ > > >
On Sat, Apr 02, 2011 at 11:14:00AM -0700, Zed A. Shaw wrote: > 3. You can then set protocl='tnetstring' and you'll get the tnetstring > protocol. Sorry this should be protocol='tnetstring'. Typo. -- Zed A. Shaw http://zedshaw.com/