Hello, Is there anything special that needs to be done when sending binary data in a response? In my application, the binary data in a response is being corrupted. I have checked, using tcpdump, that the binary data is uncorrupted in the ZeroMQ packets being sent from my application to Mongrel. My environment: Ubuntu server 11.04 beta Mongrel2 from April 27, 2011 (comment "Got the SQL wrong on the new cache ttl, ... ") zeromq version 2.1.6 tar ball (from www.zeromq.org) Plack version 0.9976 Plack::Handler::Mongrel2 version 0.01000_03 with bug fixes (https://github.com/hbaragar/Plack-Handler-Mongrel2) Jifty version 1.10228 Any assistance would be appreciated. Thanks, Henry -- Henry Baragar Instantiated Software
On Wed, Apr 27, 2011 at 05:53:00PM -0400, Henry Baragar wrote: > Hello, > > Is there anything special that needs to be done when sending binary data in a > response? > Ok, two things to try: 1. If the connection that comes in isn't an HTTP request, then you'll be in a mode that Mongrel2 assumes needs base64 (like a json socket). Add raw=1 to that Handler to force it raw. 2. Go into handler.c and put a call to debug_dump() inside handler_process_request() to make sure it's sending what you think it is. This does a hex dump of what you want. Then rebuild with: make clean dev sudo make install Warning, this will print a huge ton of crap to the logs so do a test with some small stuff. Best place is put: debug_dump(bdata(payload), blength(payload)); At line handler.c:141. Let me know if either of those gives you a clue. -- Zed A. Shaw http://zedshaw.com/
On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > Hello, > > Is there anything special that needs to be done when sending binary > data in a response? There shouldn't be, all data is treated as binary by Mongrel2. > In my application, the binary data in a response is being corrupted. I > have checked, using tcpdump, that the binary data is uncorrupted in > the ZeroMQ packets being sent from my application to Mongrel. Have you sniffed it coming out of Mongrel2, and is it corrupted there? Or is the application getting confused about e.g. content-length or mime type and corrupting the data internally? Cheers, Ryan -- Ryan Kelly http://www.rfk.id.au | This message is digitally signed. Please visit ryan@rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details
On April 27, 2011 06:08:52 pm Ryan Kelly wrote: > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > > Hello, > > > > Is there anything special that needs to be done when sending binary > > data in a response? > > There shouldn't be, all data is treated as binary by Mongrel2. > That is what I had thought. > > In my application, the binary data in a response is being corrupted. I > > have checked, using tcpdump, that the binary data is uncorrupted in > > the ZeroMQ packets being sent from my application to Mongrel. > > Have you sniffed it coming out of Mongrel2, and is it corrupted there? I had only looked at things with a web browser (or two). I just did a tcpdump of the output and it definitely is corrupted. For one thing, the output is almost 50% larger than the input. The kind of corruption I see looks similar to what happens when a binary stream is UTF-8 encoded. > Or is the application getting confused about e.g. content-length or mime > type and corrupting the data internally? > The application is reporting the correct mime-type (application/pdf) and content-length. Any other suggestions? Henry > > Cheers, > > Ryan -- Henry Baragar Instantiated Software
Is your handler library doing something dodgy and handling it as utf-8 per-chance? On Thu, Apr 28, 2011 at 8:56 AM, Henry Baragar < Henry.Baragar@instantiated.ca> wrote: > On April 27, 2011 06:08:52 pm Ryan Kelly wrote: > > > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > > > > Hello, > > > > > > > > Is there anything special that needs to be done when sending binary > > > > data in a response? > > > > > > There shouldn't be, all data is treated as binary by Mongrel2. > > > > > That is what I had thought. > > > > In my application, the binary data in a response is being corrupted. I > > > > have checked, using tcpdump, that the binary data is uncorrupted in > > > > the ZeroMQ packets being sent from my application to Mongrel. > > > > > > Have you sniffed it coming out of Mongrel2, and is it corrupted there? > > I had only looked at things with a web browser (or two). I just did a > tcpdump of the output and it definitely is corrupted. For one thing, the > output is almost 50% larger than the input. > > The kind of corruption I see looks similar to what happens when a binary > stream is UTF-8 encoded. > > > Or is the application getting confused about e.g. content-length or mime > > > type and corrupting the data internally? > > > > > The application is reporting the correct mime-type (application/pdf) and > content-length. > > Any other suggestions? > > Henry > > > > > > Cheers, > > > > > > Ryan > > > -- > > Henry Baragar > > Instantiated Software > >
On April 27, 2011 07:09:10 pm joshua simmons wrote: > Is your handler library doing something dodgy and handling it as utf-8 > per-chance? > If we take a look a the the data path: Application => Mongrel2 => Web browser We find that the binary data is fine on the first link (represented by "=>") but not the second. Hear are the headers being sent to the Web browser: HTTP/1.1 200 OK Content-Type: application/pdf Expires: Wed, 27 Apr 2011 22:45:45 GMT Set-Cookie: JIFTY_SID_80=7178daff5f53c0aff8973191ffbbec00; path=/; expires=30 min Content-Length: 31176 I don't think that the handler is doing anything dodgy. Something is happening in mongrel2. Regards, Henry > On Thu, Apr 28, 2011 at 8:56 AM, Henry Baragar < > > Henry.Baragar@instantiated.ca> wrote: > > On April 27, 2011 06:08:52 pm Ryan Kelly wrote: > > > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > > > > Hello, > > > > > > > > > > > > > > > > Is there anything special that needs to be done when sending binary > > > > > > > > data in a response? > > > > > > There shouldn't be, all data is treated as binary by Mongrel2. > > > > That is what I had thought. > > > > > > In my application, the binary data in a response is being corrupted. > > > > I > > > > > > > > have checked, using tcpdump, that the binary data is uncorrupted in > > > > > > > > the ZeroMQ packets being sent from my application to Mongrel. > > > > > > Have you sniffed it coming out of Mongrel2, and is it corrupted there? > > > > I had only looked at things with a web browser (or two). I just did a > > tcpdump of the output and it definitely is corrupted. For one thing, the > > output is almost 50% larger than the input. > > > > The kind of corruption I see looks similar to what happens when a binary > > stream is UTF-8 encoded. > > > > > Or is the application getting confused about e.g. content-length or > > > mime > > > > > > type and corrupting the data internally? > > > > The application is reporting the correct mime-type (application/pdf) and > > content-length. > > > > Any other suggestions? > > > > Henry > > > > > Cheers, > > > > > > > > > > > > Ryan > > > > -- > > > > Henry Baragar > > > > Instantiated Software -- Henry Baragar Instantiated Software
I created a simple test-handler that sends bytes from 0-255 in the
reply. This is using my own handler library that is on github
(jasom/mymongrel2) but I'm pretty sure nobody else is using.
This did expose a flaw in my backend that I will have to fix (Apparently
the common-lisp zmq bindings default to UTF-8 for encoding)
However, once I was sending the correct data to mongrel2, it came out
clean, as you can see from the following hex-dump.
This would suggest that the problem is in th
0000000: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f ................
0000010: 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f ................
0000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f !"#$%&'()*+,-./
0000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f 0123456789:;<=>?
0000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f @ABCDEFGHIJKLMNO
0000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f PQRSTUVWXYZ[\]^_
0000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f `abcdefghijklmno
0000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e7f pqrstuvwxyz{|}~.
0000080: 8081 8283 8485 8687 8889 8a8b 8c8d 8e8f ................
0000090: 9091 9293 9495 9697 9899 9a9b 9c9d 9e9f ................
00000a0: a0a1 a2a3 a4a5 a6a7 a8a9 aaab acad aeaf ................
00000b0: b0b1 b2b3 b4b5 b6b7 b8b9 babb bcbd bebf ................
00000c0: c0c1 c2c3 c4c5 c6c7 c8c9 cacb cccd cecf ................
00000d0: d0d1 d2d3 d4d5 d6d7 d8d9 dadb dcdd dedf ................
00000e0: e0e1 e2e3 e4e5 e6e7 e8e9 eaeb eced eeef ................
00000f0: f0f1 f2f3 f4f5 f6f7 f8f9 fafb fcfd feff ................
On 19:26 Wed 27 Apr , Henry Baragar wrote:
> On April 27, 2011 07:09:10 pm joshua simmons wrote:
> > Is your handler library doing something dodgy and handling it as utf-8
> > per-chance?
> >
>
> If we take a look a the the data path:
>
> Application => Mongrel2 => Web browser
>
> We find that the binary data is fine on the first link (represented by "=>")
> but not the second.
>
> Hear are the headers being sent to the Web browser:
> HTTP/1.1 200 OK
> Content-Type: application/pdf
> Expires: Wed, 27 Apr 2011 22:45:45 GMT
> Set-Cookie: JIFTY_SID_80=7178daff5f53c0aff8973191ffbbec00; path=/; expires=30
> min
> Content-Length: 31176
> I don't think that the handler is doing anything dodgy. Something is
> happening in mongrel2.
>
> Regards,
> Henry
>
> > On Thu, Apr 28, 2011 at 8:56 AM, Henry Baragar <
> >
> > Henry.Baragar@instantiated.ca> wrote:
> > > On April 27, 2011 06:08:52 pm Ryan Kelly wrote:
> > > > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote:
> > > > > Hello,
> > > > >
> > > > >
> > > > >
> > > > > Is there anything special that needs to be done when sending binary
> > > > >
> > > > > data in a response?
> > > >
> > > > There shouldn't be, all data is treated as binary by Mongrel2.
> > >
> > > That is what I had thought.
> > >
> > > > > In my application, the binary data in a response is being corrupted.
> > > > > I
> > > > >
> > > > > have checked, using tcpdump, that the binary data is uncorrupted in
> > > > >
> > > > > the ZeroMQ packets being sent from my application to Mongrel.
> > > >
> > > > Have you sniffed it coming out of Mongrel2, and is it corrupted there?
> > >
> > > I had only looked at things with a web browser (or two). I just did a
> > > tcpdump of the output and it definitely is corrupted. For one thing, the
> > > output is almost 50% larger than the input.
> > >
> > > The kind of corruption I see looks similar to what happens when a binary
> > > stream is UTF-8 encoded.
> > >
> > > > Or is the application getting confused about e.g. content-length or
> > > > mime
> > > >
> > > > type and corrupting the data internally?
> > >
> > > The application is reporting the correct mime-type (application/pdf) and
> > > content-length.
> > >
> > > Any other suggestions?
> > >
> > > Henry
> > >
> > > > Cheers,
> > > >
> > > >
> > > >
> > > > Ryan
> > >
> > > --
> > >
> > > Henry Baragar
> > >
> > > Instantiated Software
>
> --
> Henry Baragar
> Instantiated Software
On Wed, Apr 27, 2011 at 07:35:41PM -0700, Jason Miller wrote: > I created a simple test-handler that sends bytes from 0-255 in the > reply. This is using my own handler library that is on github > (jasom/mymongrel2) but I'm pretty sure nobody else is using. > > This did expose a flaw in my backend that I will have to fix (Apparently > the common-lisp zmq bindings default to UTF-8 for encoding) Hmmm, I may want to test this in a few handlers as well. I know the mp3stream handler test sends binary data in a complex way but it's not in the regular test suite. -- Zed A. Shaw http://zedshaw.com/
Apparently perl stores strings as utf-8 so this could be an issue there too without delving too far into the zmq binding code. On Thu, Apr 28, 2011 at 12:35 PM, Jason Miller <jason@milr.com> wrote: > I created a simple test-handler that sends bytes from 0-255 in the > reply. This is using my own handler library that is on github > (jasom/mymongrel2) but I'm pretty sure nobody else is using. > > This did expose a flaw in my backend that I will have to fix (Apparently > the common-lisp zmq bindings default to UTF-8 for encoding) > > However, once I was sending the correct data to mongrel2, it came out > clean, as you can see from the following hex-dump. > > This would suggest that the problem is in th > 0000000: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f ................ > 0000010: 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f ................ > 0000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f !"#$%&'()*+,-./ > 0000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f 0123456789:;<=>? > 0000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f @ABCDEFGHIJKLMNO > 0000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f PQRSTUVWXYZ[\]^_ > 0000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f `abcdefghijklmno > 0000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e7f pqrstuvwxyz{|}~. > 0000080: 8081 8283 8485 8687 8889 8a8b 8c8d 8e8f ................ > 0000090: 9091 9293 9495 9697 9899 9a9b 9c9d 9e9f ................ > 00000a0: a0a1 a2a3 a4a5 a6a7 a8a9 aaab acad aeaf ................ > 00000b0: b0b1 b2b3 b4b5 b6b7 b8b9 babb bcbd bebf ................ > 00000c0: c0c1 c2c3 c4c5 c6c7 c8c9 cacb cccd cecf ................ > 00000d0: d0d1 d2d3 d4d5 d6d7 d8d9 dadb dcdd dedf ................ > 00000e0: e0e1 e2e3 e4e5 e6e7 e8e9 eaeb eced eeef ................ > 00000f0: f0f1 f2f3 f4f5 f6f7 f8f9 fafb fcfd feff ................ > > On 19:26 Wed 27 Apr , Henry Baragar wrote: > > On April 27, 2011 07:09:10 pm joshua simmons wrote: > > > Is your handler library doing something dodgy and handling it as utf-8 > > > per-chance? > > > > > > > If we take a look a the the data path: > > > > Application => Mongrel2 => Web browser > > > > We find that the binary data is fine on the first link (represented by > "=>") > > but not the second. > > > > Hear are the headers being sent to the Web browser: > > HTTP/1.1 200 OK > > Content-Type: application/pdf > > Expires: Wed, 27 Apr 2011 22:45:45 GMT > > Set-Cookie: JIFTY_SID_80=7178daff5f53c0aff8973191ffbbec00; path=/; > expires=30 > > min > > Content-Length: 31176 > > I don't think that the handler is doing anything dodgy. Something is > > happening in mongrel2. > > > > Regards, > > Henry > > > > > On Thu, Apr 28, 2011 at 8:56 AM, Henry Baragar < > > > > > > Henry.Baragar@instantiated.ca> wrote: > > > > On April 27, 2011 06:08:52 pm Ryan Kelly wrote: > > > > > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > Is there anything special that needs to be done when sending > binary > > > > > > > > > > > > data in a response? > > > > > > > > > > There shouldn't be, all data is treated as binary by Mongrel2. > > > > > > > > That is what I had thought. > > > > > > > > > > In my application, the binary data in a response is being > corrupted. > > > > > > I > > > > > > > > > > > > have checked, using tcpdump, that the binary data is uncorrupted > in > > > > > > > > > > > > the ZeroMQ packets being sent from my application to Mongrel. > > > > > > > > > > Have you sniffed it coming out of Mongrel2, and is it corrupted > there? > > > > > > > > I had only looked at things with a web browser (or two). I just did a > > > > tcpdump of the output and it definitely is corrupted. For one thing, > the > > > > output is almost 50% larger than the input. > > > > > > > > The kind of corruption I see looks similar to what happens when a > binary > > > > stream is UTF-8 encoded. > > > > > > > > > Or is the application getting confused about e.g. content-length or > > > > > mime > > > > > > > > > > type and corrupting the data internally? > > > > > > > > The application is reporting the correct mime-type (application/pdf) > and > > > > content-length. > > > > > > > > Any other suggestions? > > > > > > > > Henry > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Ryan > > > > > > > > -- > > > > > > > > Henry Baragar > > > > > > > > Instantiated Software > > > > -- > > Henry Baragar > > Instantiated Software > >
On April 27, 2011 11:05:04 pm joshua simmons wrote: > Apparently perl stores strings as utf-8 so this could be an issue there too > without delving too far into the zmq binding code. > How is it communicating this information to mongrel2 and how do I turn it off? If you recall, the tcpdump showed that the data stream was fine between the application and mongrel2 and corrupted between mongrel2 and the browser. Regards, Henry > On Thu, Apr 28, 2011 at 12:35 PM, Jason Miller <jason@milr.com> wrote: > > I created a simple test-handler that sends bytes from 0-255 in the > > reply. This is using my own handler library that is on github > > (jasom/mymongrel2) but I'm pretty sure nobody else is using. > > > > This did expose a flaw in my backend that I will have to fix (Apparently > > the common-lisp zmq bindings default to UTF-8 for encoding) > > > > However, once I was sending the correct data to mongrel2, it came out > > clean, as you can see from the following hex-dump. > > > > This would suggest that the problem is in th > > 0000000: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f ................ > > 0000010: 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f ................ > > 0000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f !"#$%&'()*+,-./ > > 0000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f 0123456789:;<=>? > > 0000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f @ABCDEFGHIJKLMNO > > 0000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f PQRSTUVWXYZ[\]^_ > > 0000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f `abcdefghijklmno > > 0000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e7f pqrstuvwxyz{|}~. > > 0000080: 8081 8283 8485 8687 8889 8a8b 8c8d 8e8f ................ > > 0000090: 9091 9293 9495 9697 9899 9a9b 9c9d 9e9f ................ > > 00000a0: a0a1 a2a3 a4a5 a6a7 a8a9 aaab acad aeaf ................ > > 00000b0: b0b1 b2b3 b4b5 b6b7 b8b9 babb bcbd bebf ................ > > 00000c0: c0c1 c2c3 c4c5 c6c7 c8c9 cacb cccd cecf ................ > > 00000d0: d0d1 d2d3 d4d5 d6d7 d8d9 dadb dcdd dedf ................ > > 00000e0: e0e1 e2e3 e4e5 e6e7 e8e9 eaeb eced eeef ................ > > 00000f0: f0f1 f2f3 f4f5 f6f7 f8f9 fafb fcfd feff ................ > > > > On 19:26 Wed 27 Apr , Henry Baragar wrote: > > > On April 27, 2011 07:09:10 pm joshua simmons wrote: > > > > Is your handler library doing something dodgy and handling it as > > > > utf-8 per-chance? > > > > > > If we take a look a the the data path: > > > Application => Mongrel2 => Web browser > > > > > > We find that the binary data is fine on the first link (represented by > > > > "=>") > > > > > but not the second. > > > > > > Hear are the headers being sent to the Web browser: > > > HTTP/1.1 200 OK > > > Content-Type: application/pdf > > > Expires: Wed, 27 Apr 2011 22:45:45 GMT > > > Set-Cookie: JIFTY_SID_80=7178daff5f53c0aff8973191ffbbec00; path=/; > > > > expires=30 > > > > > min > > > Content-Length: 31176 > > > I don't think that the handler is doing anything dodgy. Something is > > > happening in mongrel2. > > > > > > Regards, > > > Henry > > > > > > > On Thu, Apr 28, 2011 at 8:56 AM, Henry Baragar < > > > > > > > > Henry.Baragar@instantiated.ca> wrote: > > > > > On April 27, 2011 06:08:52 pm Ryan Kelly wrote: > > > > > > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Is there anything special that needs to be done when sending > > > > binary > > > > > > > > > data in a response? > > > > > > > > > > > > There shouldn't be, all data is treated as binary by Mongrel2. > > > > > > > > > > That is what I had thought. > > > > > > > > > > > > In my application, the binary data in a response is being > > > > corrupted. > > > > > > > > > I > > > > > > > > > > > > > > have checked, using tcpdump, that the binary data is > > > > > > > uncorrupted > > > > in > > > > > > > > > the ZeroMQ packets being sent from my application to Mongrel. > > > > > > > > > > > > Have you sniffed it coming out of Mongrel2, and is it corrupted > > > > there? > > > > > > > I had only looked at things with a web browser (or two). I just did > > > > > a tcpdump of the output and it definitely is corrupted. For one > > > > > thing, > > > > the > > > > > > > output is almost 50% larger than the input. > > > > > > > > > > The kind of corruption I see looks similar to what happens when a > > > > binary > > > > > > > stream is UTF-8 encoded. > > > > > > > > > > > Or is the application getting confused about e.g. content-length > > > > > > or mime > > > > > > > > > > > > type and corrupting the data internally? > > > > > > > > > > The application is reporting the correct mime-type > > > > > (application/pdf) > > > > and > > > > > > > content-length. > > > > > > > > > > Any other suggestions? > > > > > > > > > > Henry > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > Ryan > > > > > > > > > > -- > > > > > > > > > > Henry Baragar > > > > > > > > > > Instantiated Software > > > > > > -- > > > Henry Baragar > > > Instantiated Software -- Henry Baragar Instantiated Software
If m2 is breaking it somehow, it's certainly a bug. But in order to have a better look we really need a test case to work with. I keep suggesting the handler library because it's the most obvious point of failure, and as we see with Jason's test, m2 should be handling it fine. Thanks, Josh. On Thu, Apr 28, 2011 at 1:13 PM, Henry Baragar < Henry.Baragar@instantiated.ca> wrote: > On April 27, 2011 11:05:04 pm joshua simmons wrote: > > > Apparently perl stores strings as utf-8 so this could be an issue there > too > > > without delving too far into the zmq binding code. > > > > > How is it communicating this information to mongrel2 and how do I turn it > off? > > If you recall, the tcpdump showed that the data stream was fine between the > application and mongrel2 and corrupted between mongrel2 and the browser. > > Regards, > > Henry > > > On Thu, Apr 28, 2011 at 12:35 PM, Jason Miller <jason@milr.com> wrote: > > > > I created a simple test-handler that sends bytes from 0-255 in the > > > > reply. This is using my own handler library that is on github > > > > (jasom/mymongrel2) but I'm pretty sure nobody else is using. > > > > > > > > This did expose a flaw in my backend that I will have to fix > (Apparently > > > > the common-lisp zmq bindings default to UTF-8 for encoding) > > > > > > > > However, once I was sending the correct data to mongrel2, it came out > > > > clean, as you can see from the following hex-dump. > > > > > > > > This would suggest that the problem is in th > > > > 0000000: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f ................ > > > > 0000010: 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f ................ > > > > 0000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f !"#$%&'()*+,-./ > > > > 0000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f 0123456789:;<=>? > > > > 0000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f @ABCDEFGHIJKLMNO > > > > 0000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f PQRSTUVWXYZ[\]^_ > > > > 0000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f `abcdefghijklmno > > > > 0000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e7f pqrstuvwxyz{|}~. > > > > 0000080: 8081 8283 8485 8687 8889 8a8b 8c8d 8e8f ................ > > > > 0000090: 9091 9293 9495 9697 9899 9a9b 9c9d 9e9f ................ > > > > 00000a0: a0a1 a2a3 a4a5 a6a7 a8a9 aaab acad aeaf ................ > > > > 00000b0: b0b1 b2b3 b4b5 b6b7 b8b9 babb bcbd bebf ................ > > > > 00000c0: c0c1 c2c3 c4c5 c6c7 c8c9 cacb cccd cecf ................ > > > > 00000d0: d0d1 d2d3 d4d5 d6d7 d8d9 dadb dcdd dedf ................ > > > > 00000e0: e0e1 e2e3 e4e5 e6e7 e8e9 eaeb eced eeef ................ > > > > 00000f0: f0f1 f2f3 f4f5 f6f7 f8f9 fafb fcfd feff ................ > > > > > > > > On 19:26 Wed 27 Apr , Henry Baragar wrote: > > > > > On April 27, 2011 07:09:10 pm joshua simmons wrote: > > > > > > Is your handler library doing something dodgy and handling it as > > > > > > utf-8 per-chance? > > > > > > > > > > If we take a look a the the data path: > > > > > Application => Mongrel2 => Web browser > > > > > > > > > > We find that the binary data is fine on the first link (represented > by > > > > > > > > "=>") > > > > > > > > > but not the second. > > > > > > > > > > Hear are the headers being sent to the Web browser: > > > > > HTTP/1.1 200 OK > > > > > Content-Type: application/pdf > > > > > Expires: Wed, 27 Apr 2011 22:45:45 GMT > > > > > Set-Cookie: JIFTY_SID_80=7178daff5f53c0aff8973191ffbbec00; path=/; > > > > > > > > expires=30 > > > > > > > > > min > > > > > Content-Length: 31176 > > > > > I don't think that the handler is doing anything dodgy. Something is > > > > > happening in mongrel2. > > > > > > > > > > Regards, > > > > > Henry > > > > > > > > > > > On Thu, Apr 28, 2011 at 8:56 AM, Henry Baragar < > > > > > > > > > > > > Henry.Baragar@instantiated.ca> wrote: > > > > > > > On April 27, 2011 06:08:52 pm Ryan Kelly wrote: > > > > > > > > On Wed, 2011-04-27 at 17:53 -0400, Henry Baragar wrote: > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Is there anything special that needs to be done when sending > > > > > > > > binary > > > > > > > > > > > > > data in a response? > > > > > > > > > > > > > > > > There shouldn't be, all data is treated as binary by Mongrel2. > > > > > > > > > > > > > > That is what I had thought. > > > > > > > > > > > > > > > > In my application, the binary data in a response is being > > > > > > > > corrupted. > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > have checked, using tcpdump, that the binary data is > > > > > > > > > uncorrupted > > > > > > > > in > > > > > > > > > > > > > the ZeroMQ packets being sent from my application to Mongrel. > > > > > > > > > > > > > > > > Have you sniffed it coming out of Mongrel2, and is it corrupted > > > > > > > > there? > > > > > > > > > > > I had only looked at things with a web browser (or two). I just > did > > > > > > > a tcpdump of the output and it definitely is corrupted. For one > > > > > > > thing, > > > > > > > > the > > > > > > > > > > > output is almost 50% larger than the input. > > > > > > > > > > > > > > The kind of corruption I see looks similar to what happens when a > > > > > > > > binary > > > > > > > > > > > stream is UTF-8 encoded. > > > > > > > > > > > > > > > Or is the application getting confused about e.g. > content-length > > > > > > > > or mime > > > > > > > > > > > > > > > > type and corrupting the data internally? > > > > > > > > > > > > > > The application is reporting the correct mime-type > > > > > > > (application/pdf) > > > > > > > > and > > > > > > > > > > > content-length. > > > > > > > > > > > > > > Any other suggestions? > > > > > > > > > > > > > > Henry > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ryan > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Henry Baragar > > > > > > > > > > > > > > Instantiated Software > > > > > > > > > > -- > > > > > Henry Baragar > > > > > Instantiated Software > > > -- > > Henry Baragar > > Instantiated Software > >
On Wed, Apr 27, 2011 at 11:13:29PM -0400, Henry Baragar wrote: > On April 27, 2011 11:05:04 pm joshua simmons wrote: > > Apparently perl stores strings as utf-8 so this could be an issue there too > > without delving too far into the zmq binding code. > > > > How is it communicating this information to mongrel2 and how do I turn it off? > > If you recall, the tcpdump showed that the data stream was fine between the > application and mongrel2 and corrupted between mongrel2 and the browser. Well, mongrel2 literally takes the message raw bytes from zeromq and writes it to the straight to the socket. There's a way to dump what mongrel2 is writing to the socket, but I'll have to go look at the code for you to see the best place to put the debug message. Also, there's an older examples/mp3stream that streams out mp3s using the ICY protocol. I'll dust that off and make sure it works today as a confirmation that mongrel2 isn't destroying anything. -- Zed A. Shaw http://zedshaw.com/
On 23:13 Wed 27 Apr , Henry Baragar wrote: > On April 27, 2011 11:05:04 pm joshua simmons wrote: > > Apparently perl stores strings as utf-8 so this could be an issue there too > > without delving too far into the zmq binding code. > > > > How is it communicating this information to mongrel2 and how do I turn it off? > > If you recall, the tcpdump showed that the data stream was fine between the > application and mongrel2 and corrupted between mongrel2 and the browser. Can you double-check this? It makes no sense at all since mongrel2 literally just pushes the bytes it receives directly to the TCP socket. I did a more complete test-case in common-lisp with a pdf and it went through unmolested. I'd try it with the perl bindings, but my perl-fu is weak. -Jason
On April 28, 2011 12:20:23 am Jason Miller wrote: > On 23:13 Wed 27 Apr , Henry Baragar wrote: > > On April 27, 2011 11:05:04 pm joshua simmons wrote: > > > Apparently perl stores strings as utf-8 so this could be an issue there > > > too without delving too far into the zmq binding code. > > > > How is it communicating this information to mongrel2 and how do I turn it > > off? > > > > If you recall, the tcpdump showed that the data stream was fine between > > the application and mongrel2 and corrupted between mongrel2 and the > > browser. > > Can you double-check this? It makes no sense at all since mongrel2 > literally just pushes the bytes it receives directly to the TCP socket. > I don't seem to be able to recreate the data that I had yesterday. In today's tests the data sent by mongrel2 is exactly the data that it receives. That is, the data is being corrupted before it gets to mongrel2. Everyone, please accept my apologies for this noise on the list, especially those of you who did some testing on my behalf. Regards, Henry > I did a more complete test-case in common-lisp with a pdf and it went > through unmolested. I'd try it with the perl bindings, but my perl-fu > is weak. > -Jason -- Henry Baragar Instantiated Software
On Thu, Apr 28, 2011 at 12:26:03PM -0400, Henry Baragar wrote: > > Can you double-check this? It makes no sense at all since mongrel2 > > literally just pushes the bytes it receives directly to the TCP socket. > > > > I don't seem to be able to recreate the data that I had yesterday. In today's > tests the data sent by mongrel2 is exactly the data that it receives. That > is, the data is being corrupted before it gets to mongrel2. > > Everyone, please accept my apologies for this noise on the list, especially > those of you who did some testing on my behalf. Hehe, shit happens. :-) Anyway, check my other email about some debugging stuff you can do. -- Zed A. Shaw http://zedshaw.com/
On April 28, 2011 12:29:44 pm Zed A. Shaw wrote: > On Thu, Apr 28, 2011 at 12:26:03PM -0400, Henry Baragar wrote: > > > Can you double-check this? It makes no sense at all since mongrel2 > > > literally just pushes the bytes it receives directly to the TCP socket. > > > > I don't seem to be able to recreate the data that I had yesterday. In > > today's tests the data sent by mongrel2 is exactly the data that it > > receives. That is, the data is being corrupted before it gets to > > mongrel2. > > > > Everyone, please accept my apologies for this noise on the list, > > especially those of you who did some testing on my behalf. > > Hehe, shit happens. :-) > > Anyway, check my other email about some debugging stuff you can do. Looks like I won't be needing that stuff:) Just in case someone finds this thread in the archive and wants to know the resolution, here is a quick update. Indeed, the Plack::Handler::Mongrel2 perl module was UTF-8 encoding binary data. I fixed this problem, along with a couple others I uncovered and put the changed module to https://github.com/hbaragar/Plack-Handler-Mongrel2, which will exist until Daisuke incorporates them into his fork and uploads the package to CPAN. At this point, all my Perl module issues have been resolved and I have a working application. Now onto SSL. Thanks, Henry -- Henry Baragar Instantiated Software
On 16:49 Thu 28 Apr , Henry Baragar wrote: > > At this point, all my Perl module issues have been resolved and I have a > working application. > > Now onto SSL. > May I suggest using stunnel for SSL? It's fast and pretty easy to setup if it's in your distro's package manager. -Jason
On April 29, 2011 12:02:30 am Jason Miller wrote: > On 16:49 Thu 28 Apr , Henry Baragar wrote: > > At this point, all my Perl module issues have been resolved and I have a > > working application. > > > > Now onto SSL. > > May I suggest using stunnel for SSL? It's fast and pretty easy to setup > if it's in your distro's package manager. > I meant SSL for the client browsers (i.e. https). Is that what you meant? Henry > -Jason -- Henry Baragar Instantiated Software
On Fri, Apr 29, 2011 at 12:12:25AM -0400, Henry Baragar wrote: > > May I suggest using stunnel for SSL? It's fast and pretty easy to setup > > if it's in your distro's package manager. > > > > I meant SSL for the client browsers (i.e. https). Is that what you meant? Yep, I used to think stunnel was slow and crap, but these days it's looking like most SSL is done with a second process, terminating proxy, or some other finely tuned SSL processing machine outside the web server. -- Zed A. Shaw http://zedshaw.com/
On April 29, 2011 02:27:28 AM Zed A. Shaw wrote: > On Fri, Apr 29, 2011 at 12:12:25AM -0400, Henry Baragar wrote: > > > May I suggest using stunnel for SSL? It's fast and pretty easy to > > > setup if it's in your distro's package manager. > > > > I meant SSL for the client browsers (i.e. https). Is that what you > > meant? > > Yep, I used to think stunnel was slow and crap, but these days it's > looking like most SSL is done with a second process, terminating proxy, > or some other finely tuned SSL processing machine outside the web > server. So, what is the recommended configuration for mongrel2? Stunnel, or the barely documented SSL built into mongrel2 (http://bnpcs.blogspot.com/2011/01/mongrel2-and-ssl.html)? Henry -- Henry Baragar Instantiated Software
On 00:12 Fri 29 Apr , Henry Baragar wrote: > On April 29, 2011 12:02:30 am Jason Miller wrote: > > On 16:49 Thu 28 Apr , Henry Baragar wrote: > > > At this point, all my Perl module issues have been resolved and I have a > > > working application. > > > > > > Now onto SSL. > > > > May I suggest using stunnel for SSL? It's fast and pretty easy to setup > > if it's in your distro's package manager. > > > > I meant SSL for the client browsers (i.e. https). Is that what you meant? Yes
On April 29, 2011 12:45:23 AM Jason Miller wrote: > On 00:12 Fri 29 Apr , Henry Baragar wrote: > > On April 29, 2011 12:02:30 am Jason Miller wrote: > > > On 16:49 Thu 28 Apr , Henry Baragar wrote: > > > > At this point, all my Perl module issues have been resolved and I > > > > have a working application. > > > > > > > > Now onto SSL. > > > > > > May I suggest using stunnel for SSL? It's fast and pretty easy to > > > setup if it's in your distro's package manager. > > > > I meant SSL for the client browsers (i.e. https). Is that what you > > meant? > > Yes Interesting. As you said, it was pretty easy to set up and get working (other than I don't know how to log client IP addresses). I think I like it. Thanks of the suggestion. Regards, Henry -- Henry Baragar Instantiated Software
On Wed, Apr 27, 2011 at 4:26 PM, Henry Baragar <Henry.Baragar@instantiated.ca> wrote: > I don't think that the handler is doing anything dodgy. Something is > happening in mongrel2. The best way to make this argument is to accompany it with a minimal test case that reproduces your problem.