librelist archives

« back to archive

Proposal: Iterator for the response body

Proposal: Iterator for the response body

From:
Loic d'Anterroches
Date:
2011-06-05 @ 12:16
Hello,

At the moment, I have a toy project for Indefero 2, that is, a git http
backend support with Photon. Yes, I can pull/push to a git repository
served by Photon.

The thing is, the current approach to provide the response body as a
string is not working when you push 20MB of data down the pipe. So, I am
looking at an elegant way to solve the problem.

At the moment we have as most common case:

1. request from mongrel2
2. middleware
3. view generates response with full body
4. middleware in reverse
5. on message over zmq for mongrel2

The steps 4. and 5. can be skipped when answering false. In this case,
Photon itself will not send any answer, you have to do that from your
view of from another component.

So, to serve a large answer (the large request is already available as a
stream, so we are memory aware), you need at the moment to follow this
"skipped" route.

What I want is to be able to do this more intuitively, by passing an
iterable as the content of the answer.

You pass an iterator to the content of the view, this means that at
render time, each iteration will result in one zmq message to mongrel2,
so this is for large chunks, not for line by line answers. An iterable
can be a simple array() but it more to be able to stream a file from a
mongodb backend, the large output of a git command, etc.

+ you can send basically everything
+ we keep the current middleware stack
+ you can inspect/update the headers of the answer as usual (Vary,
  Cache, etc.) in the middleware
- you do not have the full answer in the middleware stack return path
- you monopalize an handler process while you answer

I think we can do that without change at the code, that is, if
$response->content is a string, basically generate only one Mongrel2
message.

This would just add flexibility "as needed". Feedback welcome!

loïc

--
Indefero - Project management and code hosting - http://www.indefero.net
Photon - High Performance PHP Framework - http://photon-project.com
Céondo Ltd - Web + Science = Fun - http://www.ceondo.com

Re: [photon.users] Proposal: Iterator for the response body

From:
Mickaël Desfrênes
Date:
2011-06-05 @ 12:24
I asked a similar question on the AiP list and someone answered that it was
possible return a php stream as the response content. It works but still I
think an Iterator is easier to implement at the application level than a
stream (or maybe I just know the iterator interface better than php
streams).

2011/6/5 Loic d'Anterroches <loic@ceondo.com>

> Hello,
>
> At the moment, I have a toy project for Indefero 2, that is, a git http
> backend support with Photon. Yes, I can pull/push to a git repository
> served by Photon.
>
> The thing is, the current approach to provide the response body as a
> string is not working when you push 20MB of data down the pipe. So, I am
> looking at an elegant way to solve the problem.
>
> At the moment we have as most common case:
>
> 1. request from mongrel2
> 2. middleware
> 3. view generates response with full body
> 4. middleware in reverse
> 5. on message over zmq for mongrel2
>
> The steps 4. and 5. can be skipped when answering false. In this case,
> Photon itself will not send any answer, you have to do that from your
> view of from another component.
>
> So, to serve a large answer (the large request is already available as a
> stream, so we are memory aware), you need at the moment to follow this
> "skipped" route.
>
> What I want is to be able to do this more intuitively, by passing an
> iterable as the content of the answer.
>
> You pass an iterator to the content of the view, this means that at
> render time, each iteration will result in one zmq message to mongrel2,
> so this is for large chunks, not for line by line answers. An iterable
> can be a simple array() but it more to be able to stream a file from a
> mongodb backend, the large output of a git command, etc.
>
> + you can send basically everything
> + we keep the current middleware stack
> + you can inspect/update the headers of the answer as usual (Vary,
>  Cache, etc.) in the middleware
> - you do not have the full answer in the middleware stack return path
> - you monopalize an handler process while you answer
>
> I think we can do that without change at the code, that is, if
> $response->content is a string, basically generate only one Mongrel2
> message.
>
> This would just add flexibility "as needed". Feedback welcome!
>
> loïc
>
> --
> Indefero - Project management and code hosting - http://www.indefero.net
> Photon - High Performance PHP Framework - http://photon-project.com
> Céondo Ltd - Web + Science = Fun - http://www.ceondo.com
>



--

Re: [photon.users] Proposal: Iterator for the response body

From:
Loic d'Anterroches
Date:
2011-06-05 @ 12:32

On 2011-06-05 14:24, Mickaël Desfrênes wrote:
> I asked a similar question on the AiP list and someone answered that it
> was possible return a php stream as the response content. It works but
> still I think an Iterator is easier to implement at the application
> level than a stream (or maybe I just know the iterator interface better
> than php streams).

You can very easily wrap a stream in an iterator, in fact, this is what
I want to do for my git stuff. The reverse is not true, if your
application is not getting a stream, for example you get data in chunks
of 256kb from a mongodb filefs storage, you are dead, because you need
first to pull everything from mongodb and push that into a tmp stream
(possibly with a swap) and then pass it to AiP or whatever system you
have. Not so good.

loïc

> 2011/6/5 Loic d'Anterroches <loic@ceondo.com <mailto:loic@ceondo.com>>
> 
>     Hello,
> 
>     At the moment, I have a toy project for Indefero 2, that is, a git http
>     backend support with Photon. Yes, I can pull/push to a git repository
>     served by Photon.
> 
>     The thing is, the current approach to provide the response body as a
>     string is not working when you push 20MB of data down the pipe. So, I am
>     looking at an elegant way to solve the problem.
> 
>     At the moment we have as most common case:
> 
>     1. request from mongrel2
>     2. middleware
>     3. view generates response with full body
>     4. middleware in reverse
>     5. on message over zmq for mongrel2
> 
>     The steps 4. and 5. can be skipped when answering false. In this case,
>     Photon itself will not send any answer, you have to do that from your
>     view of from another component.
> 
>     So, to serve a large answer (the large request is already available as a
>     stream, so we are memory aware), you need at the moment to follow this
>     "skipped" route.
> 
>     What I want is to be able to do this more intuitively, by passing an
>     iterable as the content of the answer.
> 
>     You pass an iterator to the content of the view, this means that at
>     render time, each iteration will result in one zmq message to mongrel2,
>     so this is for large chunks, not for line by line answers. An iterable
>     can be a simple array() but it more to be able to stream a file from a
>     mongodb backend, the large output of a git command, etc.
> 
>     + you can send basically everything
>     + we keep the current middleware stack
>     + you can inspect/update the headers of the answer as usual (Vary,
>      Cache, etc.) in the middleware
>     - you do not have the full answer in the middleware stack return path
>     - you monopalize an handler process while you answer
> 
>     I think we can do that without change at the code, that is, if
>     $response->content is a string, basically generate only one Mongrel2
>     message.
> 
>     This would just add flexibility "as needed". Feedback welcome!
> 
>     loïc
> 
>     --
>     Indefero - Project management and code hosting - http://www.indefero.net
>     Photon - High Performance PHP Framework - http://photon-project.com
>     Céondo Ltd - Web + Science = Fun - http://www.ceondo.com
> 
> 
> 
> 
> -- 
> 

Re: [photon.users] Proposal: Iterator for the response body

From:
Loic d'Anterroches
Date:
2011-06-05 @ 13:04

On 2011-06-05 14:32, Loic d'Anterroches wrote:
> 
> 
> On 2011-06-05 14:24, Mickaël Desfrênes wrote:
>> I asked a similar question on the AiP list and someone answered that it
>> was possible return a php stream as the response content. It works but
>> still I think an Iterator is easier to implement at the application
>> level than a stream (or maybe I just know the iterator interface better
>> than php streams).
> 
> You can very easily wrap a stream in an iterator, in fact, this is what
> I want to do for my git stuff. The reverse is not true, if your
> application is not getting a stream, for example you get data in chunks
> of 256kb from a mongodb filefs storage, you are dead, because you need
> first to pull everything from mongodb and push that into a tmp stream
> (possibly with a swap) and then pass it to AiP or whatever system you
> have. Not so good.

Error, you can create a special stream wrapper which you register, you
then can have a custom stream accessing your data. But I must say, this
looks very complicated.

loïc

>> 2011/6/5 Loic d'Anterroches <loic@ceondo.com <mailto:loic@ceondo.com>>
>>
>>     Hello,
>>
>>     At the moment, I have a toy project for Indefero 2, that is, a git http
>>     backend support with Photon. Yes, I can pull/push to a git repository
>>     served by Photon.
>>
>>     The thing is, the current approach to provide the response body as a
>>     string is not working when you push 20MB of data down the pipe. So, I am
>>     looking at an elegant way to solve the problem.
>>
>>     At the moment we have as most common case:
>>
>>     1. request from mongrel2
>>     2. middleware
>>     3. view generates response with full body
>>     4. middleware in reverse
>>     5. on message over zmq for mongrel2
>>
>>     The steps 4. and 5. can be skipped when answering false. In this case,
>>     Photon itself will not send any answer, you have to do that from your
>>     view of from another component.
>>
>>     So, to serve a large answer (the large request is already available as a
>>     stream, so we are memory aware), you need at the moment to follow this
>>     "skipped" route.
>>
>>     What I want is to be able to do this more intuitively, by passing an
>>     iterable as the content of the answer.
>>
>>     You pass an iterator to the content of the view, this means that at
>>     render time, each iteration will result in one zmq message to mongrel2,
>>     so this is for large chunks, not for line by line answers. An iterable
>>     can be a simple array() but it more to be able to stream a file from a
>>     mongodb backend, the large output of a git command, etc.
>>
>>     + you can send basically everything
>>     + we keep the current middleware stack
>>     + you can inspect/update the headers of the answer as usual (Vary,
>>      Cache, etc.) in the middleware
>>     - you do not have the full answer in the middleware stack return path
>>     - you monopalize an handler process while you answer
>>
>>     I think we can do that without change at the code, that is, if
>>     $response->content is a string, basically generate only one Mongrel2
>>     message.
>>
>>     This would just add flexibility "as needed". Feedback welcome!
>>
>>     loïc
>>
>>     --
>>     Indefero - Project management and code hosting - http://www.indefero.net
>>     Photon - High Performance PHP Framework - http://photon-project.com
>>     Céondo Ltd - Web + Science = Fun - http://www.ceondo.com
>>
>>
>>
>>
>> --