I have just released version 0.2 of flask-mail on PyPi. This is mostly some bug and documentation fixes, plus bulk email sending functionality (reusing the same SMTP connection for multiple email messages).
The send_file() method in Flask has support for "X-Sendfile" HTTP header (the "use_x_sendfile" attribute) - this is the recommended method to deliver static files efficiently under compliant webservers such as apache, lighttpd & cherokee. Nginx also has this feature, but implemented slightly differently: in Nginx this feature is named "X-Accel-Redirect". Hence, setting the Flask.use_x_sendfile attribute to True will not work on Nginx. I have added an attribute named "use_x_accel_redirect" to Flask: http://github.com/masroore/flask/ If your webapp is running under Nginx, set the "Flask.use_x_accel_redirect" (or USE_X_ACCEL_REDIRECT) attribute to True. (Cherokee has support for both "X-Sendfile" and "X-Accel-Redirect" - so this patch isn't really necessary) The changes are pretty trivial - but important. I hope these patches will find their way into the official distribution. Best regards. - Masroor Ehsan
Hi, On 6/17/10 5:58 PM, Dr Masroor Ehsan wrote: > Nginx also has this feature, but implemented slightly differently: in > Nginx this feature is named "X-Accel-Redirect". Hence, setting the > Flask.use_x_sendfile attribute to True will not work on Nginx. If the behavior is otherwise the same I strongly recommend instead of adding this to Flask that a WSGI middleware is used to correct the header for Nginx. This could be added to the werkzeug.fixers package which already contains a couple of other hacks for Lighttpd and nginx that fix a few inconsistencies and bugs. Regards, Armin
Hi again, On 6/17/10 6:02 PM, Armin Ronacher wrote: > If the behavior is otherwise the same I strongly recommend instead of > adding this to Flask that a WSGI middleware is used to correct the > header for Nginx. Just checked back with the nginx docs. It appears that nginx does not use actual filenames on the filesystem but instead URIs with mapped paths on the server. So the correct usage would be that one defines an internal "/internal" path that is the alias for a folder somewhere on the filesystem. This however is totally different to how X-Sendfile works so it's quite hard to implement this in Flask in a reusable manner. I strongly recommend that people add a custom code path in their own code for nginx setups like these. In related news: someone should write X-Sendfile for nginx :) Regards, Armin
On 06/17/2010 10:02 PM, Armin Ronacher wrote: > Hi, > > On 6/17/10 5:58 PM, Dr Masroor Ehsan wrote: > >> Nginx also has this feature, but implemented slightly differently: in >> Nginx this feature is named "X-Accel-Redirect". Hence, setting the >> Flask.use_x_sendfile attribute to True will not work on Nginx. >> > If the behavior is otherwise the same I strongly recommend instead of > adding this to Flask that a WSGI middleware is used to correct the > header for Nginx. This could be added to the werkzeug.fixers package > which already contains a couple of other hacks for Lighttpd and nginx > that fix a few inconsistencies and bugs. > > > Regards, > Armin > Indeed, the middleware solution sounds more elegant. Details about Nginx's X-Accel-Redirect can be found here: http://wiki.nginx.org/NginxXSendfile Regards, Dr. Masroor Ehsan
On batch email, what about batching the connection? Our SMTP server is through (mt) and only allows 1000 emails to be sent per connection. Obviously I don't want to have to exit the cron script, I would like to be able to close the connection, and reopen another one after a certain number of sent emails. Could this be handled automatically by flask-mail? -- Thadeus On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote: > I have just released version 0.2 of flask-mail on PyPi. > > This is mostly some bug and documentation fixes, plus bulk email > sending functionality (reusing the same SMTP connection for multiple > email messages). >
So basically you would like something like this: BATCH_SIZE = 1000 app.mail_relay.send_many(messages, BATCH_SIZE) Where the connection is restarted each time BATCH_SIZE messages are sent ? On 17 June 2010 05:56, Thadeus Burgess <thadeusb@thadeusb.com> wrote: > On batch email, what about batching the connection? > > Our SMTP server is through (mt) and only allows 1000 emails to be sent > per connection. Obviously I don't want to have to exit the cron > script, I would like to be able to close the connection, and reopen > another one after a certain number of sent emails. Could this be > handled automatically by flask-mail? > > -- > Thadeus > > > > > > On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote: >> I have just released version 0.2 of flask-mail on PyPi. >> >> This is mostly some bug and documentation fixes, plus bulk email >> sending functionality (reusing the same SMTP connection for multiple >> email messages). >> >
I've made a change to the send_many() method to add an additional batch_size parameter: app.mail_relay.send_many(messages, 1000) You can also set a default MAIL_BATCH_SIZE in the configuration. If the number of messages sent reaches this batch size, the connection is automatically closed and re-opened. If someone would like to help test this I'll put it into the next release. On 17 June 2010 06:54, Dan Jacob <danjac354@gmail.com> wrote: > So basically you would like something like this: > > BATCH_SIZE = 1000 > > app.mail_relay.send_many(messages, BATCH_SIZE) > > Where the connection is restarted each time BATCH_SIZE messages are sent ? > > On 17 June 2010 05:56, Thadeus Burgess <thadeusb@thadeusb.com> wrote: >> On batch email, what about batching the connection? >> >> Our SMTP server is through (mt) and only allows 1000 emails to be sent >> per connection. Obviously I don't want to have to exit the cron >> script, I would like to be able to close the connection, and reopen >> another one after a certain number of sent emails. Could this be >> handled automatically by flask-mail? >> >> -- >> Thadeus >> >> >> >> >> >> On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote: >>> I have just released version 0.2 of flask-mail on PyPi. >>> >>> This is mostly some bug and documentation fixes, plus bulk email >>> sending functionality (reusing the same SMTP connection for multiple >>> email messages). >>> >> >
More or less.... init_mail(app, BATCH_SIZE). This way you can continue to use regular msg.send() without having to modify much code. Of course this can be stored in the configuration file as well. -- Thadeus On Thu, Jun 17, 2010 at 1:44 AM, Dan Jacob <danjac354@gmail.com> wrote: > I've made a change to the send_many() method to add an additional > batch_size parameter: > > app.mail_relay.send_many(messages, 1000) > > You can also set a default MAIL_BATCH_SIZE in the configuration. > > If the number of messages sent reaches this batch size, the connection > is automatically closed and re-opened. > > If someone would like to help test this I'll put it into the next release. > > On 17 June 2010 06:54, Dan Jacob <danjac354@gmail.com> wrote: >> So basically you would like something like this: >> >> BATCH_SIZE = 1000 >> >> app.mail_relay.send_many(messages, BATCH_SIZE) >> >> Where the connection is restarted each time BATCH_SIZE messages are sent ? >> >> On 17 June 2010 05:56, Thadeus Burgess <thadeusb@thadeusb.com> wrote: >>> On batch email, what about batching the connection? >>> >>> Our SMTP server is through (mt) and only allows 1000 emails to be sent >>> per connection. Obviously I don't want to have to exit the cron >>> script, I would like to be able to close the connection, and reopen >>> another one after a certain number of sent emails. Could this be >>> handled automatically by flask-mail? >>> >>> -- >>> Thadeus >>> >>> >>> >>> >>> >>> On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote: >>>> I have just released version 0.2 of flask-mail on PyPi. >>>> >>>> This is mostly some bug and documentation fixes, plus bulk email >>>> sending functionality (reusing the same SMTP connection for multiple >>>> email messages). >>>> >>> >> >
See the previous post. The MAIL_BATCH_SIZE can be set as a general rule, but you can override in send_many() on a case-by-case basis. On 17 June 2010 08:28, Thadeus Burgess <thadeusb@thadeusb.com> wrote: > More or less.... init_mail(app, BATCH_SIZE). This way you can continue > to use regular msg.send() without having to modify much code. > > Of course this can be stored in the configuration file as well. > > -- > Thadeus > > > > > > On Thu, Jun 17, 2010 at 1:44 AM, Dan Jacob <danjac354@gmail.com> wrote: >> I've made a change to the send_many() method to add an additional >> batch_size parameter: >> >> app.mail_relay.send_many(messages, 1000) >> >> You can also set a default MAIL_BATCH_SIZE in the configuration. >> >> If the number of messages sent reaches this batch size, the connection >> is automatically closed and re-opened. >> >> If someone would like to help test this I'll put it into the next release. >> >> On 17 June 2010 06:54, Dan Jacob <danjac354@gmail.com> wrote: >>> So basically you would like something like this: >>> >>> BATCH_SIZE = 1000 >>> >>> app.mail_relay.send_many(messages, BATCH_SIZE) >>> >>> Where the connection is restarted each time BATCH_SIZE messages are sent ? >>> >>> On 17 June 2010 05:56, Thadeus Burgess <thadeusb@thadeusb.com> wrote: >>>> On batch email, what about batching the connection? >>>> >>>> Our SMTP server is through (mt) and only allows 1000 emails to be sent >>>> per connection. Obviously I don't want to have to exit the cron >>>> script, I would like to be able to close the connection, and reopen >>>> another one after a certain number of sent emails. Could this be >>>> handled automatically by flask-mail? >>>> >>>> -- >>>> Thadeus >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote: >>>>> I have just released version 0.2 of flask-mail on PyPi. >>>>> >>>>> This is mostly some bug and documentation fixes, plus bulk email >>>>> sending functionality (reusing the same SMTP connection for multiple >>>>> email messages). >>>>> >>>> >>> >> >
Do I have to use send_many or can I just use send? The issue being that to use send_many means I have to load all of my messages into memory first before sending. I would prefer to be able to send as I iterate through my database results. -- Thadeus On Thu, Jun 17, 2010 at 2:31 AM, Dan Jacob <danjac354@gmail.com> wrote: > See the previous post. The MAIL_BATCH_SIZE can be set as a general > rule, but you can override in send_many() on a case-by-case basis. > > On 17 June 2010 08:28, Thadeus Burgess <thadeusb@thadeusb.com> wrote: >> More or less.... init_mail(app, BATCH_SIZE). This way you can continue >> to use regular msg.send() without having to modify much code. >> >> Of course this can be stored in the configuration file as well. >> >> -- >> Thadeus >> >> >> >> >> >> On Thu, Jun 17, 2010 at 1:44 AM, Dan Jacob <danjac354@gmail.com> wrote: >>> I've made a change to the send_many() method to add an additional >>> batch_size parameter: >>> >>> app.mail_relay.send_many(messages, 1000) >>> >>> You can also set a default MAIL_BATCH_SIZE in the configuration. >>> >>> If the number of messages sent reaches this batch size, the connection >>> is automatically closed and re-opened. >>> >>> If someone would like to help test this I'll put it into the next release. >>> >>> On 17 June 2010 06:54, Dan Jacob <danjac354@gmail.com> wrote: >>>> So basically you would like something like this: >>>> >>>> BATCH_SIZE = 1000 >>>> >>>> app.mail_relay.send_many(messages, BATCH_SIZE) >>>> >>>> Where the connection is restarted each time BATCH_SIZE messages are sent ? >>>> >>>> On 17 June 2010 05:56, Thadeus Burgess <thadeusb@thadeusb.com> wrote: >>>>> On batch email, what about batching the connection? >>>>> >>>>> Our SMTP server is through (mt) and only allows 1000 emails to be sent >>>>> per connection. Obviously I don't want to have to exit the cron >>>>> script, I would like to be able to close the connection, and reopen >>>>> another one after a certain number of sent emails. Could this be >>>>> handled automatically by flask-mail? >>>>> >>>>> -- >>>>> Thadeus >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote: >>>>>> I have just released version 0.2 of flask-mail on PyPi. >>>>>> >>>>>> This is mostly some bug and documentation fixes, plus bulk email >>>>>> sending functionality (reusing the same SMTP connection for multiple >>>>>> email messages). >>>>>> >>>>> >>>> >>> >> >
Flask-mail isn't really designed for large volume cronjobs, but rather
for web application usage where you don't tend to send huge numbers of
emails in a request context.
That said, one workaround might be to have a Relay instance which
keeps a connection alive:
from flaskext.mail import Relay
class PersistentRelay(Relay):
def __init__(self, *args, **kwargs):
self.batch_size = kwargs.pop('batch_size', 1000)
super(PersistentRelay, self).__init__(*args, **kwargs)
self.counter = 0
self.host = None
def deliver(self, message, To=None, From=None):
if self.host is None:
self.host = self.configure_relay(self.hostname)
sender = To or message['To']
recipients = From or message['From']
self.host.sendmail(sender,
recipients,
str(response))
self.counter += 1
if self.counter == self.batch_size:
self.host.quit()
self.host = None
self.counter = 0
def close(self):
if self.host is not None:
self.host.quit()
relay = PersistentRelay(...)
msg = Message(...)
msg.send(relay=relay)
relay.quit()
On 17 June 2010 08:40, Thadeus Burgess <thadeusb@thadeusb.com> wrote:
> Do I have to use send_many or can I just use send?
>
> The issue being that to use send_many means I have to load all of my
> messages into memory first before sending. I would prefer to be able
> to send as I iterate through my database results.
>
> --
> Thadeus
>
>
>
>
>
> On Thu, Jun 17, 2010 at 2:31 AM, Dan Jacob <danjac354@gmail.com> wrote:
>> See the previous post. The MAIL_BATCH_SIZE can be set as a general
>> rule, but you can override in send_many() on a case-by-case basis.
>>
>> On 17 June 2010 08:28, Thadeus Burgess <thadeusb@thadeusb.com> wrote:
>>> More or less.... init_mail(app, BATCH_SIZE). This way you can continue
>>> to use regular msg.send() without having to modify much code.
>>>
>>> Of course this can be stored in the configuration file as well.
>>>
>>> --
>>> Thadeus
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jun 17, 2010 at 1:44 AM, Dan Jacob <danjac354@gmail.com> wrote:
>>>> I've made a change to the send_many() method to add an additional
>>>> batch_size parameter:
>>>>
>>>> app.mail_relay.send_many(messages, 1000)
>>>>
>>>> You can also set a default MAIL_BATCH_SIZE in the configuration.
>>>>
>>>> If the number of messages sent reaches this batch size, the connection
>>>> is automatically closed and re-opened.
>>>>
>>>> If someone would like to help test this I'll put it into the next release.
>>>>
>>>> On 17 June 2010 06:54, Dan Jacob <danjac354@gmail.com> wrote:
>>>>> So basically you would like something like this:
>>>>>
>>>>> BATCH_SIZE = 1000
>>>>>
>>>>> app.mail_relay.send_many(messages, BATCH_SIZE)
>>>>>
>>>>> Where the connection is restarted each time BATCH_SIZE messages are sent ?
>>>>>
>>>>> On 17 June 2010 05:56, Thadeus Burgess <thadeusb@thadeusb.com> wrote:
>>>>>> On batch email, what about batching the connection?
>>>>>>
>>>>>> Our SMTP server is through (mt) and only allows 1000 emails to be sent
>>>>>> per connection. Obviously I don't want to have to exit the cron
>>>>>> script, I would like to be able to close the connection, and reopen
>>>>>> another one after a certain number of sent emails. Could this be
>>>>>> handled automatically by flask-mail?
>>>>>>
>>>>>> --
>>>>>> Thadeus
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 16, 2010 at 5:46 AM, Dan Jacob <danjac354@gmail.com> wrote:
>>>>>>> I have just released version 0.2 of flask-mail on PyPi.
>>>>>>>
>>>>>>> This is mostly some bug and documentation fixes, plus bulk email
>>>>>>> sending functionality (reusing the same SMTP connection for multiple
>>>>>>> email messages).
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>