Hello happy flaskers, :] In a previous thread<http://librelist.com/browser//flask/2011/8/8/generate-pdf-files-from-templates-adding-user-specific-info/>, I was wondering if Eventlet was good enough to process e-mail sending (with Flask-Mail). In general I would ask this question: Is this a good idea to use Eventlet's monkey patching with Flask? (Is it advisable?) I speak in terms of performance, reliability, security, etc. The code I wrote: http://paste.pocoo.org/show/460232/ Some resources: - Eventlet: http://eventlet.net/ - Monkey patching with Eventlet: http://eventlet.net/doc/patching.html#monkeypatching-the-standard-library - Flask-Mail: http://packages.python.org/flask-mail/ My other problem is that this technique works well locally, but when I push my code on the server (uWSGI + Nginx on DotCloud) e-mails take a long time (5 to 15 minutes) to arrive (Some never received). Do you think that set the *worker_processes*<http://wiki.nginx.org/CoreModule#worker_processes>option to 2 (instead of 1 by default) can solve my problem? Thank you for future answers. :) Regards, -- Marc
> Hello happy flaskers, :] > > In a previous > thread<http://librelist.com/browser//flask/2011/8/8/generate-pdf-files-from-templates-adding-user-specific-info/>, > I was wondering if Eventlet was good enough to process e-mail sending > (with > Flask-Mail). > > In general I would ask this question: Is this a good idea to use > Eventlet's > monkey patching with Flask? (Is it advisable?) I speak in terms of > performance, reliability, security, etc. > The code I wrote: http://paste.pocoo.org/show/460232/ > This is one of those (rare) kind of tasks where async/nonblocking is not good. Opening massive connections to smtp servers is bad (and 99.9% of the time disallowed). You will end (normally) sending one mail at time in background (using something like celery, the uWSGI spooler or a background thread) > My other problem is that this technique works well locally, but when I > push my > code on the server (uWSGI + Nginx on DotCloud) e-mails take a long time (5 > to 15 minutes) to arrive (Some never received). > Do you think that set the > *worker_processes*<http://wiki.nginx.org/CoreModule#worker_processes>option > to > 2 (instead of 1 by default) can solve my problem? > > Thank you for future answers. :) > > Regards, > Ask dotcloud guys to enable the spooler for you (i do not remember if the version they use includes it). if it is not possible ask for celery, and if it not possible again enable threading mode in uWSGI (--enable-threads) and use a thread queue to send the emails (there is an example of thread queue in the uWSGI example wiki page) Regarding worker processes it is good to abuse them, but they will not help on this specific case. -- Roberto De Ioris http://unbit.it
If you are using dotcloud, why dont you use their SMTP service instead of using eventlet. On Thu, Aug 18, 2011 at 9:13 PM, Roberto De Ioris <roberto@unbit.it> wrote: > > > Hello happy flaskers, :] > > > > In a previous > > thread< > http://librelist.com/browser//flask/2011/8/8/generate-pdf-files-from-templates-adding-user-specific-info/ > >, > > I was wondering if Eventlet was good enough to process e-mail sending > > (with > > Flask-Mail). > > > > In general I would ask this question: Is this a good idea to use > > Eventlet's > > monkey patching with Flask? (Is it advisable?) I speak in terms of > > performance, reliability, security, etc. > > The code I wrote: http://paste.pocoo.org/show/460232/ > > > > > This is one of those (rare) kind of tasks where async/nonblocking is not > good. > Opening massive connections to smtp servers is bad (and 99.9% of the time > disallowed). You will end (normally) sending one mail at time in > background > (using something like celery, the uWSGI spooler or a background thread) > > > > My other problem is that this technique works well locally, but when I > > push my > > code on the server (uWSGI + Nginx on DotCloud) e-mails take a long time > (5 > > to 15 minutes) to arrive (Some never received). > > Do you think that set the > > *worker_processes*<http://wiki.nginx.org/CoreModule#worker_processes > >option > > to > > 2 (instead of 1 by default) can solve my problem? > > > > Thank you for future answers. :) > > > > Regards, > > > > Ask dotcloud guys to enable the spooler for you (i do not remember if the > version they use includes it). if it is not possible ask for celery, and > if it not possible again enable threading mode in uWSGI (--enable-threads) > and use a thread queue to send the emails (there is an example of thread > queue in the uWSGI example wiki page) > > Regarding worker processes it is good to abuse them, but they will not > help on this specific case. > > > -- > Roberto De Ioris > http://unbit.it >
@Daniel , Dao Quang Minh For now all my DotCloud services are used for other tasks. I did not plan the SMTP service. Moreover the Google SMTP service (I use it) suits me. @Roberto De Ioris Thank you for your detailed explanation. If possible, I would prefer to not depend on uWSGI (in case of hosting change). So I'm going to try with Celery. I'll keep you informed. -- Marc On Thu, Aug 18, 2011 at 3:23 PM, Daniel , Dao Quang Minh <dqminh89@gmail.com > wrote: > If you are using dotcloud, why dont you use their SMTP service instead of > using eventlet. > > > On Thu, Aug 18, 2011 at 9:13 PM, Roberto De Ioris <roberto@unbit.it>wrote: > >> >> > Hello happy flaskers, :] >> > >> > In a previous >> > thread< >> http://librelist.com/browser//flask/2011/8/8/generate-pdf-files-from-templates-adding-user-specific-info/ >> >, >> > I was wondering if Eventlet was good enough to process e-mail sending >> > (with >> > Flask-Mail). >> > >> > In general I would ask this question: Is this a good idea to use >> > Eventlet's >> > monkey patching with Flask? (Is it advisable?) I speak in terms of >> > performance, reliability, security, etc. >> > The code I wrote: http://paste.pocoo.org/show/460232/ >> > >> >> >> This is one of those (rare) kind of tasks where async/nonblocking is not >> good. >> Opening massive connections to smtp servers is bad (and 99.9% of the time >> disallowed). You will end (normally) sending one mail at time in >> background >> (using something like celery, the uWSGI spooler or a background thread) >> >> >> > My other problem is that this technique works well locally, but when I >> > push my >> > code on the server (uWSGI + Nginx on DotCloud) e-mails take a long time >> (5 >> > to 15 minutes) to arrive (Some never received). >> > Do you think that set the >> > *worker_processes*<http://wiki.nginx.org/CoreModule#worker_processes >> >option >> > to >> > 2 (instead of 1 by default) can solve my problem? >> > >> > Thank you for future answers. :) >> > >> > Regards, >> > >> >> Ask dotcloud guys to enable the spooler for you (i do not remember if the >> version they use includes it). if it is not possible ask for celery, and >> if it not possible again enable threading mode in uWSGI (--enable-threads) >> and use a thread queue to send the emails (there is an example of thread >> queue in the uWSGI example wiki page) >> >> Regarding worker processes it is good to abuse them, but they will not >> help on this specific case. >> >> >> -- >> Roberto De Ioris >> http://unbit.it >> > >
Finally I am using Celery. It works fine, even using MongoDB as broker.
pip install celery
My celeryconfig.py file:
BROKER_BACKEND = "mongodb"
BROKER_HOST = "db_url.domain.com"
BROKER_PORT = 27017
BROKER_USER = 'mongodb_user'
BROKER_PASSWORD = 'mongodb_password'
BROKER_VHOST = 'db_name_for_queue'
CELERY_IMPORTS = ('myapp.workers.mailing', )
CELERY_IGNORE_RESULT = True
The myapp/workers/mailing.py file:
from celery.task import task
@task
def send_mail(email):
from myapp import mail # mail = Mail(app) # (Flask-Mail)
mail.send(email)
Thank you.
--
Marc
On Sat, Aug 20, 2011 at 9:54 AM, Marc de Verdelhan <
marc.deverdelhan@yahoo.com> wrote:
> @Daniel , Dao Quang Minh
> For now all my DotCloud services are used for other tasks. I did not plan
> the SMTP service. Moreover the Google SMTP service (I use it) suits me.
>
> @Roberto De Ioris
> Thank you for your detailed explanation.
> If possible, I would prefer to not depend on uWSGI (in case of hosting
> change). So I'm going to try with Celery.
>
> I'll keep you informed.
>
> --
> Marc
>
>
> On Thu, Aug 18, 2011 at 3:23 PM, Daniel , Dao Quang Minh <
> dqminh89@gmail.com> wrote:
>
>> If you are using dotcloud, why dont you use their SMTP service instead of
>> using eventlet.
>>
>>
>> On Thu, Aug 18, 2011 at 9:13 PM, Roberto De Ioris <roberto@unbit.it>wrote:
>>
>>>
>>> > Hello happy flaskers, :]
>>> >
>>> > In a previous
>>> > thread<
>>>
http://librelist.com/browser//flask/2011/8/8/generate-pdf-files-from-templates-adding-user-specific-info/
>>> >,
>>> > I was wondering if Eventlet was good enough to process e-mail sending
>>> > (with
>>> > Flask-Mail).
>>> >
>>> > In general I would ask this question: Is this a good idea to use
>>> > Eventlet's
>>> > monkey patching with Flask? (Is it advisable?) I speak in terms of
>>> > performance, reliability, security, etc.
>>> > The code I wrote: http://paste.pocoo.org/show/460232/
>>> >
>>>
>>>
>>> This is one of those (rare) kind of tasks where async/nonblocking is not
>>> good.
>>> Opening massive connections to smtp servers is bad (and 99.9% of the time
>>> disallowed). You will end (normally) sending one mail at time in
>>> background
>>> (using something like celery, the uWSGI spooler or a background thread)
>>>
>>>
>>> > My other problem is that this technique works well locally, but when I
>>> > push my
>>> > code on the server (uWSGI + Nginx on DotCloud) e-mails take a long time
>>> (5
>>> > to 15 minutes) to arrive (Some never received).
>>> > Do you think that set the
>>> > *worker_processes*<http://wiki.nginx.org/CoreModule#worker_processes
>>> >option
>>> > to
>>> > 2 (instead of 1 by default) can solve my problem?
>>> >
>>> > Thank you for future answers. :)
>>> >
>>> > Regards,
>>> >
>>>
>>> Ask dotcloud guys to enable the spooler for you (i do not remember if the
>>> version they use includes it). if it is not possible ask for celery, and
>>> if it not possible again enable threading mode in uWSGI
>>> (--enable-threads)
>>> and use a thread queue to send the emails (there is an example of thread
>>> queue in the uWSGI example wiki page)
>>>
>>> Regarding worker processes it is good to abuse them, but they will not
>>> help on this specific case.
>>>
>>>
>>> --
>>> Roberto De Ioris
>>> http://unbit.it
>>>
>>
>>
>
To ease your trying: http://www.celeryproject.org/ http://www.slideshare.net/richleland/celery-the-distributed-task-queue http://www.slideshare.net/tarequeh/life-in-a-queue-using-message-queue-with-django http://www.turnkeylinux.org/blog/django-celery-rabbitmq Honza On Sat, Aug 20, 2011 at 09:54, Marc de Verdelhan <marc.deverdelhan@yahoo.com> wrote: > @Daniel , Dao Quang Minh > For now all my DotCloud services are used for other tasks. I did not plan > the SMTP service. Moreover the Google SMTP service (I use it) suits me. > > @Roberto De Ioris > Thank you for your detailed explanation. > If possible, I would prefer to not depend on uWSGI (in case of hosting > change). So I'm going to try with Celery. > > I'll keep you informed. > > -- > Marc > > On Thu, Aug 18, 2011 at 3:23 PM, Daniel , Dao Quang Minh > <dqminh89@gmail.com> wrote: >> >> If you are using dotcloud, why dont you use their SMTP service instead of >> using eventlet. >> >> On Thu, Aug 18, 2011 at 9:13 PM, Roberto De Ioris <roberto@unbit.it> >> wrote: >>> >>> > Hello happy flaskers, :] >>> > >>> > In a previous >>> > >>> > thread<http://librelist.com/browser//flask/2011/8/8/generate-pdf-files-from-templates-adding-user-specific-info/>, >>> > I was wondering if Eventlet was good enough to process e-mail sending >>> > (with >>> > Flask-Mail). >>> > >>> > In general I would ask this question: Is this a good idea to use >>> > Eventlet's >>> > monkey patching with Flask? (Is it advisable?) I speak in terms of >>> > performance, reliability, security, etc. >>> > The code I wrote: http://paste.pocoo.org/show/460232/ >>> > >>> >>> >>> This is one of those (rare) kind of tasks where async/nonblocking is not >>> good. >>> Opening massive connections to smtp servers is bad (and 99.9% of the time >>> disallowed). You will end (normally) sending one mail at time in >>> background >>> (using something like celery, the uWSGI spooler or a background thread) >>> >>> >>> > My other problem is that this technique works well locally, but when I >>> > push my >>> > code on the server (uWSGI + Nginx on DotCloud) e-mails take a long time >>> > (5 >>> > to 15 minutes) to arrive (Some never received). >>> > Do you think that set the >>> > >>> > *worker_processes*<http://wiki.nginx.org/CoreModule#worker_processes>option >>> > to >>> > 2 (instead of 1 by default) can solve my problem? >>> > >>> > Thank you for future answers. :) >>> > >>> > Regards, >>> > >>> >>> Ask dotcloud guys to enable the spooler for you (i do not remember if the >>> version they use includes it). if it is not possible ask for celery, and >>> if it not possible again enable threading mode in uWSGI >>> (--enable-threads) >>> and use a thread queue to send the emails (there is an example of thread >>> queue in the uWSGI example wiki page) >>> >>> Regarding worker processes it is good to abuse them, but they will not >>> help on this specific case. >>> >>> >>> -- >>> Roberto De Ioris >>> http://unbit.it >> > >