librelist archives

Re: [flask] [Ann] flask-celery (request for comments)

From:

Ask Solem

Date:

2010-09-10 @ 07:47

On Sep 9, 2010, at 11:47 PM, Armin Ronacher wrote:

> Hi,
> 
> On 9/9/10 6:31 AM, Ask Solem wrote:
>> I've been playing around with Flask lately, and made a Flask<->Celery 
extension.
> First of all: awesome :)
> 
>> It should work for more than one app instance, but this is not something I have
>> actually tested. I was not sure how flask users would like to run the 
worker, so
>> in my example the worker is started by running "app.py -w", does that seem
>> reasonably flasky (ish?) ?
> I guess the best way would be to provide a command for Flask-Script. 
> That's how that sort of stuff is currently kinda semi-officially handled.

Aha. Haven't seen that yet. I'll find out.

> 
> I am quite surprised that it works with multiple applications because I 
> the last time I was looking at celery it had an implicit configuration. 
>  The flask extension approval guidelines currently say that it has to 
> support multiple applications so if that extension supports that, I 
> would be the first one to approve it right after the release :)

Well. Many of the classes uses the celery.conf module to get default values, e.g.

    class Foo(object):
        send_emails = conf.SEND_EMAILS

        def __init__(self, send_emails=None):
             if send_emails is not None:
                 self.send_emails = send_emails


Applying a task using the global default configuration:

     add.apply_async((16, 16))

Manually:

    from tasks import add
    from celery.messaging import establish_connection


    conn = establish_connection(hostname="localhost",
                                                          userid="guest",
                                                          password="guest",
                                                          virtual_host="/")

    router = Router(routes=
    add.apply_async((16, 16) connection=conn)


You can even use routers:

    from celery import routers

    routes = routers.prepare({add.name: {"queue": "cpubound"})
    router = Router(routes, queues={}, create_missing=True)
    add.apply_async((16, 16), connection=conn, router=router)
 
So all the options can also be configured for each instance. I'm working
on moving over to the "app that can be instantiated" approach.


But then there's the worker part, there are still some 
parts that can't be easily configured without changing the globals,
but then again having multiple workers
in the same process space doesn't really make sense afaics, so i'm not
sure how limiting this is. It's not pretty that's for sure, but at the time
it seemed better than passing conf everywhere (which in hindsight was just
lazy negligence)

> 
>> The way it works currently is not optimal (doing a lot of imports at runtime),
>> but I'm cleaning up Celery as I go along, so in the future I hope this 
isn't necessary.
> Very cool.  I will give you detailed feedback later today, currently 
> busily cleaning up some stuff in logbook

Thanks :) logbook owns!

-- 
{Ask Solem,
 Developer ~ Webteam,
 Opera Software,
 +47 98435213 | twitter.com/asksol }.

>> I guess the best way would be to provide a command for Flask-Script.
>> That's how that sort of stuff is currently kinda semi-officially handled.
>
> Aha. Haven't seen that yet. I'll find out.
>

The easiest way is to provide a Command class that users can drop into
their own manager setup:

from flaskext.celery import RunCelery
from flaskext.script import Manager

manager = Manager(my_app)
manager.add_command("runcelery", RunCelery())

if __name__ == "__main__":
    manager.run()

On Sep 10, 2010, at 9:53 AM, Dan Jacob wrote:

>>> I guess the best way would be to provide a command for Flask-Script.
>>> That's how that sort of stuff is currently kinda semi-officially handled.
>> 
>> Aha. Haven't seen that yet. I'll find out.
>> 
> 
> The easiest way is to provide a Command class that users can drop into
> their own manager setup:
> 
> from flaskext.celery import RunCelery
> from flaskext.script import Manager
> 
> manager = Manager(my_app)
> manager.add_command("runcelery", RunCelery())
> 
> if __name__ == "__main__":
>    manager.run()

Thanks for the help. Flask-Script did have excellent documentation!
> 



-- 
{Ask Solem,
 Developer ~ Webteam,
 Opera Software,
 +47 98435213 | twitter.com/asksol }.

BTW, I've pulled your fork changes in Flask-Script. It does make more
sense to have the arg parsing handled by the Command. Thanks.

On 10 September 2010 12:34, Ask Solem <askh@opera.com> wrote:
>
> On Sep 10, 2010, at 9:53 AM, Dan Jacob wrote:
>
>>>> I guess the best way would be to provide a command for Flask-Script.
>>>> That's how that sort of stuff is currently kinda semi-officially handled.
>>>
>>> Aha. Haven't seen that yet. I'll find out.
>>>
>>
>> The easiest way is to provide a Command class that users can drop into
>> their own manager setup:
>>
>> from flaskext.celery import RunCelery
>> from flaskext.script import Manager
>>
>> manager = Manager(my_app)
>> manager.add_command("runcelery", RunCelery())
>>
>> if __name__ == "__main__":
>>    manager.run()
>
> Thanks for the help. Flask-Script did have excellent documentation!
>>
>
>
>
> --
> {Ask Solem,
>  Developer ~ Webteam,
>  Opera Software,
>  +47 98435213 | twitter.com/asksol }.
>
>