librelist archives

« back to archive

another question about background queue processing (without celery/...)

another question about background queue processing (without celery/...)

From:
gaikokujin kun
Date:
2013-09-10 @ 11:31
Good day,

my question is maybe more werkzeug oriented but i post it here because
it is in a flask context; please let me know if you prefer to see it
posted on the werkzeug mailing list.

This is a recurring topic, but i didn't found any satisfactory (for my
needs) answer on the posts i saw on the mailing, if i missed it, please
point it to me :^)

Context:
I've a flask-through-uwsgi web application executing asynchronous jobs,
when a job fail it is put in a database to be processed at a later time.
I want to have a background process or thread tidying the failed queue
at a given interval with the following constraints:

* i would like it to live in the web application, not in a separate cron
  script, mainly because i would like the tidying thing start and die
  at the same time the web application
* everybody is advising to use greenlets/celery/whatever but i'm sure it
  could be done using the standard Python library
* the main application is not expecting data from the background job;
  but it would be nice if it could ensure it is alive

I though using multiprocessing.Process, but i don't know if uwsgi
could happen to instantiate the application more than once, which is
going to start more than one tidying job which is not what i want.

I will gladly listen to any advice you could give me.

Regards.

Re: [flask] another question about background queue processing (without celery/...)

From:
Dave Shawley
Date:
2013-09-10 @ 11:56
On Sep 10, 2013, at 7:31 AM, gaikokujin kun <gaikokujinkun@gmail.com> wrote:

> I've a flask-through-uwsgi web application executing asynchronous jobs,
> when a job fail it is put in a database to be processed at a later time.
> I want to have a background process or thread tidying the failed queue
> at a given interval with the following constraints:


You might want to consider a data store that manages the life time of the
data for you.  Redis, Cassandra, and MongoDB all have support for expiring
values based on a time-to-live.  Choose a data storage mechanism that satisfies
your requirements for resiliency, fault tolerance, and throughput.  Each of
them differ in how they are deployed and managed, but having your data store
manage the lifetime of your data is a really nice feature compared to rolling
your own pruning daemon.

If you decide to manage this in application code, I would recommend not doing
it in your Flask code.  Either write a Celery worker that prunes your task list
periodically or have a separate internal application that is invoked using an
HTTP API to vacuum your data set.  You really do want to keep the logic and
code separate if at all possible.

-- dave
--
Mathematics provides a framework for dealing precisely with notions of "what is".
Computation provides a framework for dealing precisely with notions of "how to".
SICP Preface

Re: [flask] another question about background queue processing (without celery/...)

From:
Andres Rand
Date:
2013-09-10 @ 11:48
Hello!

Take a look at

http://stackoverflow.com/questions/9931738/flask-throwing-working-outside-of-request-context-when-starting-sub-thread/9932189#9932189

Still, greenlets or celery is still better suited for this kind of work.

Regards

--
Andres Rand



2013/9/10 gaikokujin kun <gaikokujinkun@gmail.com>

> Good day,
>
> my question is maybe more werkzeug oriented but i post it here because
> it is in a flask context; please let me know if you prefer to see it
> posted on the werkzeug mailing list.
>
> This is a recurring topic, but i didn't found any satisfactory (for my
> needs) answer on the posts i saw on the mailing, if i missed it, please
> point it to me :^)
>
> Context:
> I've a flask-through-uwsgi web application executing asynchronous jobs,
> when a job fail it is put in a database to be processed at a later time.
> I want to have a background process or thread tidying the failed queue
> at a given interval with the following constraints:
>
> * i would like it to live in the web application, not in a separate cron
>   script, mainly because i would like the tidying thing start and die
>   at the same time the web application
> * everybody is advising to use greenlets/celery/whatever but i'm sure it
>   could be done using the standard Python library
> * the main application is not expecting data from the background job;
>   but it would be nice if it could ensure it is alive
>
> I though using multiprocessing.Process, but i don't know if uwsgi
> could happen to instantiate the application more than once, which is
> going to start more than one tidying job which is not what i want.
>
> I will gladly listen to any advice you could give me.
>
> Regards.
>