librelist archives

« back to archive

MultiThreading development:need suggestion

MultiThreading development:need suggestion

From:
Date:
2015-07-09 @ 21:17
Hi, all


I’d like to know how everyone handle the mutilthreading stuff, says,
I have a flask app deploying in an apache server with gunicore,
at the same time, I have a function that inheriated Threading python package
to crawl some infomation from some website, I want them both running,


flask - deal with incoming request
craw function - crawing information as I let it when to start and stop


my implementation is simple,
just import threading package and writing the crawl function,
I write it in the __init__.py file, so that when flask app start, it runs with it.


Need some best practice here,
Thanks a lot


Wallace

Re: [flask] MultiThreading development:need suggestion

From:
Jeremy Epstein
Date:
2015-07-09 @ 23:29
Hey Wallace,

I built a Flask app that handles multi-threading a few years back,
check out the article I wrote about it:

http://greenash.net.au/thoughts/2011/06/thread-progress-monitoring-in-python/

The code examples are for an older version of Flask (and older jQuery
etc), but shouldn't need much work to be compatible with latest stable
Flask.

As I make clear in my article, a big NOTE: what I built was for a
"web-based desktop app" (i.e. an app only designed to be run in a
local dev environment), not an actual web app. I don't know if it's
even possible to build something like what I did, and to make it
safely deployable to gunicorn or similar, where there are many limits
on threading, and where there are concurrency / performance / etc
considerations.

As Nick said, you're probably better off looking into Celery. (Or
having a threaded Python daemon app running on your back-end, which a
Flask app could talk to, but which is separate from the Flask app).

Cheers,
Jeremy.

On Fri, Jul 10, 2015 at 7:17 AM, wm140906 <wm140906@gmail.com> wrote:
> Hi, all
>
>
> I’d like to know how everyone handle the mutilthreading stuff, says,
>
> I have a flask app deploying in an apache server with gunicore,
>
> at the same time, I have a function that inheriated Threading python package
>
> to crawl some infomation from some website, I want them both running,
>
>
> flask -> deal with incoming request
>
> craw function -> crawing information as I let it when to start and stop
>
>
> my implementation is simple,
>
> just import threading package and writing the crawl function,
>
> I write it in the __init__.py file, so that when flask app start, it runs
> with it.
>
>
> Need some best practice here,
>
> Thanks a lot
>
>
> Wallace

Re: [flask] MultiThreading development:need suggestion

From:
Nick Janetakis
Date:
2015-07-09 @ 22:07
Are you able to use Celery to run the crawler in the background?

On Thu, Jul 9, 2015 at 5:17 PM, wm140906 <wm140906@gmail.com> wrote:

> Hi, all
>
>
> I’d like to know how everyone handle the mutilthreading stuff, says,
>
> I have a flask app deploying in an apache server with gunicore,
>
> at the same time, I have a function that inheriated Threading python
> package
>
> to crawl some infomation from some website, I want them both running,
>
>
> flask -> deal with incoming request
>
> craw function -> crawing information as I let it when to start and stop
>
>
> my implementation is simple,
>
> just import threading package and writing the crawl function,
>
> I write it in the __init__.py file, so that when flask app start, it runs
> with it.
>
>
> Need some best practice here,
>
> Thanks a lot
>
>
> Wallace
>

Re: [flask] MultiThreading development:need suggestion

From:
Date:
2015-07-10 @ 00:56
Hi,Jeremy


I am pretty new into everything about threading,
thanks for your information, that would give me some clue to go deeper.
I will check your blog, and also Celery.
thanks for all.


Wallace


Original Message
Sender:Jeremy Epsteinjazepstein@gmail.com
Recipient:flaskflask@librelist.com
Date:Thursday, Jul 9, 2015 19:29
Subject:Re: [flask] MultiThreading development:need suggestion


Hey Wallace, I built a Flask app that handles multi-threading a few years 
back, check out the article I wrote about it: 
http://greenash.net.au/thoughts/2011/06/thread-progress-monitoring-in-python/
The code examples are for an older version of Flask (and older jQuery 
etc), but shouldn't need much work to be compatible with latest stable 
Flask. As I make clear in my article, a big NOTE: what I built was for a 
"web-based desktop app" (i.e. an app only designed to be run in a local 
dev environment), not an actual web app. I don't know if it's even 
possible to build something like what I did, and to make it safely 
deployable to gunicorn or similar, where there are many limits on 
threading, and where there are concurrency / performance / etc 
considerations. As Nick said, you're probably better off looking into 
Celery. (Or having a threaded Python daemon app running on your back-end, 
which a Flask app could talk to, but which is separate from the Flask 
app). Cheers, Jeremy. On Fri, Jul 10, 2015 at 7:17 AM, wm140906 
wm140906@gmail.com wrote:  Hi, all    I’d like to know how everyone handle
the mutilthreading stuff, says,   I have a flask app deploying in an 
apache server with gunicore,   at the same time, I have a function that 
inheriated Threading python package   to crawl some infomation from some 
website, I want them both running,    flask - deal with incoming request
craw function - crawing information as I let it when to start and stop    
my implementation is simple,   just import threading package and writing 
the crawl function,   I write it in the __init__.py file, so that when 
flask app start, it runs  with it.    Need some best practice here,   
Thanks a lot    Wallace

Re: [flask] MultiThreading development:need suggestion

From:
Date:
2015-07-10 @ 00:52
Hi, Nick
Sorry, I am not farmiliar withCelery.
wallace


Original Message
Sender:Nick Janetakisnick.janetakis@gmail.com
Recipient:flaskflask@librelist.com
Date:Thursday, Jul 9, 2015 18:07
Subject:Re: [flask] MultiThreading development:need suggestion


Are you able to use Celery to run the crawler in the background?



On Thu, Jul 9, 2015 at 5:17 PM, wm140906 wm140906@gmail.com wrote:

Hi, all


I’d like to know how everyone handle the mutilthreading stuff, says,
I have a flask app deploying in an apache server with gunicore,
at the same time, I have a function that inheriated Threading python package
to crawl some infomation from some website, I want them both running,


flask - deal with incoming request
craw function - crawing information as I let it when to start and stop


my implementation is simple,
just import threading package and writing the crawl function,
I write it in the __init__.py file, so that when flask app start, it runs with it.


Need some best practice here,
Thanks a lot


Wallace