librelist archives

Re: [flask] Beginner's question

From:

Garrett Robinson

Date:

2011-09-29 @ 21:11

Wow, thanks for all the responses, everybody. Sorry this is much later
on. I'm currently using beanstalkd, with the beanstalkc python
library. It is the perfect solution for what I'm doing. Thanks to Joe
for recommending it, and thanks to everyone else for all your
thoughtful suggestions!

If anyone is interested, I'm using beanstalkd in these projects:
https://github.com/handsomeransoms/haps (with Flask)
https://github.com/handsomeransoms/haps-hidserv (no Flask here)

On Sat, Sep 3, 2011 at 7:05 PM, Joe Esposito <espo58@gmail.com> wrote:
> Wow Beanstalk looks amazing. Going to try it out. Thanks for sharing!
> Some observations:
>
> beanstalkd is the external dependency. It's very lightweight though and has
> no configuration files. You simply run it as a daemon.
> There are many clients written for various languages. Two for Python,
> though beanstalkc is the more active one and is the only one on PyPi
> The clients look incredibly simple to use. Simply import, call Connect using
> the host/port specified in the beanstalkd cmdline, then:
>
> On the producer side, call "put" to create a job
> On the consumer side, call "reserve" to safely get a job and "delete" the
> job is complete
>
> You can have multiple named job queues (called "tubes")
> Jobs can be persistent (configurable via beanstalkd cmdline)
> All messages are strings. This shouldn't be a problem though since you
> should really use a DB for more complex communication.
> There's no built-in mechanism for reporting progress, though using a tube
> and the job id, you can manually communicate the other direction
> There's no Windows support. Perhaps you could handle the
> beanstalkc.SocketError to call the job function directly when in debug mode
> if you're developing on Windows.
>
> References:
> Beanstalkd
> Beanstalkc (Python client)
> Beanstalk FAQ
> A simple tutorial
> On Sat, Sep 3, 2011 at 5:12 PM, Cheng-Han Lee <lee.chenghan@gmail.com>
> wrote:
>>
>> Celery can be a bit difficult to setup.
>>
>> I would also recommend beanstalkd. Its work-queue similar to that of
>> celery, but its easier to setup and use (at least from what I've read).
>>
>> On Sat, Sep 3, 2011 at 7:50 AM, Garrett Robinson
>> <garrett.f.robinson@gmail.com> wrote:
>>>
>>> Thanks for the replies. I agree, Celery seems like too much for me
>>> too. All I want is to implement is a simple file queue to rate-limit
>>> the transfer of files from the web server to a dedicated file server,
>>> in a threadsafe way.
>>>
>>> I will check out multiprocessing. Thanks!
>>>
>>> On Sat, Sep 3, 2011 at 10:45 AM, Joe Esposito <espo58@gmail.com> wrote:
>>> > The Django community has had some talk about using the multiprocessing
>>> > module (Python 2.6+). That would probably be the easiest thing to set
>>> > up and
>>> > get you going since there's no external dependencies and it's all pure
>>> > Python code. I have yet to try it out but I also have an upcoming
>>> > project
>>> > that requires extensive server-side processing.
>>> > I've tried Celery so far. If you can get it going, I think it'll be
>>> > nice. I
>>> > had a lot of problems with it though. Seems to do too much, so it's
>>> > hard to
>>> > find docs on what you actually want to do with it. Was also looking for
>>> > an
>>> > alternative.
>>> > Anyone have any good experience with multiprocessing?
>>> >
>>> > On Sat, Sep 3, 2011 at 1:25 AM, Rodrigo Aliste P. <raliste@gmail.com>
>>> > wrote:
>>> >>
>>> >> AFAIK file uploading works in a per-request basis, so I think at best
>>> >> you
>>> >> could have more than 1 server and load balance uploads betwen them.
>>> >> For post processing or moving the files I would encourage you to think
>>> >> of
>>> >> other (nicer) options, such as Gearman or Flask-Celery.
>>> >> On Sat, Sep 3, 2011 at 2:10 AM, Garrett Robinson
>>> >> <garrett.f.robinson@gmail.com> wrote:
>>> >>>
>>> >>> I'm working on a site that will have a queue for transferring files
>>> >>> uploaded by its users, to better control bandwidth. I'm wondering how
>>> >>> to design the queue.
>>> >>>
>>> >>> My thoughts are:
>>> >>> 1) keep track of it inside the flask app using Python's Queue; but
>>> >>> then i don't know how to poll it regularly because everything in
>>> >>> Flask
>>> >>> is request-driven
>>> >>> 2) use a file or database to keep track of the queue, then write a
>>> >>> separate python script to manage it, and run it regularly with cron
>>> >>>
>>> >>> Then is there anything special about accessing the file that I need
>>> >>> to
>>> >>> know re: Flask, multithreading, etc.? Sorry if this is a n00b
>>> >>> question, I don't know much about multithreading/file access issues -
>>> >>> just enough to know that it could be a problem : )
>>> >>>
>>> >>> I'd really appreciate any help or guidance.
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Rodrigo Aliste P.
>>> >>
>>> >
>>> >
>>
>
>