librelist archives

Feedback requested: Flask-PyMongo (new extension)

From:

Dan Crosta

Date:

2011-12-27 @ 23:15

Over the past few days, I've been working on an extension to add PyMongo 
and MongoDB support to Flask. I'd appreciate feedback from Flask users 
(especially those using MongoDB) and developers.


Background:

There already exist two MongoDB-related extensions, Flask-MongoAlchemy and
Flask-MongoKit, but both require the use of an object-document mapper 
("ODM"). These require explicit data-modeling, or at the very least 
additional class definitions, in order to be useful. PyMongo and 
Flask-PyMongo work with native Python dictionaries, but requires the 
application developer to manage schemas, migrations, indexes, etc, by 
hand.


The Extension:

Using Flask-PyMongo primarily involves creating a PyMongo object (which 
takes a Flask app as a constructor argument), which consults configuration
and auto-connects to the database server. Each PyMongo object is designed 
to connect to a specific database (like most database management systems, 
MongoDB has a server > database > collection > document hierarchy, which 
is analagous to server > database > table > row in RDBMSes), though it 
exposes the connection object for access to any other database on the same
server.

Flask-PyMongo configures itself (by default) from config variables named 
beginning with "MONGO_". A second constructor argument allows the user to 
change this (which is necessary in the event that he/she wants to connect 
to several distinct MongoDB servers).


Documentation & Source:

docs: http://flask-pymongo.readthedocs.org/en/latest/index.html
source: https://github.com/dcrosta/flask-pymongo


Questions:

1. I notice that most Flask extensions have documentation at 
packages.python.org rather than Read the Docs. Any particular reason for 
this?
2. The extension development guide suggests adding an "init_app" function 
-- what is the use case in which this is necessary?
3. The extension development guide says "there must be a link to 
automatically install the development version (PackageName==dev)" -- what 
exactly needs to be done for this? (Flask-PyMongo is not yet listed in 
PyPi, as I've yet to release version 0.1)
4. General and specific feedback on the approach, code, utility, any 
missing features I should consider adding, etc.


Thanks!
- Dan

Le 28/12/2011 00:15, Dan Crosta a écrit :
> 1. I notice that most Flask extensions have documentation at 
packages.python.org rather than Read the Docs. Any particular reason for 
this?

Unless I have a domain name with a website specific to the project (in 
which case the docs goes there) I use packages.python.org for every 
published project.

After installing Sphinx-PyPI-upload, it Just Works®:

python setup.py build_sphinx upload_sphinx

No particular reason other than that.

For small projects I did not find the need to have different 
documentation for different versions. Are there other reasons to use 
Read the Docs?

Regards,
-- 
Simon Sapin

On Dec 28, 2011, at 2:41 AM, Simon Sapin wrote:
> For small projects I did not find the need to have different 
> documentation for different versions. Are there other reasons to use 
> Read the Docs?

It has nice integration with GitHub so that docs are automatically built 
when I push to master, so I don't ever have to manually manage docs there.

- Dan

On Dec 27, 2011, at 18:15 , Dan Crosta wrote:
> Background:
> 
> There already exist two MongoDB-related extensions, Flask-MongoAlchemy 
and Flask-MongoKit, but both require the use of an object-document mapper 
("ODM"). These require explicit data-modeling, or at the very least 
additional class definitions, in order to be useful. PyMongo and 
Flask-PyMongo work with native Python dictionaries, but requires the 
application developer to manage schemas, migrations, indexes, etc, by 
hand.

Certainly looks like it fills a useful niche. I have to admit I haven't 
done much with MongoDB much.

> The Extension:
> 
> [snip]
> 
> Flask-PyMongo configures itself (by default) from config variables named
beginning with "MONGO_". A second constructor argument allows the user to 
change this (which is necessary in the event that he/she wants to connect 
to several distinct MongoDB servers).

Makes sense. The prefix is a nice touch.

> Questions:
> 
> 1. I notice that most Flask extensions have documentation at 
packages.python.org rather than Read the Docs. Any particular reason for 
this?

Mostly because ReadTheDocs.org wasn't really that big when Flask 
extensions really got started.

> 2. The extension development guide suggests adding an "init_app" 
function -- what is the use case in which this is necessary?

If you're using an app factory, the use of init_app means that you can 
create an instance of PyMongo with all your settings in your toplevel 
code, then attach it to the app when it's actually created (even if you 
create multiple apps, e.g. for unit testing). The central instance doesn't
actually keep a reference to the app, it merely sets everything up and 
uses `current_app` whenever it needs a reference to something stored on 
the app.

To be honest, if I were in charge of Flask extensions `self.app = app` in 
a central instance would be banned. What I would recommend instead for 
your particular case is:

- Don't store the configuration on the actual PyMongo instance, just the 
configuration prefix. (That way, each instance represents a single 
prefix.)
- In your init_app function, create the connection, but store it on the 
app in `app.extensions["pymongo"][PREFIX]`. Possibly do the same with the 
database, either by using a tuple or a cx/db wrapper class and stashing it
in the dictionary instead. Then set up the hooks like normal.
- Whenever the methods or properties on the instance need the database or 
connection, they use 
`current_app.extensions["pymongo"][self.config_prefix]`.

This allows each instance to be used to refer to the database with the 
proper settings regardless of which app is active at the moment.

> 3. The extension development guide says "there must be a link to 
automatically install the development version (PackageName==dev)" -- what 
exactly needs to be done for this? (Flask-PyMongo is not yet listed in 
PyPi, as I've yet to release version 0.1)

I think it involves a listing in your setup.py file's long_description 
that looks like this:

    * `development version
      
<http://bitbucket.org/leafstorm/flask-couchdb/get/tip.gz#egg=Flask-CouchDB-dev>`_

Not 100% sure if that's it though.

> 4. General and specific feedback on the approach, code, utility, any 
missing features I should consider adding, etc.

ARGH MONKEYPATCHING DIE DIE DIE

In all seriousness, I'm not 100% sure monkeypatching is the best way to 
accomplish your find_or_404 method. What Flask-SQLAlchemy does for 
Web-specific methods like that is create a customized Query subclass with 
the methods it needs, then rig the session to produce those Queries 
instead of SQLAlchemy's defaults. Flask-CouchDB does something similar so 
Document will use the default database. It's a bit more work, but it 
doesn't muck about with other modules' classes, and the "Flask culture" in
general is very much against monkeypatching and global state.

> Thanks!
> - Dan

Thanks,
Matthew Frazier
http://leafstorm.us/

On Dec 27, 2011, at 6:44 PM, Matthew Frazier wrote:
> On Dec 27, 2011, at 18:15 , Dan Crosta wrote:
>> 2. The extension development guide suggests adding an "init_app" 
function -- what is the use case in which this is necessary?
> 
> If you're using an app factory, the use of init_app means that you can 
create an instance of PyMongo with all your settings in your toplevel 
code, then attach it to the app when it's actually created (even if you 
create multiple apps, e.g. for unit testing). The central instance doesn't
actually keep a reference to the app, it merely sets everything up and 
uses `current_app` whenever it needs a reference to something stored on 
the app.
> 
> To be honest, if I were in charge of Flask extensions `self.app = app` 
in a central instance would be banned. What I would recommend instead for 
your particular case is:
> 
> - Don't store the configuration on the actual PyMongo instance, just the
configuration prefix. (That way, each instance represents a single 
prefix.)
> - In your init_app function, create the connection, but store it on the 
app in `app.extensions["pymongo"][PREFIX]`. Possibly do the same with the 
database, either by using a tuple or a cx/db wrapper class and stashing it
in the dictionary instead. Then set up the hooks like normal.
> - Whenever the methods or properties on the instance need the database 
or connection, they use 
`current_app.extensions["pymongo"][self.config_prefix]`.
> 
> This allows each instance to be used to refer to the database with the 
proper settings regardless of which app is active at the moment.

This mostly makes sense to me, but I'm not sure why the PyMongo shouldn't 
hold a reference to cx and db. Is this just The Way Flask Extensions Are 
Done (TM)? Also, thanks for the tip on current_app, which I had not seen 
before.


>> 4. General and specific feedback on the approach, code, utility, any 
missing features I should consider adding, etc.
> 
> ARGH MONKEYPATCHING DIE DIE DIE
> 
> In all seriousness, I'm not 100% sure monkeypatching is the best way to 
accomplish your find_or_404 method. What Flask-SQLAlchemy does for 
Web-specific methods like that is create a customized Query subclass with 
the methods it needs, then rig the session to produce those Queries 
instead of SQLAlchemy's defaults. Flask-CouchDB does something similar so 
Document will use the default database. It's a bit more work, but it 
doesn't muck about with other modules' classes, and the "Flask culture" in
general is very much against monkeypatching and global state.

I was hoping to avoid having to wrap PyMongo classes, but point taken.

- Dan

On Dec 27, 2011, at 20:48 , Dan Crosta wrote:
> This mostly makes sense to me, but I'm not sure why the PyMongo 
shouldn't hold a referenc e to cx and db. Is this just The Way Flask 
Extensions Are Done (TM)? Also, thanks for the tip on current_app, which I
had not seen before.

Because if you use the single PyMongo instance for multiple apps but store
cx and db on the PyMongo, the last database registered will be used for 
both apps.

Thanks,
Matthew Frazier
http://leafstorm.us/

On Dec 27, 2011, at 8:54 PM, Matthew Frazier wrote:
> On Dec 27, 2011, at 20:48 , Dan Crosta wrote:
>> This mostly makes sense to me, but I'm not sure why the PyMongo 
shouldn't hold a referenc e to cx and db. Is this just The Way Flask 
Extensions Are Done (TM)? Also, thanks for the tip on current_app, which I
had not seen before.
> 
> Because if you use the single PyMongo instance for multiple apps but 
store cx and db on the PyMongo, the last database registered will be used 
for both apps.

Ah... that's a weird concept to me, that the PyMongo is like a 
configuration callable (with `init_app`), rather than an instance with 
internal state. But it makes sense and it's easy enough to support. Thanks

- Dan

OK, I've integrated your suggestions. Since no one else seems to have 
volunteered any thoughts about Flask-PyMongo, and since I'm eager to blog 
about it, I've now launched the project at version 0.1:


PyPi: http://pypi.python.org/pypi/Flask-PyMongo/0.1
Docs: http://readthedocs.org/docs/flask-pymongo/en/0-1/
GitHub: https://github.com/dcrosta/flask-pymongo
Downloads: https://github.com/dcrosta/flask-pymongo/tags

Armin: can you add this to the Extensions page on flask.pocoo.org? 
Anything else you need from me?


Thanks!
- Dan

On Dec 27, 2011, at 9:07 PM, Dan Crosta wrote:

> On Dec 27, 2011, at 8:54 PM, Matthew Frazier wrote:
>> On Dec 27, 2011, at 20:48 , Dan Crosta wrote:
>>> This mostly makes sense to me, but I'm not sure why the PyMongo 
shouldn't hold a referenc e to cx and db. Is this just The Way Flask 
Extensions Are Done (TM)? Also, thanks f or the tip on current_app, which 
I had not seen before.
>> 
>> Because if you use the single PyMongo instance for multiple apps but 
store cx and db on the PyMongo, the last database registered will be used 
for both apps.
> 
> Ah... that's a weird concept to me, that the PyMongo is like a 
configuration callable (with `init_app`), rather than an instance with 
internal state. But it makes sense and it's easy enough to support. Thanks
> 
> - Dan
> 

On Wed, Dec 28, 2011 at 3:04 PM, Dan Crosta <dcrosta@late.am> wrote:
> OK, I've integrated your suggestions. Since no one else seems to have
> volunteered any thoughts about Flask-PyMongo, and since I'm eager to blog
> about it, I've now launched the project at version 0.1:
>
>
> PyPi: http://pypi.python.org/pypi/Flask-PyMongo/0.1
> Docs: http://readthedocs.org/docs/flask-pymongo/en/0-1/
> GitHub: https://github.com/dcrosta/flask-pymongo
> Downloads: https://github.com/dcrosta/flask-pymongo/tags
>
> Armin: can you add this to the Extensions page on flask.pocoo. org? Anything
> else you need from me?

Listed.
http://flask.pocoo.org/extensions/#Flask-PyMongo

Thanks,

Ron

Excellent, thanks!

- Dan


On Dec 29, 2011, at 10:53 PM, Ron DuPlain wrote:

> On Wed, Dec 28, 2011 at 3:04 PM, Dan Crosta <dcrosta@late.am> wrote:
>> OK, I've integrated your suggestions. Since no one else seems to have
>> volunteered any thoughts about Flask-PyMongo, and since I'm eager to blog
>> about it, I've now launched the project at version 0.1:
>> 
>> 
>> PyPi: http://pypi.python.org/pypi/Flask-PyMongo/0.1
>> Docs: http://readthedocs.org/docs/flask-pymongo/en/0-1/
>> GitHub: https://github.com/dcrosta/flask-pymongo
>> Downloads: https://github.com/dcrosta/flask-pymongo/tags
>> 
>> Armin: can you add this to the Extensions page on flask.pocoo. org? Anything
>> else you need from me?
> 
> Listed.
> http://flask.pocoo.org/extensions/#Flask-PyMongo
> 
> Thanks,
> 
> Ron

One more follow-up: how does one go about becoming an approved extension? 
I saw one or two issues against flask asking for approval, is that the 
preferred approach?

- Dan


On Dec 29, 2011, at 11:51 PM, Dan Crosta wrote:

> Excellent, thanks!
> 
> - Dan
> 
> 
> On Dec 29, 2011, at 10:53 PM, Ron DuPlain wrote:
> 
>> On Wed, Dec 28, 2011 at 3:04 PM, Dan Crosta <dcrosta@late.am> wrote:
>>> OK, I've integrated your suggestions. Since no one else seems to have
>>> volunteered any thoughts about Flask-PyMongo, and since I'm eager to blog
>>> about it, I've now launched the project at version 0.1:
>>> 
>>> 
>>> PyPi: http://pypi.python.org/pypi/Flask-PyMongo/0.1
>>> Docs: http://readthedocs.org/docs/flask-pymongo/en/0-1/
>>> GitHub: https://github.com/dcrosta/flask-pymongo
>>> Downloads: https://github.com/dcrosta/flask-pymongo/tags
>>> 
>>> Armin: can you add this to the Extensions page on flask.pocoo. org? Anything
>>> else you need from me?
>> 
>> Listed.
>> http://flask.pocoo.org/extensions/#Flask-PyMongo
>> 
>> Thanks,
>> 
>> Ron
> 

Le 31/12/2011 02:12, Dan Crosta a écrit :
> One more follow-up: how does one go about becoming an approved 
extension? I saw one or two issues against flask asking for approval, is 
that the preferred approach?
>
> - Dan

Hi,

The guidelines for approved extensions are here:

http://flask.pocoo.org/docs/extensiondev/#approved-extensions


For the actual listing as approved, as Ron said recently, you can get 
other people to post reviews to the issue tracker:


http://flask.pocoo.org/mailinglist/archive/2011/12/28/code-review-requested-for-flask-seasurf/#6f89ba6ec5b99b45f36adcba7c3b4091

https://github.com/mitsuhiko/flask/issues?labels=extension+review


Regards,
-- 
Simon Sapin

Got it, thanks. I must have missed that when it came by, my apologies.

- Dan


On Dec 30, 2011, at 10:24 PM, Simon Sapin wrote:

> Le 31/12/2011 02:12, Dan Crosta a écrit :
>> One more follow-up: how does one go about becoming an approved 
extension? I saw one or two issues against flask asking for approval, is 
that the preferred approach?
>> 
>> - Dan
> 
> Hi,
> 
> The guidelines for approved extensions are here:
> 
> http://flask.pocoo.org/docs/extensiondev/#approved-extensions
> 
> 
> For the actual listing as approved, as Ron said recently, you can get 
> other people to post reviews to the issue tracker:
> 
> 
http://flask.pocoo.org/mailinglist/archive/2011/12/28/code-review-requested-for-flask-seasurf/#6f89ba6ec5b99b45f36adcba7c3b4091
> 
> https://github.com/mitsuhiko/flask/issues?labels=extension+review
> 
> 
> Regards,
> -- 
> Simon Sapin

Over the past few days, I've been working on an extension to add PyMongo 
and MongoDB support to Flask. I'd appreciate feedback from Flask users 
(especially those using MongoDB) and developers.


Background:

There already exist two MongoDB-related extensions, Flask-MongoAlchemy and
Flask-MongoKit, but both require the use of an object-document mapper 
("ODM"). These require explicit data-modeling, or at the very least 
additional class definitions, in order to be useful. PyMongo and 
Flask-PyMongo work with native Python dictionaries, but requires the 
application developer to manage schemas, migrations, indexes, etc, by 
hand.


The Extension:

Using Flask-PyMongo primarily involves creating a PyMongo object (which 
takes a Flask app as a constructor argument), which consults configuration
and auto-connects to the database server. Each PyMongo object is designed 
to connect to a specific database (like most database management systems, 
MongoDB has a server > database > collection > document hierarchy, which 
is analagous to server > database > table > row in RDBMSes), though it 
exposes the connection object for access to any other database on the same
server.

Flask-PyMongo configures itself (by default) from config variables named 
beginning with "MONGO_". A second constructor argument allows the user to 
change this (which is necessary in the event that he/she wants to connect 
to several distinct MongoDB servers).


Documentation & Source:

docs: http://flask-pymongo.readthedocs.org/en/latest/index.html
source: https://github.com/dcrosta/flask-pymongo


Questions:

1. I notice that most Flask extensions have documentation at 
packages.python.org rather than Read the Docs. Any particular reason for 
this?
2. The extension development guide suggests adding an "init_app" function 
-- what is the use case in which this is necessary?
3. The extension development guide says "there must be a link to 
automatically install the development version (PackageName==dev)" -- what 
exactly needs to be done for this? (Flask-PyMongo is not yet listed in 
PyPi, as I've yet to release version 0.1)
4. General and specific feedback on the approach, code, utility, any 
missing features I should consider adding, etc.


Thanks!
- Dan