librelist archives

Re: [flask] Need Input: New Module System

From:

Jukka Männistö

Date:

2010-10-24 @ 20:20

Hi,


New potential Flask user here.

This is my almost-first-time participating in a mailing list, so
please bear with me.. :) Feel free to give me some pointers on what to
do here.



The docs talk about separating your view handlers into multiple python modules:

http://flask.pocoo.org/docs/patterns/packages/#larger-applications


But these two rules seem somewhat confusing:

1) The Flask application object creation has to be in the __init__.py
file. That way each module can import it safely and the __name__
variable will resolve to the correct package.

2) All the view functions (the ones with a route() decorator on top)
have to be imported when in the __init__.py file. Not the object
itself, but the module it is in. Do the importing at the bottom of the
file.



First, I don't like the idea of having to "hide" stuff in the
__init__.py file - it would be much more pleasant & clear to have
everything "out in the open", in a normal python file.

Second, I'm confused about how the __name__ variable behaves/is used,
and how it's related to me connecting my handler methods to URLs.

- Does Flask's operation depend on it? As far as I understand, it may
be used as a reference to your app's root directory, to be able to
find ./templates, etc. Or, it may be used to make url_for() work?
Wouldn't it be possible to just give something in Flask a direct
reference to my app's root directory?

I don't really need url_for() to work - I'm content with writing
hard-coded relative URLs into my templates etc.

Anyway, I've looked into CherryPy too, and I kind of like its "routing" system.

As you may know, you give CherryPy a "root" object for your
application, which will correspond to the root path "/" in your app.
Then, any additional path in the URL will be matched against methods
in your root object, to an arbitrary depth.

For example, for the path "/admin", CherryPy would look into your root
application object, and see if there's a member called "admin", and
see if it can be invoked to handle the URL. Then you could have
something like "/admin/login", with the same deal further along the
object hierarchy, and so on.

I like this kind of arrangement, because you don't have to enumerate
all of the possible URLs to be handled, with all their handler
methods.

In CherryPy though, you have to mark all handlers as "exposed", by
introducing an "exposed = True" property on them, to make them
actually respond to corresponding URLs, which I found to be a bit ugly
/ verbose. This is for security reasons of course, but I'm not
convinced it's entirely necessary. Maybe I just don't know enough :)
Luckily you can fullfill that requirement neatly with some metaclass
magic.


OK, back to the Flask docs. There's an example with the following
directory structure:

/yourapplication
    /yourapplication
        /__init__.py
        /views
            __init__.py
            admin.py

- Where some view handlers live in admin.py. It's then explained that
in this case, you'd have to introduce handler methods in admin.py like
this:


from flask import Module

admin = Module(__name__)

@admin.route('/')
def index():
    pass

@admin.route('/login')
def login():
    pass


-- Again, there's __name__ in the mix, and again I'm confused about
why..  Of course, I don't know how Flask works, but as a new user,
this feels somewhat cludgy.

Couldn't I just somehow tell Flask that I've got a module named
"admin.py" (in a certain package), that corresponds to the path
"/admin", and then Flask would automatically find the methods "index"
and "login" (eg. "/admin/login") as necessary? ("index" could
automatically be invoked for "/" too)

I suppose this would be similar to CherryPy, but hopefully without the
requirement of "exposing" stuff explicitly.

If there's a method you _don't_ want exposed, maybe there could be a
way to mark it hidden? Or maybe you could just settle for making view
handler modules where every method is supposed to be exposed? :p

As an additional bonus, it would be great if Flask fed the handler
methods parameters straight from the URL, like
"/admin/login/<username>/<password>", for "def login(username,
password):", but it seems Armin thinks this would lead to trouble..
What kind of trouble would it be, by the way? :)

IIRC, now Flask has "request.args", and "request.form", which are
alright, but the aforementioned way would result in less typing when
making your app. Less things to type is usually a good thing :)


Alright, that's enough rambling for now..  I should go to sleep!




- Jukka












On Mon, Oct 25, 2010 at 2:37 AM, Thadeus Burgess <thadeusb@thadeusb.com> wrote:
> I am fairly happy with flask just the way it is. Albeit, I'm not doing
> things such as static files within modules, nor do I have any use for
> pluggable components with the sites that I write. If I wanted to plug in a
> module that only does 80% of what I need yet need to take 120% of my time to
> modify it to work exactly how I want then I would use django. With flask, it
> allows me the freedom to design my system exactly how it needs to be to all
> work together nicely.
>
> Idea for static files, though I might be missing part of the larger story.
>
> 1. Have a command that ships with flask that exports all static files to a
> single directory ready for deployment. During development, flask exposes
> static files as domain.com/static/<module name>/<module static files> or
> domain.com/static/<app static files>
> 2. Create a separate routing function just for static files. Call the
> function static, configure it with a domain with the flask app config.
> url_for is for internal functions, static for static files. Something
> like...
>
> flask.config['STATIC_URL'] = 'http://fs1.domain.com'
>
> If the above value is None, then route to http://domain.com/static/,
> otherwise use the above value as the root.
>
> If the files don't exist at the target static url, then they don't exist. I
> don't know if its flasks job to know what is there and what is not except
> during development.
>
> if you called ``static('js/jquery.js')`` it would translate to one of
> http://fs1.domain.com/js/jquery.js or
> http://127.0.0.1:5000/static/js/jquery.js.
>
> --
> Thadeus
>
>
>
>
> On Sun, Oct 24, 2010 at 12:06 PM, Armin Ronacher
> <armin.ronacher@active-4.com> wrote:
>>
>> Hi everybody,
>>
>> Everybody that worked with flask.Module so far will probably have
>> figured out now that it's far from perfect.  I am currently thinking
>> about an alternative solution and I have two independent things on local
>> branches plus a separate fix for the static file problem on github, but
>> I am totally not convinced that any of my attempts is useful.
>>
>> Last time I tackled the module problem I got feedback that this is a
>> good solution but we obviously missed a few things there and now it's
>> pretty clear that the design for that is unfortunately pretty broken.
>>
>> I encourage all of you to share your ideas on how a pluggable
>> architecture for Flask would look like.
>>
>> The following things should be kept in mind:
>>
>>   - how does it affect circular imports?
>>   - can a module be used more than once?
>>   - what is a module, what is the Flask object?
>>   - what about modules that need configuration?
>>   - what about modules shipped as flask extensions?
>>
>> The whole pluggable application wish is a huge mess, not just in Flask,
>> also in Django and they are currently also trying hard to clean it up.
>> Unfortunately their plan is very, very close to our current solution
>> which does not help us much (at least they got static files figured out).
>>
>> The problem with static files currently is that they are magically
>> exposed and if two modules are mounted on the same path, undefined magic
>> happens.  Furthermore GAE deploys static files to a different server
>> which breaks static files.  Last but not least our static exports do not
>> properly support S3.
>>
>> So if you have ideas, please share it here and discuss that.  I will
>> reveal my current attempts somewhere later that week, I just want to
>> avoid that some solutions fall under the table because I bring some bias
>> into that discussion :)
>>
>>
>> Regards,
>> Armin
>
>

As a way of grouping views in larger applications, while avoiding
circular import issues, the current module system works well - it's
similar in principle to controllers in Rails and Pylons and likewise
allows the developer to group similar views under a single URL prefix,
with before/after request rules and so forth.

It may need some tinkering - for example, as Thadeus pointed out, it
would be better to have static files managed outside of modules as
they are really separate things. But the basic idea is fine.

I don't think "pluggable apps" are a practical goal. As the the Django
guys have found out, the idea of a bundled app which includes views,
templates, media, models etc leads to a coarse-grained component
that's difficult to reuse without a great deal of compromise -
contrib.auth being the most notorious example. Extensions are a good
way of providing generic functionality, on the other hand (and the
most useful Django apps are in fact extensions in all but name, such
as south or debug toolbar).

In addition, pluggable apps don't make much sense in Flask due to
having no predefined ORM - you couldn't make a comments app for
example without having to use e.g. SQLAlchemy or MongoDB, which would
decrease its usefulness for those projects using a completely
different backend.

> I don't think "pluggable apps" are a practical goal.

All your points were quite valid.   Django is great really, it's the
users who have forgotten it's origin:  It was extracted from their
workplace, a newspaper.
Implicit in some django constructs is that team and that environment.
The settings system.  The pluggable apps.  The vague suggestions about
directory structure.  The ghastly deployment.  The admin app.  All
that makes sense at LJWorld, I assume.  But seriously, enough about
django.  I don't even think Flask and Django share many use cases.

   We keep mentioning django, but  what other places can we draw
inspiration from?  How does repoze do pluggables?  ( Do they? )  Or,
let's be original!

I'd suggest for any pluggable app the same sort of process that the
extensions have.  Even something automated , to keep the quality high.
 ( I'd also love it to be used on pypi, but that is another issue.)

The admin may not be you, or a python programmer, so I'd vote for
something like an .ini file if it has to be configured.  Writing
config files in python is great, but everyone can read .ini files.  or
yaml.

I disagree with the sense ... no predefined ORM point, since an app
does would not serve all people anyway.  I can confidently state that
I'll be using SQLAlchemy if i'm relational... and that is the sort of
app I might just want to get.



So, the unit of code reuse is the extension in flask?









What is the basic unit of code reuse in Flask?  The extension?

I am with danjac354 on this. Module system works quite well here without
breaking the flexibility.


On Sun, Oct 24, 2010 at 10:36 PM, danjac354@gmail.com
<danjac354@gmail.com>wrote:

> As a way of grouping views in larger applications, while avoiding
> circular import issues, the current module system works well - it's
> similar in principle to controllers in Rails and Pylons and likewise
> allows the developer to group similar views under a single URL prefix,
> with before/after request rules and so forth.
>
> It may need some tinkering - for example, as Thadeus pointed out, it
> would be better to have static files managed outside of modules as
> they are really separate things. But the basic idea is fine.
>
> I don't think "pluggable apps" are a practical goal. As the the Django
> guys have found out, the idea of a bundled app which includes views,
> templates, media, models etc leads to a coarse-grained component
> that's difficult to reuse without a great deal of compromise -
> contrib.auth being the most notorious example. Extensions are a good
> way of providing generic functionality, on the other hand (and the
> most useful Django apps are in fact extensions in all but name, such
> as south or debug toolbar).
>
> In addition, pluggable apps don't make much sense in Flask due to
> having no predefined ORM - you couldn't make a comments app for
> example without having to use e.g. SQLAlchemy or MongoDB, which would
> decrease its usefulness for those projects using a completely
> different backend.
>



-- 
--
Dejan Noveski
Web Developer
dr.mote@gmail.com
Twitter: http://twitter.com/dekomote | LinkedIn:
http://mk.linkedin.com/in/dejannoveski

Atomidata http://www.atomidata.com

For most parts I agree with danjac, however the one issue I have with module
is the fact that you cant register sub modules, an example might clarify:

App
   +-- Authentication <Module>
       +-- Verification <Module>
       +-- Management <Module>
   +--Foo<Module>

This would allow modules to be small and focused. I currently "monkey patch"
my way around it, but it would be nice if this were built in.

*Somewhere in my mind, is a drill Sergent constantly badgering me that; a
function mustn't be longer than 70 lines and that a module shouldn't contain
any more than 5 such functions...
*
Regards,
Mayowa.

On Mon, Oct 25, 2010 at 1:29 AM, Dejan Noveski <dr.mote@gmail.com> wrote:

> I am with danjac354 on this. Module system works quite well here without
> breaking the flexibility.
>
>
> On Sun, Oct 24, 2010 at 10:36 PM, danjac354@gmail.com <danjac354@gmail.com
> > wrote:
>
>> As a way of grouping views in larger applications, while avoiding
>> circular import issues, the current module system works well - it's
>> similar in principle to controllers in Rails and Pylons and likewise
>> allows the developer to group similar views under a single URL prefix,
>> with before/after request rules and so forth.
>>
>> It may need some tinkering - for example, as Thadeus pointed out, it
>> would be better to have static files managed outside of modules as
>> they are really separate things. But the basic idea is fine.
>>
>> I don't think "pluggable apps" are a practical goal. As the the Django
>> guys have found out, the idea of a bundled app which includes views,
>> templates, media, models etc leads to a coarse-grained component
>> that's difficult to reuse without a great deal of compromise -
>> contrib.auth being the most notorious example. Extensions are a good
>> way of providing generic functionality, on the other hand (and the
>> most useful Django apps are in fact extensions in all but name, such
>> as south or debug toolbar).
>>
>> In addition, pluggable apps don't make much sense in Flask due to
>> having no predefined ORM - you couldn't make a comments app for
>> example without having to use e.g. SQLAlchemy or MongoDB, which would
>> decrease its usefulness for those projects using a completely
>> different backend.
>>
>
>
>
> --
> --
> Dejan Noveski
> Web Developer
> dr.mote@gmail.com
> Twitter: http://twitter.com/dekomote | LinkedIn:
> http://mk.linkedin.com/in/dejannoveski
>
> Atomidata http://www.atomidata.com
>
>

On Mon, 2010-10-25 at 04:29 +0100, Mayowa Akinyemi wrote:
> For most parts I agree with danjac, however the one issue I have with
> module
> is the fact that you cant register sub modules 

Just make authentication a Python package. You will have to register
them all in your app of course but I don't think it's that bad. Can even
do it dynamically with a loop. Werkzeug even has a helper function for
finding modules below a package: werkzeug.find_modules. Or export a list
of your Modules in your package and iterate that.

Dag,
The way I see it, the value in using modules is compartmentalization and (to
a certain extent) encapsulation; consequently, registering them with the app
instance kinda defeats the purpose. Ultimately the app will have to be made
aware of the (for routing) but it would be more "pluggable" if it was done
in the module.

I already have a solution that I can wrap my head around, but I'll have a
look at werkzeug.find_modules.
However I do think this kinda thing should be part of the framework or at
least documented, so that its obvious (to folks who just want to use the
framework and not muck around with the plumbing).



On Mon, Oct 25, 2010 at 7:13 AM, Dag Odenhall <dag.odenhall@gmail.com>wrote:

> On Mon, 2010-10-25 at 04:29 +0100, Mayowa Akinyemi wrote:
> > For most parts I agree with danjac, however the one issue I have with
> > module
> > is the fact that you cant register sub modules
>
> Just make authentication a Python package. You will have to register
> them all in your app of course but I don't think it's that bad. Can even
> do it dynamically with a loop. Werkzeug even has a helper function for
> finding modules below a package: werkzeug.find_modules. Or export a list
> of your Modules in your package and iterate that.
>
>
>
>

First, apologies for my last email, I was pretty tired.  I thought I
typed this at the top.

Armin,
Can you tell us why you feel the way you do about flask.Module?  You
seem to feel strongly that it will not be sufficient.   What is _your_
vision for, and where would you demarcate "pluggable functionality?"
What are the pitfalls

Or, do you even need to solve this?  Flask works now.  Its cool and
you can put stuff online and all that sort of thing, just like the
docs say.
Although, people who want to host images on S3, or separate domains,
are thinking big. To my eyes, as long as it's not an async server

> least documented, so that its obvious (to folks who just want to use the
> framework and not muck around with the plumbing).

Agreed!  That seems to be the division here, Flask as goal or Flask as
tool to achieve a goal.

>  Ultimately the app will have to be made aware of the (for routing) but 
it would be more "pluggable" if it was done in the module.

Again +1.  I'm uncomfortable with discarding the idea of code reuse.

Of course, we could always turn it around, so to speak, make what is
currently "Flask" the rough equivalent of an 'app' and put a bunch of
then on different mount points.  Then everything is the unchanged:
small jobs, 1 flask, big jobs, more flasks.  This might be lunacy.   I
am reading

http://projects.unbit.it/uwsgi/wiki/ApplicationsDict

while typing this, and perhaps that is a uwsgi thing only...

Talk of pluggable apps aside, it would be useful to be able to have
submodules (or indeed allow apps to be used as modules, if that's
possible).

Django isn't the only place where pluggable apps have been tried -
Zope3 components are another example. However I think the concept is
more useful in a CMS such as Drupal where you can make a lot of
assumptions about the environment - for example, things like the user
model, permissions system, db backend etc are a given. Django has very
obvious CMS origins and so has inherited this very CMS-like feature.

A more common use case is that you have a basic application that needs
to be "skinned" for different clients. Flask-Themes is great for this
- you can change templates and media - and signals allow cross-cutting
functionality (e.g. send an email to this client on signup). However
it would be handy to be able to take an existing module and replace it
or swap out individual URLs. Something like this:

@app.get_module("signup").route
def custom_signup():
   ....

In newsmeme at the moment I'm leaning toward wrapping the
configuration in a class (containing the Flask app) to allow for
overridable behaviour:

http://bitbucket.org/danjac/newsmeme/src/tip/newsmeme/application.py

Having more fine-grained control over modules would definitely be an
added benefit.

On 25 October 2010 07:32, Mayowa Akinyemi <mayowa@gmail.com> wrote:
> Dag,
> The way I see it, the value in using modules is compartmentalization and (to
> a certain extent) encapsulation; consequently, registering them with the app
> instance kinda defeats the purpose. Ultimately the app will have to be made
> aware of the (for routing) but it would be more "pluggable" if it was done
> in the module.
>
> I already have a solution that I can wrap my head around, but I'll have a
> look at werkzeug.find_modules.
> However I do think this kinda thing should be part of the framework or at
> least documented, so that its obvious (to folks who just want to use the
> framework and not muck around with the plumbing).
>
>
>
> On Mon, Oct 25, 2010 at 7:13 AM, Dag Odenhall <dag.odenhall@gmail.com>
> wrote:
>>
>> On Mon, 2010-10-25 at 04:29 +0100, Mayowa Akinyemi wrote:
>> > For most parts I agree with danjac, however the one issue I have with
>> > module
>> > is the fact that you cant register sub modules
>>
>> Just make authentication a Python package. You will have to register
>> them all in your app of course but I don't think it's that bad. Can even
>> do it dynamically with a loop. Werkzeug even has a helper function for
>> finding modules below a package: werkzeug.find_modules. Or export a list
>> of your Modules in your package and iterate that.
>>
>>
>>
>
>

 However I think the concept is
> more useful in a CMS such as Drupal where you can make a lot of
> assumptions about the environment - for example, things like the user
> model, permissions system, db backend etc are a given.

Odd that you say that about Drupal, I was thinking the loop to
register applications that Dag proposed reminded me of Drupal a few
years ago.

Otherwise your point is well taken.  Extensions and middleware would
be sufficient.

> Otherwise your point is well taken.  Extensions and middleware would
> be sufficient.

Forgot to mention (WSGI) middleware, which is another good example of
reuse, although at a higher level of abstraction.