librelist archives

Configuration Support

From:

Armin Ronacher

Date:

2010-05-18 @ 00:47

Hi,

I created a branch for the configuration support now.  The idea is that 
the application has a .config object attached that holds at least the 
following values: debug, secret_key, session_cookie_name, 
permanent_session_lifetime, use_x_sendfile

The config can also be used by extensions to configure the extension and 
the application itself to store its own config values.

There are two ways currently to populate a config:

     # load from a file with python syntax
     app.config.from_pyfile('myconfig.cfg')

     # load from any module, including the calling one
     app.config.from_module('the.module')

You can also tell it to load from the current module.  This makes it 
possible to define the config keys in the same module:

    # configuration here
     DEBUG = True
     SECRET_KEY = 'devkey'

     app.config.from_module(__name__)

The config only stores keys in uppercase.  The attributes on the class 
transparently mirror the setting in the dict:

     >>> app.config['debug'] = True
     >>> app.debug
     True
     >>> app.debug = False
     >>> app.debug
     False

The problems left to solve are: documentation.  I am not exactly sure 
how to properly document that, which is why it's still in a branch. 
Also there might be some other things that would be good to solve.  The 
big question is also if it would make sense to add a proxy to the config 
like with request and g, or if "current_app.config" is good enough for 
extensions and if the application itself should do app.config.

The biggest issue however is the bootstrapping.  If the config is loaded 
in the .wsgi file for instance, the import machinery might have already 
initialized SQLAlchemy (for example) with the wrong (the default) 
database URI.  And for that, i don't really have a solution except for 
"load the config in the __init__.py as early as possible" like Django 
does.  This however is not very flexible because it would mean one has 
to define the config as environment parameter (FLASK_CONFIG_FILENAME / 
FLASK_CONFIG_MODULE or something).

Any ideas on that?


Regards,
Armin

PS.: I hope I was clear enough with the last issue :)

This looks pretty cool. A whole new proxy for the configuration
shouldn't be necessary. I'm assuming this is targeted for 0.3 if it
works all right?

I can't think of any immediate, really good answers to the bootstrapping
situation. For some things, it might be possible to use a LocalProxy to
the configuration, but that still wouldn't work if the value is used
immediately.

One possible solution uses an app.on_setup decorator - when the first
request is handled, all the functions registered with on_setup are
called. By declaring the variable global during the function you're
on_setup-ing, you can set up whatever then. It's also possible to use
instance variables and a LocalProxy (I really like LocalProxy, can't you
tell?) for a similar effect. I have examples of both on the pastebin at
http://paste.pocoo.org/show/215101/.

Of course, there are a few problems with this approach, all relating to
concurrency. For one, keeping track of an app's "first request" could be
difficult and is probably not threadsafe. The obvious problem is that
you could get a first request, then, milliseconds later, get another
request while the first request is still setting up the database.
Instant 500 as the app tries to access something that isn't there (or
sets it up twice). Even with a lock, there would still be either race
conditions or performance issues due to constant locking (and probably
both). And I have no idea how non-threading concurrency models would
work with this.

Still, it's the best solution I can think of. You can probably fix the
remaining problems, or maybe think of something better. Until then, we
just need to encourage people to write lazy code.

By the way, for the SQLAlchemy problem, a solution similar to the one
used in SimpleWiki would work now, without an on_setup decorator (see

http://bitbucket.org/mitsuhiko/werkzeug-main/src/tip/examples/simplewiki/database.py).
Simply pass a custom function to scoped_session (instead of a
sessionmaker) that gets the engine from the application object, and
create said engine when the app is having its configuration loaded.

On 05/17/2010 08:47 PM, Armin Ronacher wrote:
> Hi,
> 
> I created a branch for the configuration support now.  The idea is that 
> the application has a .config object attached that holds at least the 
> following values: debug, secret_key, session_cookie_name, 
> permanent_session_lifetime, use_x_sendfile
> 
> The config can also be used by extensions to configure the extension and 
> the application itself to store its own config values.
> 
> There are two ways currently to populate a config:
> 
>      # load from a file with python syntax
>      app.config.from_pyfile('myconfig.cfg')
> 
>      # load from any module, including the calling one
>      app.config.from_module('the.module')
> 
> You can also tell it to load from the current module.  This makes it 
> possible to define the config keys in the same module:
> 
>     # configuration here
>      DEBUG = True
>      SECRET_KEY = 'devkey'
> 
>      app.config.from_module(__name__)
> 
> The config only stores keys in uppercase.  The attributes on the class 
> transparently mirror the setting in the dict:
> 
>      >>> app.config['debug'] = True
>      >>> app.debug
>      True
>      >>> app.debug = False
>      >>> app.debug
>      False
> 
> The problems left to solve are: documentation.  I am not exactly sure 
> how to properly document that, which is why it's still in a branch. 
> Also there might be some other things that would be good to solve.  The 
> big question is also if it would make sense to add a proxy to the config 
> like with request and g, or if "current_app.config" is good enough for 
> extensions and if the application itself should do app.config.
> 
> The biggest issue however is the bootstrapping.  If the config is loaded 
> in the .wsgi file for instance, the import machinery might have already 
> initialized SQLAlchemy (for example) with the wrong (the default) 
> database URI.  And for that, i don't really have a solution except for 
> "load the config in the __init__.py as early as possible" like Django 
> does.  This however is not very flexible because it would mean one has 
> to define the config as environment parameter (FLASK_CONFIG_FILENAME / 
> FLASK_CONFIG_MODULE or something).
> 
> Any ideas on that?
> 
> 
> Regards,
> Armin
> 
> PS.: I hope I was clear enough with the last issue :)


-- 
Regards,
LeafStorm (http://www.leafstorm.us/)

I'm not sure this solves any problems, and creates new ones (i.e. the
bootstrapping issue).

Importing a config module where you need it is straightforward, and
avoids bootstrapping issues.

Plus, having more than one way to do it makes it harder to document -
I would go with either loading a config file, or loading a Python
module, but not both.

On 18 May 2010 03:48, LeafStorm <leafstormrush@gmail.com> wrote:
> This looks pretty cool. A whole new proxy for the configuration
> shouldn't be necessary. I'm assuming this is targeted for 0.3 if it
> works all right?
>
> I can't think of any immediate, really good answers to the bootstrapping
> situation. For some things, it might be possible to use a LocalProxy to
> the configuration, but that still wouldn't work if the value is used
> immediately.
>
> One possible solution uses an app.on_setup decorator - when the first
> request is handled, all the functions registered with on_setup are
> called. By declaring the variable global during the function you're
> on_setup-ing, you can set up whatever then. It's also possible to use
> instance variables and a LocalProxy (I really like LocalProxy, can't you
> tell?) for a similar effect. I have examples of both on the pastebin at
> http://paste.pocoo.org/show/215101/.
>
> Of course, there are a few problems with this approach, all relating to
> concurrency. For one, keeping track of an app's "first request" could be
> difficult and is probably not threadsafe. The obvious problem is that
> you could get a first request, then, milliseconds later, get another
> request while the first request is still setting up the database.
> Instant 500 as the app tries to access something that isn't there (or
> sets it up twice). Even with a lock, there would still be either race
> conditions or performance issues due to constant locking (and probably
> both). And I have no idea how non-threading concurrency models would
> work with this.
>
> Still, it's the best solution I can think of. You can probably fix the
> remaining problems, or maybe think of something better. Until then, we
> just need to encourage people to write lazy code.
>
> By the way, for the SQLAlchemy problem, a solution similar to the one
> used in SimpleWiki would work now, without an on_setup decorator (see
> 
http://bitbucket.org/mitsuhiko/werkzeug-main/src/tip/examples/simplewiki/database.py).
> Simply pass a custom function to scoped_session (instead of a
> sessionmaker) that gets the engine from the application object, and
> create said engine when the app is having its configuration loaded.
>
> On 05/17/2010 08:47 PM, Armin Ronacher wrote:
>> Hi,
>>
>> I created a branch for the configuration support now.  The idea is that
>> the application has a .config object attached that holds at least the
>> following values: debug, secret_key, session_cookie_name,
>> permanent_session_lifetime, use_x_sendfile
>>
>> The config can also be used by extensions to configure the extension and
>> the application itself to store its own config values.
>>
>> There are two ways currently to populate a config:
>>
>>      # load from a file with python syntax
>>      app.config.from_pyfile('myconfig.cfg')
>>
>>      # load from any module, including the calling one
>>      app.config.from_module('the.module')
>>
>> You can also tell it to load from the current module.  This makes it
>> possible to define the config keys in the same module:
>>
>>     # configuration here
>>      DEBUG = True
>>      SECRET_KEY = 'devkey'
>>
>>      app.config.from_module(__name__)
>>
>> The config only stores keys in uppercase.  The attributes on the class
>> transparently mirror the setting in the dict:
>>
>>      >>> app.config['debug'] = True
>>      >>> app.debug
>>      True
>>      >>> app.debug = False
>>      >>> app.debug
>>      False
>>
>> The problems left to solve are: documentation.  I am not exactly sure
>> how to properly document that, which is why it's still in a branch.
>> Also there might be some other things that would be good to solve.  The
>> big question is also if it would make sense to add a proxy to the config
>> like with request and g, or if "current_app.config" is good enough for
>> extensions and if the application itself should do app.config.
>>
>> The biggest issue however is the bootstrapping.  If the config is loaded
>> in the .wsgi file for instance, the import machinery might have already
>> initialized SQLAlchemy (for example) with the wrong (the default)
>> database URI.  And for that, i don't really have a solution except for
>> "load the config in the __init__.py as early as possible" like Django
>> does.  This however is not very flexible because it would mean one has
>> to define the config as environment parameter (FLASK_CONFIG_FILENAME /
>> FLASK_CONFIG_MODULE or something).
>>
>> Any ideas on that?
>>
>>
>> Regards,
>> Armin
>>
>> PS.: I hope I was clear enough with the last issue :)
>
>
> --
> Regards,
> LeafStorm (http://www.leafstorm.us/)
>

Hi,

On 2010-05-18 9:05 AM, Dan Jacob wrote:
> Importing a config module where you need it is straightforward, and
> avoids bootstrapping issues.
Only if everything imports the same config module of if that happens 
very early on in the __init__.py.

A possible solution would be this (in the __init__.py, at the top of the 
file):

     import os
     from flask import Flask
     app = Flask(__name__)
     app.config.from_pyfile(os.environ.get('FLASK_CONFIG_FILE')
                            or 'development-settings.cfg')

The database file can then do something like this:

     from yourapplication import app
     from sqlalchemy import create_engine
     engine = create_engine(app.config['sqlalchemy_database_uri'])

I'm just not sure if environment variables + fallback are the best idea. 
  If they are, I would consider even adding a function for that to the 
config object.  That is at least what I do most of the time, because 
environment variables can be modified easily from the shell running that 
thing and the .wsgi file, before the code is imported, somewhat solving 
that problem.

Regards,
Armin

Hi,

On 2010-05-18 11:09 AM, Armin Ronacher wrote:
>  If they are, I would consider even adding a function for that to the
> config object.
That function would look like this:

     app.config.from_pyfile(envvar='MYAPP_CONFIG_FILE',
                            fallback='development-settings.cfg')

The FLASK_CONFIG_FILE was just an example.  Each app would have to use 
its own environment variable name, otherwise you could not have multiple 
Flask apps running in the same Python interpreter.

Regards,
Armin

I wonder if the best solution would be to separate the config object
from the app - is there any reason to keep it there ?

Suppose you have 3 modules:

__init__.py - where you initialize your Flask app
config.py - your config object sits here
database.py - where your DB lives

In your config.py:

from flask import Config

config = Config.from_module(....)  # or Config.from_cfg(....)

Then, just import it into your other modules:

from config import config

app = Flask(__name__)
app.config = config

That avoids having to lazily bind your app all over the place.

One thing to absolutely avoid is using environment variables - one of
my pet hates with Django ;-)

On 18 May 2010 10:11, Armin Ronacher <armin.ronacher@active-4.com> wrote:
> Hi,
>
> On 2010-05-18 11:09 AM, Armin Ronacher wrote:
>>  If they are, I would consider even adding a function for that to the
>> config object.
> That function would look like this:
>
>     app.config.from_pyfile(envvar='MYAPP_CONFIG_FILE',
>                            fallback='development-settings.cfg')
>
> The FLASK_CONFIG_FILE was just an example.  Each app would have to use
> its own environment variable name, otherwise you could not have multiple
> Flask apps running in the same Python interpreter.
>
> Regards,
> Armin
>

Hi,

On 2010-05-18 11:26 AM, Dan Jacob wrote:
> I wonder if the best solution would be to separate the config object
> from the app - is there any reason to keep it there ?
What would be the reason to not have it there?  The bootstrapping issue 
remains.  If the idea of your proposal is that inside the application 
everything does "from yourapplication.config import config" then I'm 
strongly against that because this means that if you try to change the 
config module you have to adapt all the imports.

Also it does not solve anything in comparison with the "from 
yourapplication import app; app.config"

The real issue is: how to make sure the config is fully loaded in the 
__init__.py.  Because there i can't call a function to init the system 
with a specific filename, that just does not work.


Regards,
Armin

Generally I use the pattern of "from local_config import *" in my
config module, so I can override with local settings. I can then just
import the config module like any other module.

Granted though that is perhaps less maintainable in larger applications.


On 18 May 2010 10:33, Armin Ronacher <armin.ronacher@active-4.com> wrote:
> Hi,
>
> On 2010-05-18 11:26 AM, Dan Jacob wrote:
>> I wonder if the best solution would be to separate the config object
>> from the app - is there any reason to keep it there ?
> What would be the reason to not have it there?  The bootstrapping issue
> remains.  If the idea of your proposal is that inside the application
> everything does "from yourapplication.config import config" then I'm
> strongly against that because this means that if you try to change the
> config module you have to adapt all the imports.
>
> Also it does not solve anything in comparison with the "from
> yourapplication import app; app.config"
>
> The real issue is: how to make sure the config is fully loaded in the
> __init__.py.  Because there i can't call a function to init the system
> with a specific filename, that just does not work.
>
>
> Regards,
> Armin
>
>

Hi,

On 2010-05-18 11:40 AM, Dan Jacob wrote:
> Generally I use the pattern of "from local_config import *" in my
> config module, so I can override with local settings. I can then just
> import the config module like any other module.
In the new config speak, you can easily do that:

     app.config.from_module('yourapplication.default_settings')
     app.config.from_pyfile('settings.cfg')

Then the defualts are loaded from a module and then overriden from a 
config file which is unversioned.


Regards,
Armin

hi,

i had a look at config-support branch, i think that a notion of simple
namespace could be usefull for flaskext (and maybe modules)
the namespace would be the __name__ of the extension

eg:
ConfigAttribute('sqlalchimy.username')

Flask config would be in the flask ns:
ConfigAttribute('flask.session_cookie_name')

and user defined options could be in the default namespace
ConfigAttribute('my_option')

what do you think?

Sebastien Estienne



On Tue, May 18, 2010 at 11:52, Armin Ronacher
<armin.ronacher@active-4.com> wrote:
> Hi,
>
> On 2010-05-18 11:40 AM, Dan Jacob wrote:
>> Generally I use the pattern of "from local_config import *" in my
>> config module, so I can override with local settings. I can then just
>> import the config module like any other module.
> In the new config speak, you can easily do that:
>
>     app.config.from_module('yourapplication.default_settings')
>     app.config.from_pyfile('settings.cfg')
>
> Then the defualts are loaded from a module and then overriden from a
> config file which is unversioned.
>
>
> Regards,
> Armin
>

What are the advantages of using a configuration module and just plain
importing a websiteconfig.py ?

--
Thadeus





On Mon, May 17, 2010 at 7:47 PM, Armin Ronacher
<armin.ronacher@active-4.com> wrote:
> Hi,
>
> I created a branch for the configuration support now.  The idea is that
> the application has a .config object attached that holds at least the
> following values: debug, secret_key, session_cookie_name,
> permanent_session_lifetime, use_x_sendfile
>
> The config can also be used by extensions to configure the extension and
> the application itself to store its own config values.
>
> There are two ways currently to populate a config:
>
>     # load from a file with python syntax
>     app.config.from_pyfile('myconfig.cfg')
>
>     # load from any module, including the calling one
>     app.config.from_module('the.module')
>
> You can also tell it to load from the current module.  This makes it
> possible to define the config keys in the same module:
>
>    # configuration here
>     DEBUG = True
>     SECRET_KEY = 'devkey'
>
>     app.config.from_module(__name__)
>
> The config only stores keys in uppercase.  The attributes on the class
> transparently mirror the setting in the dict:
>
>     >>> app.config['debug'] = True
>     >>> app.debug
>     True
>     >>> app.debug = False
>     >>> app.debug
>     False
>
> The problems left to solve are: documentation.  I am not exactly sure
> how to properly document that, which is why it's still in a branch.
> Also there might be some other things that would be good to solve.  The
> big question is also if it would make sense to add a proxy to the config
> like with request and g, or if "current_app.config" is good enough for
> extensions and if the application itself should do app.config.
>
> The biggest issue however is the bootstrapping.  If the config is loaded
> in the .wsgi file for instance, the import machinery might have already
> initialized SQLAlchemy (for example) with the wrong (the default)
> database URI.  And for that, i don't really have a solution except for
> "load the config in the __init__.py as early as possible" like Django
> does.  This however is not very flexible because it would mean one has
> to define the config as environment parameter (FLASK_CONFIG_FILENAME /
> FLASK_CONFIG_MODULE or something).
>
> Any ideas on that?
>
>
> Regards,
> Armin
>
> PS.: I hope I was clear enough with the last issue :)
>

Hi,

On 2010-05-18 3:59 AM, Thadeus Burgess wrote:
> What are the advantages of using a configuration module and just plain
> importing a websiteconfig.py ?
What from_module does is filling a dict, importing that won't do it. 
The reason I don't like the idea of just using modules here is that they 
pull in all the information.  The from_module function just takes the 
uppercase letters which makes it possible to have temporary stuff in 
that module as well.

The reason why from_pyfile is interesting and recommended is that this 
file does not have to be on the load path.  So you could have the code 
sitting in /var/www/app/env/lib/python2.5/site-packages and the config 
in /var/www/app/appconfig.cfg or something.

Regards,
Armin