librelist archives

« back to archive

SQL database testing/migration profile

SQL database testing/migration profile

From:
Anton Khodakivskiy
Date:
2013-08-12 @ 09:35
Hi,

I'm getting started with Clojure and lein. I'm working on a small web app 
that utilizes SQL database, and I'd like to write and easily run proper 
unit tests. I use korma and ragtime for database access and migrations. 
Currently I'm trying to find proper way to configure different 
environments for database access. At the moment I need 3 profiles: dev, 
test, and prod. So somewhere I need to specify 3 different sets of 
database parameters(host, port, username, password, etc.) and have korma 
and ragtime properly read necessary set or args. What's proper way to do 
this in leiningen? 

To be more specific here is a list of questions I'm trying to answer:

1) Can this be done with lein profiles? Can I have this in project.clj: 

:profiles { 
  :dev { :db { :host … :username … } }
  :test { :db { :host … :username … } }
  :prod { :db { :host … :username … } }
} 

How do I read these values and pass them to ragtime/korma on runtime?

2) Is it better to do it with environment variables? Many apps expect 
DATABASE_URL env value to be a proper jdbc connection string. For unit 
testing and running dev server I don't want to set the env variable every 
time, how can this be automated?

3) Which is better 1 or 2?

4) When I run 'lein test' or 'lein spec' (with speclj) which profile is 
used by default? Is it :test or :dev? I obviously don't want to run unit 
tests and the local dev server in different profiles. What's the most 
robust way to separate them?

Thanks,

Anton

Re: [leiningen] SQL database testing/migration profile

From:
Colin Yates
Date:
2013-08-12 @ 10:21
I don't have all the answers, but what I do is:
 - use https://github.com/weavejester/environ to capture external
configuration
 - expect MYAPP_DB_URL etc. to be populated (and retrieved via environ)
 - use ~/.lein/profiles.clj to define the properties

So a developer machine will have {:test {:env {:myapp-db-url "xyz"}}} in
~/.lein/profiles.clj but jenkins will define MYAPP_DB_URL environment
variables. (note: environ knows how to map clojure keywords to environment
variables).

"lein test" seems to use the "test" profile but "lein midje" doesn't - it
uses the same profile as "lein ring server" which causes problems.

To work around this I have a "web" profile with a convenient "lein web!"
aliased to "lein with-profile web ring server".

Ultimately I found myself in a situation where both midje and ring used the
same profile so I had to "with-profile" one of them - we run the UI much
less frequently than we build server side logic so "lein midje" and "lein
web!" works for us (emacs and midje is a lovely place to be).

We don't deal with production from our development machines - we let
Jenkins do that for us.

Hope this helps.


On 12 August 2013 10:35, Anton Khodakivskiy <akhodakivskiy@gmail.com> wrote:

> Hi,
>
> I'm getting started with Clojure and lein. I'm working on a small web app
> that utilizes SQL database, and I'd like to write and easily run proper
> unit tests. I use korma and ragtime for database access and migrations.
> Currently I'm trying to find proper way to configure different environments
> for database access. At the moment I need 3 profiles: dev, test, and prod.
> So somewhere I need to specify 3 different sets of database
> parameters(host, port, username, password, etc.) and have korma and ragtime
> properly read necessary set or args. What's proper way to do this in
> leiningen?
>
> To be more specific here is a list of questions I'm trying to answer:
>
> 1) Can this be done with lein profiles? Can I have this in project.clj:
>
> :profiles {
>   :dev { :db { :host … :username … } }
>   :test { :db { :host … :username … } }
>   :prod { :db { :host … :username … } }
> }
>
> How do I read these values and pass them to ragtime/korma on runtime?
>
> 2) Is it better to do it with environment variables? Many apps expect
> DATABASE_URL env value to be a proper jdbc connection string. For unit
> testing and running dev server I don't want to set the env variable every
> time, how can this be automated?
>
> 3) Which is better 1 or 2?
>
> 4) When I run 'lein test' or 'lein spec' (with speclj) which profile is
> used by default? Is it :test or :dev? I obviously don't want to run unit
> tests and the local dev server in different profiles. What's the most
> robust way to separate them?
>
> Thanks,
>
> Anton
>

Re: [leiningen] SQL database testing/migration profile

From:
Phil Hagelberg
Date:
2013-08-12 @ 15:02
The other common way is to read config from the classpath, and then
change :resource-paths in various profiles. See
https://github.com/sonian/carica

Anton Khodakivskiy writes:

> Do you also use any database migration tools? Would you run them manually
> via lein (e.g. lein ragtime migrate) or trigger them from the code on
> startup?

Ragtime is decent but I agree it's a bit heavy-handed. Here's what we
use for Clojars: (just a pile of functions and a single `migrate` to run
and record them)

https://github.com/ato/clojars-web/blob/master/src/clojars/db/migrate.clj

This explicitly ignores "downward" migrations (which never work) and the
ability to abstract your migrations away from your DB so your
application can be portable to other DB systems (which also never works).

-Phil

Re: [leiningen] SQL database testing/migration profile

From:
Anton Khodakivskiy
Date:
2013-08-13 @ 10:15
Thanks for your replies! It's quite helpful. I think I will also resort to
a "pile of functions" for migrations for now.

Although I'm having an issue with environ. I have this in my
~/.lein/profiles.clj

{:user {:plugins [[lein-midje "3.0.0"] [lein-pprint "1.1.1"]]}
 :dev  {:env {:app-db-url "postgres://usr:pwd@localhost:5432/db"}}
 :test {:env {:app-db-url "postgres://usr:pwd@localhost:5432/db_test"}}}

But then (env :app-db-url) always returns nil - I've tried 'lein ring
server', 'lein with-profile dev ring server', and similar variations for
midje. Do you know what might be the problem?

It seems like lein reads ~/.lein/profiles.clj because ppring and midje
plugins are active, but environ doesn't extract relevant values...


On Mon, Aug 12, 2013 at 6:02 PM, Phil Hagelberg <phil@hagelb.org> wrote:

>
> The other common way is to read config from the classpath, and then
> change :resource-paths in various profiles. See
> https://github.com/sonian/carica
>
> Anton Khodakivskiy writes:
>
> > Do you also use any database migration tools? Would you run them manually
> > via lein (e.g. lein ragtime migrate) or trigger them from the code on
> > startup?
>
> Ragtime is decent but I agree it's a bit heavy-handed. Here's what we
> use for Clojars: (just a pile of functions and a single `migrate` to run
> and record them)
>
> https://github.com/ato/clojars-web/blob/master/src/clojars/db/migrate.clj
>
> This explicitly ignores "downward" migrations (which never work) and the
> ability to abstract your migrations away from your DB so your
> application can be portable to other DB systems (which also never works).
>
> -Phil
>

Re: [leiningen] SQL database testing/migration profile

From:
Anton Khodakivskiy
Date:
2013-08-12 @ 10:40
Colin, thanks for you answer!

Do you also use any database migration tools? Would you run them manually
via lein (e.g. lein ragtime migrate) or trigger them from the code on
startup?

Regards,

Anton


On Mon, Aug 12, 2013 at 1:21 PM, Colin Yates <colin.yates@gmail.com> wrote:

> I don't have all the answers, but what I do is:
>  - use https://github.com/weavejester/environ to capture external
> configuration
>  - expect MYAPP_DB_URL etc. to be populated (and retrieved via environ)
>  - use ~/.lein/profiles.clj to define the properties
>
> So a developer machine will have {:test {:env {:myapp-db-url "xyz"}}} in
> ~/.lein/profiles.clj but jenkins will define MYAPP_DB_URL environment
> variables. (note: environ knows how to map clojure keywords to environment
> variables).
>
> "lein test" seems to use the "test" profile but "lein midje" doesn't - it
> uses the same profile as "lein ring server" which causes problems.
>
> To work around this I have a "web" profile with a convenient "lein web!"
> aliased to "lein with-profile web ring server".
>
> Ultimately I found myself in a situation where both midje and ring used
> the same profile so I had to "with-profile" one of them - we run the UI
> much less frequently than we build server side logic so "lein midje" and
> "lein web!" works for us (emacs and midje is a lovely place to be).
>
> We don't deal with production from our development machines - we let
> Jenkins do that for us.
>
> Hope this helps.
>
>
> On 12 August 2013 10:35, Anton Khodakivskiy <akhodakivskiy@gmail.com>wrote:
>
>> Hi,
>>
>> I'm getting started with Clojure and lein. I'm working on a small web app
>> that utilizes SQL database, and I'd like to write and easily run proper
>> unit tests. I use korma and ragtime for database access and migrations.
>> Currently I'm trying to find proper way to configure different environments
>> for database access. At the moment I need 3 profiles: dev, test, and prod.
>> So somewhere I need to specify 3 different sets of database
>> parameters(host, port, username, password, etc.) and have korma and ragtime
>> properly read necessary set or args. What's proper way to do this in
>> leiningen?
>>
>> To be more specific here is a list of questions I'm trying to answer:
>>
>> 1) Can this be done with lein profiles? Can I have this in project.clj:
>>
>> :profiles {
>>   :dev { :db { :host … :username … } }
>>   :test { :db { :host … :username … } }
>>   :prod { :db { :host … :username … } }
>> }
>>
>> How do I read these values and pass them to ragtime/korma on runtime?
>>
>> 2) Is it better to do it with environment variables? Many apps expect
>> DATABASE_URL env value to be a proper jdbc connection string. For unit
>> testing and running dev server I don't want to set the env variable every
>> time, how can this be automated?
>>
>> 3) Which is better 1 or 2?
>>
>> 4) When I run 'lein test' or 'lein spec' (with speclj) which profile is
>> used by default? Is it :test or :dev? I obviously don't want to run unit
>> tests and the local dev server in different profiles. What's the most
>> robust way to separate them?
>>
>> Thanks,
>>
>> Anton
>>
>
>

Re: [leiningen] SQL database testing/migration profile

From:
Colin Yates
Date:
2013-08-12 @ 10:46
I tend to find database migration tools a bit heavy handed to be honest.
 We have built a little (unfortunately closed) library which allows
individual components to be versioned.  You could built it yourself in a
few hours I expect; the API is trivial; each component will register itself
at startup:

(register component-id current-version map-from-version-to-upgrade-function)

The library will then check what the last known version of component-id
was, calculate the diff between that version and the current-version and
then run all the functions that are keyed to those missing version up to
the current version in the map-from-version-to-upgrade-function.  Guess
what most of those functions do?  They alter the database :), but not all!.
 After running each version the library then records the fact that
component-id is at 'current-version'.  For simplicity, versions are numbers.

We are using event sourcing pretty much throughout the app which fits very
much into the "individual components" rather than "a system" so this works
well for us.


On 12 August 2013 11:40, Anton Khodakivskiy <akhodakivskiy@gmail.com> wrote:

> Colin, thanks for you answer!
>
> Do you also use any database migration tools? Would you run them manually
> via lein (e.g. lein ragtime migrate) or trigger them from the code on
> startup?
>
> Regards,
>
> Anton
>
>
> On Mon, Aug 12, 2013 at 1:21 PM, Colin Yates <colin.yates@gmail.com>wrote:
>
>> I don't have all the answers, but what I do is:
>>  - use https://github.com/weavejester/environ to capture external
>> configuration
>>  - expect MYAPP_DB_URL etc. to be populated (and retrieved via environ)
>>  - use ~/.lein/profiles.clj to define the properties
>>
>> So a developer machine will have {:test {:env {:myapp-db-url "xyz"}}} in
>> ~/.lein/profiles.clj but jenkins will define MYAPP_DB_URL environment
>> variables. (note: environ knows how to map clojure keywords to environment
>> variables).
>>
>> "lein test" seems to use the "test" profile but "lein midje" doesn't - it
>> uses the same profile as "lein ring server" which causes problems.
>>
>> To work around this I have a "web" profile with a convenient "lein web!"
>> aliased to "lein with-profile web ring server".
>>
>> Ultimately I found myself in a situation where both midje and ring used
>> the same profile so I had to "with-profile" one of them - we run the UI
>> much less frequently than we build server side logic so "lein midje" and
>> "lein web!" works for us (emacs and midje is a lovely place to be).
>>
>> We don't deal with production from our development machines - we let
>> Jenkins do that for us.
>>
>> Hope this helps.
>>
>>
>> On 12 August 2013 10:35, Anton Khodakivskiy <akhodakivskiy@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm getting started with Clojure and lein. I'm working on a small web
>>> app that utilizes SQL database, and I'd like to write and easily run proper
>>> unit tests. I use korma and ragtime for database access and migrations.
>>> Currently I'm trying to find proper way to configure different environments
>>> for database access. At the moment I need 3 profiles: dev, test, and prod.
>>> So somewhere I need to specify 3 different sets of database
>>> parameters(host, port, username, password, etc.) and have korma and ragtime
>>> properly read necessary set or args. What's proper way to do this in
>>> leiningen?
>>>
>>> To be more specific here is a list of questions I'm trying to answer:
>>>
>>> 1) Can this be done with lein profiles? Can I have this in project.clj:
>>>
>>> :profiles {
>>>   :dev { :db { :host … :username … } }
>>>   :test { :db { :host … :username … } }
>>>   :prod { :db { :host … :username … } }
>>> }
>>>
>>> How do I read these values and pass them to ragtime/korma on runtime?
>>>
>>> 2) Is it better to do it with environment variables? Many apps expect
>>> DATABASE_URL env value to be a proper jdbc connection string. For unit
>>> testing and running dev server I don't want to set the env variable every
>>> time, how can this be automated?
>>>
>>> 3) Which is better 1 or 2?
>>>
>>> 4) When I run 'lein test' or 'lein spec' (with speclj) which profile is
>>> used by default? Is it :test or :dev? I obviously don't want to run unit
>>> tests and the local dev server in different profiles. What's the most
>>> robust way to separate them?
>>>
>>> Thanks,
>>>
>>> Anton
>>>
>>
>>
>