librelist archives

Geonode Deployment and Pinax Integration

From:

Ariel Nunez

Date:

2010-06-09 @ 15:57

In order to get the deployment story right, we need to differentiate
between two different roles that GeoNode is assuming now, the first is
to provide a set of tools for people who need to create SDI's, for
example GeoNode used as a library to create the Capra site and the
Haiti portals; and the second role is to provide a reference
implementation of a SDI, we will call this a "skeleton project".

The combination of those two things on a single source tree makes it
harder to create and maintain projects based on GeoNode and it forces
users to either try to "extend" the GeoNode project (like the
src/capra one is doing) or just rewrite the settings.py and urls.py
from scratch (like the Haiti portals are doing).

For both of these alternatives, keeping in sync with GeoNode
development is painful and error prone, we should instead encourage
people to "use" GeoNode instead of extending it or rewriting it.
The solution to this is to make GeoNode behave like an application (in
the Django sense) and not like a project, that means not settings.py
for GeoNode and more documentation about how to integrate, as well as
having all the GeoNode specific endpoints (url paths) grouped.

Once GeoNode is an app, we should be able to create a skeleton project
using the Pinax framework and then benefitting from the whole range of
application that Pinax has already integrated and also it would mean
reaching out to the bigger Django community building out Social
Networks and Intranets to make them more aware of GIS and SDIs.

The two steps outlined above, are detailed in the following
paragraphs, in all the cases, the idea is to follow an incremental
approach without breaking the build. But how do we make it easier to
verify the build is not broken? Let's write a basic test framework.

=== 0. Test Framework ===
    a. For a predefined set of GeoNode specific urls, make sure none
gives a 404. (1 day)
    b. Javascript errors check. Since GeoNode uses Javascript a lot
and we are going to change the way media is included, we need sanity
tests to make sure at least there are no Javascript errors using a
command line tool. (1 day)

=== 1. Make geonode behave like an app and not like a project ===
Makes creation of new GeoNode enabled projects cleaner and easier too,
as well as updating the version of geonode in a given project during
development. Here are the substeps:
    a. Disposal of geonode/settings.py. (Move them all to
capra/settings, group together the ones we created and delete the
unused ones.) (1 days)
    b. Getting all of the capra specific things out of the GeoNode
source tree. (1 days)
    c. Fixing the static media handling: Use existing apps to serve
media, compress css and javascript to avoid duplication and avoid the
current high load times because of the number of files in deployment.
Includes hooking up jstools with django-compress. (2 days)
    d. Modify geonode/urls.py so it is possible to include the
following line in the project urls.py:
        (r"^geonode/", include("geonode.urls")),
        Or hook it up to the root of the server path if we want.  (1 day)
    e. Documenting the proper way to include the GeoNode app in a
project, including the needed settings (i.e. GEOSERVER_URL), the way
to run a jetty instance with just GeoServer/GeoNetwork, GeoNetwork
setup, etc. (1 day)

=== 2. Integration with Pinax ===

This will let us use the built-in Profile, Accounts (Sign up, Password
Reset, TimeZones, Per user language settings), Groups (optional),
Static media handling and Notifications.

    a. Integrate Pinax templates and GeoNode templates (2 days)
    b. Create the sample project skeleton -loosely based on the Capra
project and using Pinax theme- (2 days), after that is done we should
be able to create a new geonode like this:

    .. code-block::bash
       mkvirtualenv mysite-env
       pip install Pinax
       pinax-admin setup_project -b geonode mysite
       cd mysite
       ./geoserver/startup.sh #probably just calling jetty
       ./manage.py runserver
   c. Create the geonode_profile app based on the specs and use it to
replace the basic_profiles up bundled up with Pinax. (1 day)

As you may suppose, the "tongue-in-cheek" estimations presented in the
document actually mean between 3 and 4 weeks of development time.

If I get positive feedback, I'll start right away on the step 0 to try
and have it done before leaving New York, possibly borrowing some time
from someone with experience in testing Javascript with command line
tools.

Regards,
Ariel.

>
> Once GeoNode is an app, we should be able to create a skeleton project
> using the Pinax framework and then benefitting from the whole range of
> application that Pinax has already integrated and also it would mean
> reaching out to the bigger Django community building out Social
> Networks and Intranets to make them more aware of GIS and SDIs.
>

Generally, I'm +1 this direction in the long run.

Could it be broken into more than one application?  I wonder if
functionality could be split in ways that would encourage more efficient
community reuse.


> === 0. Test Framework ===
>    a. For a predefined set of GeoNode specific urls, make sure none
> gives a 404. (1 day)
>    b. Javascript errors check. Since GeoNode uses Javascript a lot
> and we are going to change the way media is included, we need sanity
> tests to make sure at least there are no Javascript errors using a
> command line tool. (1 day).


This sounds like a good thing for the project to me.

But I see Dwins' comment.  Info on this?  Introducing a new framework into
the build is kind of a big deal.

I know we've looked into different ones in the past; it's worth sharing the
knowledge.

As you may suppose, the "tongue-in-cheek" estimations presented in the
> document actually mean between 3 and 4 weeks of development time.
>
> If I get positive feedback, I'll start right away on the step 0 to try
> and have it done before leaving New York, possibly borrowing some time
> from someone with experience in testing Javascript with command line
> tools.
>

So, though I just said "cool" in response to your talking about working on
this immediately in IRC, after reading this thread and thinking it over I
think it would be a good idea to get on the same page about this some more
before rushing forward on it.

I think it's all great ideas but we've had problems in the past with
sweeping architecture changes without discussion among the developers,
because different people have different levels of understanding of the huge
and varied stack.

The other issue is that while this is all relevant to the 1.0 release which
is our top priority right now, it's not strictly necessary for it.

Unfortunately, this project has had to do a lot of things _wrong_ in order
to meet deadlines over the course of its development.  A lot of the
weirdness of the current architecture is due to that (we can talk some time
about the all-static-HTML-and-JavaScript version of the 'web app' that was
our Round 1).

I look forward to our fixing these things as much as anybody but what's more
important right now is being feature-complete for 1.0, for which a lot of
tickets already exist.

As decisions fall out of this discussion, we'll make more tickets and assign
them priority as appropriate.  Personally, I hope we can make time for a lot
of these changes in time for our 1.0 release, but I'm not in a position to
judge until there is weighing in from the perspective of JavaScript
development, building alongside the Java stack, etc.


-- 
Sebastian Benthall
OpenGeo - http://opengeo.org

On 06/09/2010 11:57 AM, Ariel Nunez wrote:
> === 0. Test Framework ===
>      a. For a predefined set of GeoNode specific urls, make sure none
> gives a 404. (1 day)
>      b. Javascript errors check. Since GeoNode uses Javascript a lot
> and we are going to change the way media is included, we need sanity
> tests to make sure at least there are no Javascript errors using a
> command line tool. (1 day)
>    
A little more detail here would be nice; what tools do you intend to use 
for these tests?
> === 1. Make geonode behave like an app and not like a project ===
> Makes creation of new GeoNode enabled projects cleaner and easier too,
> as well as updating the version of geonode in a given project during
> development. Here are the substeps:
>      a. Disposal of geonode/settings.py. (Move them all to
> capra/settings, group together the ones we created and delete the
> unused ones.) (1 days)
>      b. Getting all of the capra specific things out of the GeoNode
> source tree. (1 days)
>      c. Fixing the static media handling: Use existing apps to serve
> media, compress css and javascript to avoid duplication and avoid the
> current high load times because of the number of files in deployment.
> Includes hooking up jstools with django-compress. (2 days)
>      d. Modify geonode/urls.py so it is possible to include the
> following line in the project urls.py:
>          (r"^geonode/", include("geonode.urls")),
>          Or hook it up to the root of the server path if we want.  (1 day)
>      e. Documenting the proper way to include the GeoNode app in a
> project, including the needed settings (i.e. GEOSERVER_URL), the way
> to run a jetty instance with just GeoServer/GeoNetwork, GeoNetwork
> setup, etc. (1 day)
>    
I have some docs about the Django apps that I've been sitting on for a 
while; they are part of a larger developer manual that is not very 
correct right now.  The Django stuff is pretty recent though, I'm fairly 
confident it's all correct.

All for moving the CAPRA site out of the main repository.

As mentioned earlier, I don't think we should mess with the JavaScript 
too much until Andreas gets back from his vacation and finishes the 
refactoring currently under way.

I would also like some more details about the changes you intend to 
make; how do they affect packaging? Are they consistent with the way we 
distribute things (as I understand it, there are licensing concerns such 
that Ext/GeoExt/OpenLayers need to be minified separately)?  Would they 
result in the site using a different minified Ext for every page?
> === 2. Integration with Pinax ===
>
> This will let us use the built-in Profile, Accounts (Sign up, Password
> Reset, TimeZones, Per user language settings), Groups (optional),
> Static media handling and Notifications.
>
>      a. Integrate Pinax templates and GeoNode templates (2 days)
>      b. Create the sample project skeleton -loosely based on the Capra
> project and using Pinax theme- (2 days), after that is done we should
> be able to create a new geonode like this:
>
>      .. code-block::bash
>         mkvirtualenv mysite-env
>         pip install Pinax
>         pinax-admin setup_project -b geonode mysite
>         cd mysite
>         ./geoserver/startup.sh #probably just calling jetty
>         ./manage.py runserver
>     c. Create the geonode_profile app based on the specs and use it to
> replace the basic_profiles up bundled up with Pinax. (1 day)
>
>    
What's involved in adding a project profile to Pinax? Are we going to 
have to convince the Pinax guys that SDI is a useful profile for their 
social networking toolkit (might be a hard case to make)?
> As you may suppose, the "tongue-in-cheek" estimations presented in the
> document actually mean between 3 and 4 weeks of development time.
>
> If I get positive feedback, I'll start right away on the step 0 to try
> and have it done before leaving New York, possibly borrowing some time
> from someone with experience in testing Javascript with command line
> tools.
>
> Regards,
> Ariel.
>    
I'm not against these changes, but I'd like to see a bit more info about 
what's going on.

-d

On Wed, Jun 9, 2010 at 12:25 PM, David Winslow <dwinslow@opengeo.org> wrote:
>> === 0. Test Framework ===
> A little more detail here would be nice; what tools do you intend to use for
> these tests?

a. For the 404 tests I plan to use Django's testing framework, for example:

>>> c = Client()
>>> c.get('/customers/details/', {'name': 'fred', 'age': 7})

More info here[1]

[1] http://docs.djangoproject.com/en/dev/topics/testing/

b. For the javascript tests, I have no idea. I know I could test the
minified files with JSLint[2], but I don't know if it's even possible
to programmatically see if there are any Javascript errors after
rendering the page, for example the ones that Firebug show at the
bottom or the ones that IE explorer shows with the alert popup. If it
isn't possible then we would just skip this part.

[2] http://github.com/FND/jslint-cli

> I have some docs about the Django apps that I've been sitting on for a
> while; they are part of a larger developer manual that is not very correct
> right now.  The Django stuff is pretty recent though, I'm fairly confident
> it's all correct.

Great!

> All for moving the CAPRA site out of the main repository.

Cool.

> As mentioned earlier, I don't think we should mess with the JavaScript too
> much until Andreas gets back from his vacation and finishes the refactoring
> currently under way.

Fair enough, as far as I know the Javascript part is currently broken
in GeoNode right now so that's another reason to just leave any
changes that affect it for later. I agree with you.



> What's involved in adding a project profile to Pinax? Are we going to have
> to convince the Pinax guys that SDI is a useful profile for their social
> networking toolkit (might be a hard case to make)?

The case has already been made, there have been talks of a "geopinax"
for a lot of time and even some integration of "foursquare type"
applications. They are very inclusive and I know already a lot of the
folks in the community. In the remote case that they don't like our
geonode profile, we can just create a geonode-admin.py script that
mimics or extends pinax-admin.py, just as pinax-admin.py extends
django-admin.py.


> I'm not against these changes, but I'd like to see a bit more info about
> what's going on.

I'll add this information to Trac, expand it and seek some feedback
from Sebastian too.

Thanks for responding so quickly,

Ariel.

> a. For the 404 tests I plan to use Django's testing framework, for example:
>
> >>> c = Client()
> >>> c.get('/customers/details/', {'name': 'fred', 'age': 7})
>
> More info here[1]
>
> [1] http://docs.djangoproject.com/en/dev/topics/testing/
>
>
Oh, sorry, missed this earlier.  I see now.  Still -- Dwins, using this for
testing OK by you?


> > What's involved in adding a project profile to Pinax? Are we going to
> have
> > to convince the Pinax guys that SDI is a useful profile for their social
> > networking toolkit (might be a hard case to make)?
>
> The case has already been made, there have been talks of a "geopinax"
> for a lot of time and even some integration of "foursquare type"
> applications. They are very inclusive and I know already a lot of the
> folks in the community. In the remote case that they don't like our
> geonode profile, we can just create a geonode-admin.py script that
> mimics or extends pinax-admin.py, just as pinax-admin.py extends
> django-admin.py.
>

Are there Pinax community lists or IRC communities we should be getting
familiar with if we are going to start working with them more closely?

-- 
Sebastian Benthall
OpenGeo - http://opengeo.org

> Are there Pinax community lists or IRC communities we should be getting
> familiar with if we are going to start working with them more closely?

There are two channels in IRC: #pinax and #pinax-dev
Similarly, there are two mailing lists:
http://groups.google.com/group/pinax-users
http://groups.google.com/group/pinax-core-dev

brosner and jezdez are two core developers that are very friendly in
IRC (think aaime and dwins in #geoserver)

Ariel

Sweet.  Would we be pinax core developers if geonode apps were integrated in
pinax?  Or would we still be just users?

On Wed, Jun 9, 2010 at 5:18 PM, Ariel Nunez <ingenieroariel@gmail.com>wrote:

> > Are there Pinax community lists or IRC communities we should be getting
> > familiar with if we are going to start working with them more closely?
>
> There are two channels in IRC: #pinax and #pinax-dev
> Similarly, there are two mailing lists:
> http://groups.google.com/group/pinax-users
> http://groups.google.com/group/pinax-core-dev
>
> brosner and jezdez are two core developers that are very friendly in
> IRC (think aaime and dwins in #geoserver)
>
> Ariel
>



-- 
Sebastian Benthall
OpenGeo - http://opengeo.org

On Wed, Jun 9, 2010 at 5:27 PM, Sebastian Benthall <seb@opengeo.org> wrote:
> Sweet.  Would we be pinax core developers if geonode apps were integrated in
> pinax?  Or would we still be just users?

Since they use a git workflow I think we would maintain our version at
http://github.com/geonode/pinax and pull from upstream every now and
then. If they accept our project in the list, then they would pull
from our repo every now and then.

But to answer your question, we would be Pinax developers, not just users.

Ariel.

On 06/09/2010 05:10 PM, Sebastian Benthall wrote:
>
>     a. For the 404 tests I plan to use Django's testing framework, for
>     example:
>
>     >>> c = Client()
>     >>> c.get('/customers/details/', {'name': 'fred', 'age': 7})
>
>     More info here[1]
>
>     [1] http://docs.djangoproject.com/en/dev/topics/testing/
>
>
> Oh, sorry, missed this earlier.  I see now.  Still -- Dwins, using 
> this for testing OK by you?
We're already using the Django test suite (sort of; the tests are not 
very well maintained.)  I am more interested in figuring how the 
JavaScript tests will be set up; particularly given that testing the 
scripts in the site will probably require standing up a GeoServer 
alongside of Django.

We've basically had this discussion before and the JS gurus from OpenGeo 
have opined that manual testing will be the same or less work as 
maintaining an automated test suite, so I would want to get Andreas' 
input on a JS test environment before implementing it.
>
>     > What's involved in adding a project profile to Pinax? Are we
>     going to have
>     > to convince the Pinax guys that SDI is a useful profile for
>     their social
>     > networking toolkit (might be a hard case to make)?
>
>     The case has already been made, there have been talks of a "geopinax"
>     for a lot of time and even some integration of "foursquare type"
>     applications. They are very inclusive and I know already a lot of the
>     folks in the community. In the remote case that they don't like our
>     geonode profile, we can just create a geonode-admin.py script that
>     mimics or extends pinax-admin.py, just as pinax-admin.py extends
>     django-admin.py.
>
>
> Are there Pinax community lists or IRC communities we should be 
> getting familiar with if we are going to start working with them more 
> closely?

Good question.

>
>  <http://docs.djangoproject.com/en/dev/topics/testing/>
>
>  We're already using the Django test suite (sort of; the tests are not very
> well maintained.)
>

Oh, ok, my bad.

In that case, Ariel: yeah, definitely write whatever tests you think would
be helpful!  This probably should be common practice for the project, but we
have been on such tight timelines that we've cut these corners.

But yeah, tests are great.  Yay tests!


>   I am more interested in figuring how the JavaScript tests will be set up;
> particularly given that testing the scripts in the site will probably
> require standing up a GeoServer alongside of Django.
>
> We've basically had this discussion before and the JS gurus from OpenGeo
> have opined that manual testing will be the same or less work as maintaining
> an automated test suite, so I would want to get Andreas' input on a JS test
> environment before implementing it.
>
>
>
>> > What's involved in adding a project profile to Pinax? Are we going to
>> have
>> > to convince the Pinax guys that SDI is a useful profile for their social
>> > networking toolkit (might be a hard case to make)?
>>
>>  The case has already been made, there have been talks of a "geopinax"
>> for a lot of time and even some integration of "foursquare type"
>> applications. They are very inclusive and I know already a lot of the
>> folks in the community. In the remote case that they don't like our
>> geonode profile, we can just create a geonode-admin.py script that
>> mimics or extends pinax-admin.py, just as pinax-admin.py extends
>> django-admin.py.
>>
>
> Are there Pinax community lists or IRC communities we should be getting
> familiar with if we are going to start working with them more closely?
>
>
> Good question.
>



-- 
Sebastian Benthall
OpenGeo - http://opengeo.org

Given David's last comment about Javascript testing I think I will
just forget the matter for now.

Regarding the Django HTTP Client tests, I am adding a few sanity tests
to check the status code of the Homepage/maps/data/admin pages with
and without being signed in.