librelist archives

The Rakefile Revolution

From:

Devyn Cairns

Date:

2010-08-10 @ 04:17

Alright, so I hope all of us on the development team know that the Rakefile
*really* needs a lot of work for WalkAbout.
In fact, the entire build process could be redone.

Splitting the Rakefile into multiple files is good temporarily, but it's
really not the final solution.
I really appreciate your work on this ashbb, but…

I have come to the conclusion that if we really want to make Shoes easy to
compile and more clear,
we're going to have to do a complete rewrite of the Rakefile.

Now, one of my biggest complaints with the current Rakefile is clarity.
There's so much noise!
Only a computer could hope to truly understand that mess.
I think we could achieve better clarity by splitting the entire build
process into multiple files,
and making it all more object oriented. The current Rakefile uses far too
many constants for my liking.

It's just so icky to edit, and getting a real understanding for it is nearly
impossible.

By revolutionizing the build process, we could make it far easier to
install.

-- 
    ~devyn

> I have come to the conclusion that if we really want to make Shoes easy to
> compile and more clear,
> we're going to have to do a complete rewrite of the Rakefile.

An incremental refactoring would get us there quicker, I think. I've
taken ashbb's refactored rakefiles and done some preliminary hacking
along the following plan:

1. Separate the rakefiles into environment variables, task helpers and tasks
2. Tuck the messy details into the task helpers, and let the rakefiles
be clear and self-documenting
3. Chip away at the messy bits, trying to get consistent logical
methods in all three setups
4. Create builder classes with the same high-level methods
5. Recombine the separate rakefiles into a single rakefile that
instantiates a builder object and then dispatches polymorphically to
various task helpers

Take a look at http://github.com/martindemello/shoes/tree/rakefile -
if people think this is headed in the right direction I'll continue
hacking at it. I don't think we can really get away from the heap of
constants and shell calls, but we can build a pearl around them :)

martin

On Tue, Aug 10, 2010 at 11:28 PM, Martin DeMello
<martindemello@gmail.com> wrote:
> 1. Separate the rakefiles into environment variables, task helpers and tasks
> 2. Tuck the messy details into the task helpers, and let the rakefiles
> be clear and self-documenting
> 3. Chip away at the messy bits, trying to get consistent logical
> methods in all three setups
> 4. Create builder classes with the same high-level methods
> 5. Recombine the separate rakefiles into a single rakefile that
> instantiates a builder object and then dispatches polymorphically to
> various task helpers

Okay, I have most of this done.

 http://github.com/martindemello/shoes/tree/rakefile

There's still some cleaning up that can be done, but most of the mess
has been swept under various carpets, the main rakefile looks nice and
clean with easily understandable tasks, and the platform rakefiles are
mostly empty. It works on linux; could people test it on darwin and
mingw and let me know if anything broke? (In particular, mingw has
some tricky bits going on with monkeypatching `sh` that gave me pause;
it should all just work but a test would be reassuring)

martin

Hi Martin,

Awesome!!
Fetched and merged your code into my latest repo on github. And added a
trivial commit, then it works fine!

http://github.com/ashbb/shoes/commit/b0e8e3303716182815a8a426fe96c92b2d6c6201

I could build Policeman for Windows both with and without Video support.

After trying to build Policeman for Linux (on Windows VirtualBox though)
today, I'll send a pull request to shoes/shoes (official Shoes repo). Now,
we got a fantastic Rakefiles. B-)

Thanks,
ashbb

Great! Pull once more before you merge into shoes/shoes - I've made
some minor fixes in make/darwin too.

martin

On Wed, Aug 11, 2010 at 12:05 PM, ashbb <ashbbb@gmail.com> wrote:
> Hi Martin,
>
> Awesome!!
> Fetched and merged your code into my latest repo on github. And added a
> trivial commit, then it works fine!
>
> http://github.com/ashbb/shoes/commit/b0e8e3303716182815a8a426fe96c92b2d6c6201
>
> I could build Policeman for Windows both with and without Video support.
>
> After trying to build Policeman for Linux (on Windows VirtualBox though)
> today, I'll send a pull request to shoes/shoes (official Shoes repo). Now,
> we got a fantastic Rakefiles. B-)
>
> Thanks,
> ashbb
>

Hi Martin,
I've just finished to build Policeman for Linux with your Rakefiles. It
works perfect. There was nothing to change.
Awesome!

> Pull once more before you merge into shoes/shoes
> - I've made some minor fixes in make/darwin too.
Okay, when it's available, let me know. ;-)

Cheers,
ashbb

http://github.com/martindemello/shoes/commit/1fc1f1a857848d963864f3cd01df0b4ee6589e5c

martin

On Wed, Aug 11, 2010 at 12:48 PM, ashbb <ashbbb@gmail.com> wrote:
> Hi Martin,
> I've just finished to build Policeman for Linux with your Rakefiles. It
> works perfect. There was nothing to change.
> Awesome!
>
>> Pull once more before you merge into shoes/shoes
>> - I've made some minor fixes in make/darwin too.
> Okay, when it's available, let me know. ;-)
>
> Cheers,
> ashbb
>

Done!
(merged and sent a pull request) :)

ashbb

I will take care of these pull requests this evening.

Awesome job, guys! (I still have yet to test it on Snow Leopard, too.

On Wed, Aug 11, 2010 at 4:37 AM, ashbb <ashbbb@gmail.com> wrote:
> Done!
> (merged and sent a pull request) :)
>
> ashbb
>

On Tue, Aug 10, 2010 at 4:04 PM, Martin DeMello <martindemello@gmail.com>wrote:

> On Tue, Aug 10, 2010 at 11:28 PM, Martin DeMello
> <martindemello@gmail.com> wrote:
> > 1. Separate the rakefiles into environment variables, task helpers and
> tasks
> > 2. Tuck the messy details into the task helpers, and let the rakefiles
> > be clear and self-documenting
> > 3. Chip away at the messy bits, trying to get consistent logical
> > methods in all three setups
> > 4. Create builder classes with the same high-level methods
> > 5. Recombine the separate rakefiles into a single rakefile that
> > instantiates a builder object and then dispatches polymorphically to
> > various task helpers
>
> Okay, I have most of this done.
>
>  http://github.com/martindemello/shoes/tree/rakefile
>
> There's still some cleaning up that can be done, but most of the mess
> has been swept under various carpets, the main rakefile looks nice and
> clean with easily understandable tasks, and the platform rakefiles are
> mostly empty. It works on linux; could people test it on darwin and
> mingw and let me know if anything broke? (In particular, mingw has
> some tricky bits going on with monkeypatching `sh` that gave me pause;
> it should all just work but a test would be reassuring)
>

Error on OS X/darwin:

(in /Users/devyn/Projects/shoes)
rake aborted!
uninitialized constant VLC_CFLAGS
/Users/devyn/Projects/shoes/Rakefile:49:in `require'
(See full trace by running task with --trace)

Your work is great! Could we DRY up make/darwin/env.rb line 50 a little bit?
It's kinda huge :P I think a builder would help.


>
> martin
>



-- 
    ~devyn

On Wed, Aug 11, 2010 at 9:49 AM, Devyn Cairns <devyn.cairns@gmail.com> wrote:
>
> Error on OS X/darwin:
> (in /Users/devyn/Projects/shoes)
> rake aborted!
> uninitialized constant VLC_CFLAGS
> /Users/devyn/Projects/shoes/Rakefile:49:in `require'
> (See full trace by running task with --trace)

Okay, VLC_CFLAGS was missing from the darwin codepath even in the
original rakefile, so i wasn't sure what it needed to be. I've copied
in the linux definition so far; could you take a look at it and
substitute the right value?

> Your work is great! Could we DRY up make/darwin/env.rb line 50 a little bit?
> It's kinda huge :P I think a builder would help.

Do you mean line 58 (LINUX_LDFLAGS)? Okay, I've split off the vlc deps
into a textfile the way I did with the dylibs. I wasn't sure if I
could refactor the "-framework foo" entries because they seem to be
interspersed with the -lib entries in an order dependent manner.

martin

On Wed, Aug 11, 2010 at 2:12 AM, Martin DeMello <martindemello@gmail.com>wrote:

> On Wed, Aug 11, 2010 at 9:49 AM, Devyn Cairns <devyn.cairns@gmail.com>
> wrote:
> >
> > Error on OS X/darwin:
> > (in /Users/devyn/Projects/shoes)
> > rake aborted!
> > uninitialized constant VLC_CFLAGS
> > /Users/devyn/Projects/shoes/Rakefile:49:in `require'
> > (See full trace by running task with --trace)
>
> Okay, VLC_CFLAGS was missing from the darwin codepath even in the
> original rakefile, so i wasn't sure what it needed to be. I've copied
> in the linux definition so far; could you take a look at it and
> substitute the right value?
>

I'd love to, but I really have some work I need to get done first. Perhaps
Steve can do something? ;]


>
> > Your work is great! Could we DRY up make/darwin/env.rb line 50 a little
> bit?
> > It's kinda huge :P I think a builder would help.
>
> Do you mean line 58 (LINUX_LDFLAGS)?


Probably. My bad :P


> Okay, I've split off the vlc deps
> into a textfile the way I did with the dylibs. I wasn't sure if I
> could refactor the "-framework foo" entries because they seem to be
> interspersed with the -lib entries in an order dependent manner.
>

Could you do something like {frameworks: […], headers: […], libraries: […],
other_options: […]} ? It might help to clean things up—just compile it all
into command line arguments. You don't have to do it exactly this way, just
a suggestion.


>
> martin
>



-- 
    ~devyn

On Thu, Aug 12, 2010 at 8:46 AM, Devyn Cairns <devyn.cairns@gmail.com> wrote:
>
> Could you do something like {frameworks: […], headers: […], libraries: […],
> other_options: […]} ? It might help to clean things up—just compile it all
> into command line arguments. You don't have to do it exactly this way, just
> a suggestion.

No, what my problem is is that the code currently looks like this:

LINUX_LDFLAGS << " -framework vecLib -lpthread -lm -liconv -lintl
-liconv -lc -lpostproc -        lavformat -lavcodec -lz -la52 -lfaac
-lfaad -lmp3lame -lx264 -lxvidcore -lvorbisenc -lavutil -      lvorbis
-lm -logg -lm -lavformat -lavcodec -lz -la52 -lfaac -lfaad -lmp3lame
-lx264 -lxvidcore -    lvorbisenc -lavutil -lvorbis -lm -logg
-framework QuickTime -lm -framework CoreAudio -framework     AudioUnit
-framework AudioToolbox -framework IOKit -lobjc -ObjC -framework
OpenGL -framework AGL -  read_only_relocs suppress"

Now, is "-framework foo -lbar -framework baz -lquux" the same as
"-framework foo -framework baz -lbar -lquux"? If it isn't, then
there's very little real improvement we can get, since we have to
preserve the exact global order of all the arguments.

martin

On Wed, 2010-08-11 at 04:34 +0530, Martin DeMello wrote:
> On Tue, Aug 10, 2010 at 11:28 PM, Martin DeMello

>  http://github.com/martindemello/shoes/tree/rakefile
> 
> There's still some cleaning up that can be done, but most of the mess
> has been swept under various carpets, the main rakefile looks nice and
> clean with easily understandable tasks, and the platform rakefiles are
> mostly empty. It works on linux; could people test it on darwin and
> mingw and let me know if anything broke? (In particular, mingw has
> some tricky bits going on with monkeypatching `sh` that gave me pause;
> it should all just work but a test would be reassuring)
> 
> martin

I like direction of your clean up. I haven't tested it however because
there are some nit picky bugs to clean up before the imminent release
first.  I can confirm that curl is not part of default Ubuntu 10.4 so
when a packaged .run script runs on a virgin Ubuntu, it will fail for
the lack of curl or wget. Someone should file an issue about not copying
the curl libraries into the Shoes distribution. 

I say go ahead. Maybe a few helper files too?

On Tue, Aug 10, 2010 at 10:58 AM, Martin DeMello <martindemello@gmail.com>wrote:

> > I have come to the conclusion that if we really want to make Shoes easy
> to
> > compile and more clear,
> > we're going to have to do a complete rewrite of the Rakefile.
>
> An incremental refactoring would get us there quicker, I think. I've
> taken ashbb's refactored rakefiles and done some preliminary hacking
> along the following plan:
>
> 1. Separate the rakefiles into environment variables, task helpers and
> tasks
> 2. Tuck the messy details into the task helpers, and let the rakefiles
> be clear and self-documenting
> 3. Chip away at the messy bits, trying to get consistent logical
> methods in all three setups
> 4. Create builder classes with the same high-level methods
> 5. Recombine the separate rakefiles into a single rakefile that
> instantiates a builder object and then dispatches polymorphically to
> various task helpers
>
> Take a look at http://github.com/martindemello/shoes/tree/rakefile -
> if people think this is headed in the right direction I'll continue
> hacking at it. I don't think we can really get away from the heap of
> constants and shell calls, but we can build a pearl around them :)
>
> martin
>



-- 
    ~devyn

Yeah, though there seems to be very little that's actually common to
all three platforms

martin

On Wed, Aug 11, 2010 at 2:10 AM, Devyn Cairns <devyn.cairns@gmail.com> wrote:
> I say go ahead. Maybe a few helper files too?
>
> On Tue, Aug 10, 2010 at 10:58 AM, Martin DeMello <martindemello@gmail.com>
> wrote:
>>
>> > I have come to the conclusion that if we really want to make Shoes easy
>> > to
>> > compile and more clear,
>> > we're going to have to do a complete rewrite of the Rakefile.
>>
>> An incremental refactoring would get us there quicker, I think. I've
>> taken ashbb's refactored rakefiles and done some preliminary hacking
>> along the following plan:
>>
>> 1. Separate the rakefiles into environment variables, task helpers and
>> tasks
>> 2. Tuck the messy details into the task helpers, and let the rakefiles
>> be clear and self-documenting
>> 3. Chip away at the messy bits, trying to get consistent logical
>> methods in all three setups
>> 4. Create builder classes with the same high-level methods
>> 5. Recombine the separate rakefiles into a single rakefile that
>> instantiates a builder object and then dispatches polymorphically to
>> various task helpers
>>
>> Take a look at http://github.com/martindemello/shoes/tree/rakefile -
>> if people think this is headed in the right direction I'll continue
>> hacking at it. I don't think we can really get away from the heap of
>> constants and shell calls, but we can build a pearl around them :)
>>
>> martin
>
>
>
> --
>     ~devyn
>

Just tried building shoes on ubuntu, and it failed twice along the way
- once for want of curl-config and once for want of libgif-dev. How
about an addition to the rakefile that does dependency checking up
front, so we can return one clean set of error messages for all the
missing dependencies?

martin

On Tue, Aug 10, 2010 at 9:47 AM, Devyn Cairns <devyn.cairns@gmail.com> wrote:
> Alright, so I hope all of us on the development team know that the Rakefile
> really needs a lot of work for WalkAbout.
> In fact, the entire build process could be redone.
> Splitting the Rakefile into multiple files is good temporarily, but it's
> really not the final solution.
> I really appreciate your work on this ashbb, but…
> I have come to the conclusion that if we really want to make Shoes easy to
> compile and more clear,
> we're going to have to do a complete rewrite of the Rakefile.
> Now, one of my biggest complaints with the current Rakefile is clarity.
> There's so much noise!
> Only a computer could hope to truly understand that mess.
> I think we could achieve better clarity by splitting the entire build
> process into multiple files,
> and making it all more object oriented. The current Rakefile uses far too
> many constants for my liking.
> It's just so icky to edit, and getting a real understanding for it is nearly
> impossible.
> By revolutionizing the build process, we could make it far easier to
> install.
> --
>     ~devyn
>

On Mon, 2010-08-09 at 21:17 -0700, Devyn Cairns wrote:
> Alright, so I hope all of us on the development team know that the
> Rakefile really needs a lot of work for WalkAbout.
> In fact, the entire build process could be redone.
> 
> 
> Splitting the Rakefile into multiple files is good temporarily, but
> it's really not the final solution.
> I really appreciate your work on this ashbb, but…
> 
> 
> I have come to the conclusion that if we really want to make Shoes
> easy to compile and more clear,
> we're going to have to do a complete rewrite of the Rakefile.
> 
> 
> Now, one of my biggest complaints with the current Rakefile is
> clarity. There's so much noise!
> Only a computer could hope to truly understand that mess.
> I think we could achieve better clarity by splitting the entire build
> process into multiple files,
> and making it all more object oriented. The current Rakefile uses far
> too many constants for my liking.
> 
> 
> It's just so icky to edit, and getting a real understanding for it is
> nearly impossible.
> 
> 
> By revolutionizing the build process, we could make it far easier to
> install.

Rake is messy because the problem is messy and rake is a DSL/functional
language, not an imperative language. The X11 mkmf scheme (used in Shoes
in some places) is OO without the syntactic sugar of OO. 

What new paradigm would you propose? Prolog? 
> 
> -- 
>     ~devyn
> 

On Tue, Aug 10, 2010 at 11:02 AM, Cecil Coupe <ccoupe@cableone.net> wrote:
>
> Rake is messy because the problem is messy and rake is a DSL/functional
> language, not an imperative language. The X11 mkmf scheme (used in Shoes
> in some places) is OO without the syntactic sugar of OO.

That doesn't mean that the *actions* called in each of the rake tasks
can't be expressed in terms of clean, object-oriented ruby, with the
actual worker methods split off into their own files.

martin

On Tue, 2010-08-10 at 11:05 +0530, Martin DeMello wrote:
> On Tue, Aug 10, 2010 at 11:02 AM, Cecil Coupe <ccoupe@cableone.net> wrote:
> >
> > Rake is messy because the problem is messy and rake is a DSL/functional
> > language, not an imperative language. The X11 mkmf scheme (used in Shoes
> > in some places) is OO without the syntactic sugar of OO.
> 
> That doesn't mean that the *actions* called in each of the rake tasks
> can't be expressed in terms of clean, object-oriented ruby, with the
> actual worker methods split off into their own files.
> 
> martin

And that differs from xmkmf (or rake) in some important manner? Little
fragments of "make" code stored with the directory containing the code.
Actions and variables to act upon, specialized  (subclassed). File
systems are hierarchical. That's the messy part. It's the fact that we
have to worked with.  Relational DB to OO mapping is messy. Neither
really map physical hierarchy well. There's plenty of room in rake to do
something OO PC pure. I have't seen it yet, but I'm open to anything
that works.

On Mon, Aug 9, 2010 at 11:52 PM, Cecil Coupe <ccoupe@cableone.net> wrote:

> On Tue, 2010-08-10 at 11:05 +0530, Martin DeMello wrote:
> > On Tue, Aug 10, 2010 at 11:02 AM, Cecil Coupe <ccoupe@cableone.net>
> wrote:
> > >
> > > Rake is messy because the problem is messy and rake is a DSL/functional
> > > language, not an imperative language. The X11 mkmf scheme (used in
> Shoes
> > > in some places) is OO without the syntactic sugar of OO.
> >
> > That doesn't mean that the *actions* called in each of the rake tasks
> > can't be expressed in terms of clean, object-oriented ruby, with the
> > actual worker methods split off into their own files.
> >
> > martin
>
> And that differs from xmkmf (or rake) in some important manner? Little
> fragments of "make" code stored with the directory containing the code.
> Actions and variables to act upon, specialized  (subclassed). File
> systems are hierarchical. That's the messy part. It's the fact that we
> have to worked with.  Relational DB to OO mapping is messy.


Just wanted to point out something here: yes, it is. But you *can* map
classes to a filesystem quite easily.


> Neither
> really map physical hierarchy well. There's plenty of room in rake to do
> something OO PC pure. I have't seen it yet, but I'm open to anything
> that works.
>
>


-- 
    ~devyn

That's what I was getting at.

Beside which, Cecil, the proper term in Rake's case is declarative (just
like Rspec, Sinatra, and many other DSL-based libraries), not functional ;]

On Mon, Aug 9, 2010 at 10:35 PM, Martin DeMello <martindemello@gmail.com>wrote:

> On Tue, Aug 10, 2010 at 11:02 AM, Cecil Coupe <ccoupe@cableone.net> wrote:
> >
> > Rake is messy because the problem is messy and rake is a DSL/functional
> > language, not an imperative language. The X11 mkmf scheme (used in Shoes
> > in some places) is OO without the syntactic sugar of OO.
>
> That doesn't mean that the *actions* called in each of the rake tasks
> can't be expressed in terms of clean, object-oriented ruby, with the
> actual worker methods split off into their own files.
>
> martin
>



-- 
    ~devyn