librelist archives

Current thinking on formulas that need newer versions of Apple tools?

From:

Dave Bayer

Date:

2011-08-16 @ 05:19

Hi. I'm new to Homebrew, and enthusiastic about most of the design 
choices, though I'm wrestling with the "no dups" policy.

I have a draft of a working formula for Macaulay2, a specialized computer 
algebra system:

	http://www.macaulay2.com/
	Depends on: autoconf, gnu-tar, bdw-gc, mpfr

The authors are friends I can engage in vigorous debate, and quite 
reasonably want to avail themselves of recent versions of standard tools. 
And I'd be the first to point out that Apple is absolutely notorious for 
having brain-dead ancient versions of tools, and never updating them. For 
one of many example domains, look at file meta-data handling: A number of 
tools now do a competent job, but never the version that Apple ships. My 
view of Apple tools is fairly widespread; I found myself wondering if 
Homebrew is grooming itself to be adopted by Apple, so it turns a blind 
eye to this widely observed phenomenon?

So Macaulay2 depends on a more recent version of autoconf than Apple 
ships, and autoconf formulas have been blacklisted because of the "no 
dups" policy. What do I do? I seek adoption of macaulay2 as a vanilla 
formula, and getting a specific exemption for autoconf feels like beating 
a system that instead needs reform. Naive users should see Apple tools by 
default, but they also shouldn't have to navigate ideological debates like
this to get their package installed!

Are we on the verge of a Homebrew release that quietly supports versions? 
I'm reminded of working at a famous Lake Ontario burger restaurant as a 
kid: We served medium unless one requested otherwise, but some savvy 
customers ordered medium. It was discreet code for "I know it's the lunch 
rush, but cook mine from scratch?"

So here, requesting a specific version of anything that would otherwise 
run afoul of the "no dups" policy should just work, keg-only, with the 
formulas that accomplish this tucked further from sight. Formulas needing 
these specific versions would jimmy their paths to find them, keg-only.

Even the Haskell package manager has pretty dismal version handling, 
though one of the Macaulay authors explained to me how Ruby has an 
excellent system. So I'm surprised that with its Ruby roots, Homebrew has 
no such system. I'd argue against an overly ambitious system for Homebrew,
but one that works cleanly, at perhaps the expense of space. Requesting 
explicit, specific versions rather than version ranges wastes space, but 
is most likely to "just work".

"Just works" is what drew me to Homebrew, after all. Homebrew deftly 
builds an amazing list of tools that in my experience the other approaches
botch. Saving people time is the whole ball of wax. If this costs extra 
disk space, so be it.

I skimmed the massive wall of text you wrote so if I missed the gist then 
apologies, but you should write less ;)

Some other formula depend on autoconf that is newer than that which OS X 
provides, I suggest you grep and figure out what to do.

The no dupes policy is a policy. We are ultimately practical and will 
accept a *good* compromise for what is necessary.

The policy has helped us to not back ourselves into a corner of headaches,
that is why it is there. It forces people to think outside the box and 
come up with more maintainable solutions. Best of luck,

Max

> Hi. I'm new to Homebrew, and enthusiastic about most of the design 
choices, though I'm wrestling with the "no dups" policy.
> 
> I have a draft of a working formula for Macaulay2, a specialized 
computer algebra system:
> 
> 	http://www.macaulay2.com/
> 	Depends on: autoconf, gnu-tar, bdw-gc, mpfr
> 
> The authors are friends I can engage in vigorous debate, and quite 
reasonably want to avail themselves of recent versions of standard tools. 
And I'd be the first to point out that Apple is absolutely notorious for 
having brain-dead ancient versions of tools, and never updating them. For 
one of many example domains, look at file meta-data handling: A number of 
tools now do a competent job, but never the version that Apple ships. My 
view of Apple tools is fairly widespread; I found myself wondering if 
Homebrew is grooming itself to be adopted by Apple, so it turns a blind 
eye to this widely observed phenomenon?
> 
> So Macaulay2 depends on a more recent version of autoconf than Apple 
ships, and autoconf formulas have been blacklisted because of the "no 
dups" policy. What do I do? I seek adoption of macaulay2 as a vanilla 
formula, and getting a specific exemption for autoconf feels like beating 
a system that instead needs reform. Naive users should see Apple tools by 
default, but they also shouldn't have to navigate ideological debates like
this to get their package installed!
> 
> Are we on the verge of a Homebrew release that quietly supports 
versions? I'm reminded of working at a famous Lake Ontario burger 
restaurant as a kid: We served medium unless one requested otherwise, but 
some savvy customers ordered medium. It was discreet code for "I know it's
the lunch rush, but cook mine from scratch?"
> 
> So here, requesting a specific version of anything that would otherwise 
run afoul of the "no dups" policy should just work, keg-only, with the 
formulas that accomplish this tucked further from sight. Formulas needing 
these specific versions would jimmy their paths to find them, keg-only.
> 
> Even the Haskell package manager has pretty dismal version handling, 
though one of the Macaulay authors explained to me how Ruby has an 
excellent system. So I'm surprised that with its Ruby roots, Homebrew has 
no such system. I'd argue against an overly ambitious system for Homebrew,
but one that works cleanly, at perhaps the expense of space. Requesting 
explicit, specific versions rather than version ranges wastes space, but 
is most likely to "just work".
> 
> "Just works" is what drew me to Homebrew, after all. Homebrew deftly 
builds an amazing list of tools that in my experience the other approaches
botch. Saving people time is the whole ball of wax. If this costs extra 
disk space, so be it.
> 

On Aug 16, 2011, at 4:49 AM, Max Howell wrote:

> I missed the gist then apologies, but you should write less ;)

Point taken, but as a newbie I wish you guys wrote a bit more. This is a 
real, pervasive problem.

For kicks I tried putting a second keg-only class in my formula; you of 
course don't support this. If I roll my own installs, I lose the 
advantages of the Homebrew infrastructure. If I spin off separate keg-only
formulas for Apple dups, you won't distribute them. This is a rock and a 
hard place.

Apple only got so far with a flat file system. You can only get so far 
without some kind of version support. I'm all for keeping version support 
simpler than anyone has tried before (e.g. by recognizing that disk space 
is cheaper than it used to be), but it has to be there.

I'd be happy with hierarchical depends-on, e.g.

	depends_on 'macaulay2/autoconf'

Then you don't need any further version control, you can hide these from 
casual browsing, and your "no dups" policy could evolve into "no dups at 
top level, and dependent formulas must be keg-only".

Or you could allow additional, supporting keg-only classes in formula. 
Same effect, and easier to distribute.

On Tue, Aug 16, 2011 at 10:21 AM, Dave Bayer <bayer@cpw.math.columbia.edu>wrote:

> On Aug 16, 2011, at 4:49 AM, Max Howell wrote:
>
> > I missed the gist then apologies, but you should write less ;)
>
> Point taken, but as a newbie I wish you guys wrote a bit more. This is a
> real, pervasive problem.
>
> For kicks I tried putting a second keg-only class in my formula; you of
> course don't support this. If I roll my own installs, I lose the advantages
> of the Homebrew infrastructure. If I spin off separate keg-only formulas for
> Apple dups, you won't distribute them. This is a rock and a hard place.
>

I still don't understand why Macaulay doesn't include a pre-built configure
script in their source releases---it is a very common thing to do. That way
the developers get to use whatever version of Autotools they wish and the
users building from source don't have to worry about Debian/Red
Hat/Homebrew/whatever supplying a compatible version in their "stable"
distribution.

Have you asked them about running autoconf before packing up their source
tarballs?

-Charlie

>> If I missed the gist then apologies, but you should write less ;)
> 
> Point taken, but as a newbie I wish you guys wrote a bit more. This is a
real, pervasive problem.

Yes, it's true, we need more documentation in these sorts of areas. Mostly
it's not there as the thinking for the solution is still evolving. Not 
that this is really a good excuse.

> For kicks I tried putting a second keg-only class in my formula; you of 
course don't support this. If I roll my own installs, I lose the 
advantages of the Homebrew infrastructure. If I spin off separate keg-only
formulas for Apple dups, you won't distribute them. This is a rock and a 
hard place.

This is actually supported. Check out the git formula for instance.

> Apple only got so far with a flat file system. You can only get so far 
without some kind of version support. I'm all for keeping version support 
simpler than anyone has tried before (e.g. by recognizing that disk space 
is cheaper than it used to be), but it has to be there.

Homebrew supports multiple versions of stuff installed in that as long as 
you don't `brew link` it's fine. In fact this is what a 'keg-only' formula
is.

> I'd be happy with hierarchical depends-on, e.g.
> 
> 	depends_on 'macaulay2/autoconf'
> 
> Then you don't need any further version control, you can hide these from
casual browsing, and your "no dups" policy could evolve into "no dups at 
top level, and dependent formulas must be keg-only".
> 
> Or you could allow additional, supporting keg-only classes in formula. 
Same effect, and easier to distribute.

I think we probably need an autoconf formula for the newer version. 
Keg-only like you say. I'll let adamv comment though.

In the meantime you should be able to do what you are suggesting. I know 
that the Spidermonkey formula used to for instance. Try `brew versions 
spidermonkey` and there should be an old version that installed its own 
autoconf that it needed.

Max

Sorry to hammer this list with questions, but I've googled and studied 
your wiki to no avail. I know darcs and mercurial, but I'm just now 
learning git in order to develop Homebrew recipes. This may be unstated 
because those who know consider it obvious:

???
I installed Homebrew following the standard directions, to /usr/local. I 
then forked homebrew, to a separate directory. There were no separate 
directions for how to install Homebrew in the first place, for the case 
where one plans to contribute formulas.

Am I supposed to now symlink /usr/local/Library to my forked homebrew 
Library, so brew uses my fork? Or after developing each formula, am I to 
manually copy each formula into my fork? I could not find documentation 
for the former, but the latter seems clumsy.

Thanks

(I'm running some lengthy tests before continuing my previous question. My
experience didn't match the reply, so I'm repeating my tests.)

The general model is something like this:

1) Install homebrew, e.g. to /usr/local
2) Fork the homebrew repository on github (easy, done with the press
of a button on the site)
3) Make changes to homebrew *inside* /usr/local, make sure it's *on a
branch*, and then push your branch to *your fork on github*, and make
a pull request.

Step 3 is actually a couple of steps but it's not hard. The process
looks like this:


$ cd `brew --prefix` # by default, will take us to /usr/local
$ git add remote github git@github.com:thoughtpolice/homebrew.git #
NOTE: use your own SSH login for your fork
$ git checkout -b fix-for-foo # make a new branch, AND switch to it
... make your changes to your formula
$ git commit
$ git push github fix-for-foo # create branch 'fix-for-foo' on the
'github' remote

Once you have done all this, you will have a branch 'fix-for-foo' on
your homebrew fork. Go to your github repository for homebrew, switch
to the 'fix-for-foo' branch (look at the top left, under the 'Source'
button, there will be a dropdown with 'switch branches' and you can
change to it.) After that, just hit the 'pull request' button near the
top right. This will make a pull request on Max's master branch, so
they'll see your changes and can merge them.

A lot of this is covered in the Formula Cookbook on the wiki. Please
be sure to read over it fully (it touches on many other things you
should consider,) so everyone's life will be easier. :)

https://github.com/mxcl/homebrew/wiki/Formula-Cookbook

Hope it helps.

On Wed, Aug 17, 2011 at 11:10 AM, Dave Bayer
<bayer@cpw.math.columbia.edu> wrote:
> Sorry to hammer this list with questions, but I've googled and studied 
your wiki to no avail. I know darcs and mercurial, but I'm just now 
learning git in order to develop Homebrew recipes. This may be unstated 
because those who know consider it obvious:
>
> ???
> I installed Homebrew following the standard directions, to /usr/local. I
then forked homebrew, to a separate directory. There were no separate 
directions for how to install Homebrew in the first place, for the case 
where one plans to contribute formulas.
>
> Am I supposed to now symlink /usr/local/Library to my forked homebrew 
Library, so brew uses my fork? Or after developing each formula, am I to 
manually copy each formula into my fork? I could not find documentation 
for the former, but the latter seems clumsy.
>
> Thanks
>
> (I'm running some lengthy tests before continuing my previous question. 
My experience didn't match the reply, so I'm repeating my tests.)
>



-- 
Regards,
Austin

On Aug 17, 2011, at 9:26 AM, austin seipp wrote:

> $ cd `brew --prefix` # by default, will take us to /usr/local
> $ git add remote github git@github.com:thoughtpolice/homebrew.git #
> NOTE: use your own SSH login for your fork
> ...
> A lot of this is covered in the Formula Cookbook on the wiki. Please
> be sure to read over it fully (it touches on many other things you
> should consider,) so everyone's life will be easier. :)

Thanks. I have read the Formula Cookbook, which explains how e.g. to 
submit changes. It doesn't address letting git take over /usr/local. I did
google and read before asking.

When I attempted your instructions, git complained that /usr/local wasn't 
a git repository.

From my understanding of how other systems work, I'm in any case leery of 
letting git take over /usr/local, which contains files that I don't want 
git or Homebrew touching. (These files coexist just fine with Homebrew 
itself.)

So I'm busy reading http://book.git-scm.com/ before proceeding. My 
preference would be to symlink /usr/local/Library/Formula to my repo 
elsewhere, as a tidier way of controlling what git sees.

On Wed, Aug 17, 2011 at 1:12 PM, Dave Bayer <bayer@cpw.math.columbia.edu>wrote:

>
> When I attempted your instructions, git complained that /usr/local wasn't a
> git repository.
>
> >From my understanding of how other systems work, I'm in any case leery of
> letting git take over /usr/local, which contains files that I don't want git
> or Homebrew touching. (These files coexist just fine with Homebrew itself.)
>
>
Homebrew does establish a git repository in `/usr/local`, but it doesn't
really "take it over". Everything in `/usr/local` is placed on Git's ignore
list and then specific things that Homebrew needs to track are added back
in. You can find the list of things that Homebrew's Git repository will take
an interest in in the `.gitignore` file:

    https://github.com/Sharpie/homebrew/blob/master/.gitignore


So I'm busy reading http://book.git-scm.com/ before proceeding. My
> preference would be to symlink /usr/local/Library/Formula to my repo
> elsewhere, as a tidier way of controlling what git sees.


If you want to work with your own personal repository of formulae, I would
definitely suggest checking out my `brew-tap` addition which makes it easier
to integrate alternate repositories into Homebrew. Hopefully `brew-tap` will
make it into the standard distribution, but for now it can be found in a
pull request:

    https://github.com/mxcl/homebrew/pull/6086

The tool provides easy access to any repository in the fork network of
Adam's Homebrew-alt:

    https://github.com/adamv/homebrew-alt

The man page contains usage notes:



https://raw.github.com/Sharpie/homebrew/brew-tap/Library/Contributions/manpages/brew-tap.1.md

On 17 Aug 2011, at 21:12, Dave Bayer wrote:

> Thanks. I have read the Formula Cookbook, which explains how e.g. to 
submit changes. It doesn't address letting git take over /usr/local. I did
google and read before asking.
> 
> When I attempted your instructions, git complained that /usr/local 
wasn't a git repository.
> 
>> From my understanding of how other systems work, I'm in any case leery 
of letting git take over /usr/local, which contains files that I don't 
want git or Homebrew touching. (These files coexist just fine with 
Homebrew itself.)
> 
> So I'm busy reading http://book.git-scm.com/ before proceeding. My 
preference would be to symlink /usr/local/Library/Formula to my repo 
elsewhere, as a tidier way of controlling what git sees.

If you don't know what you're doing I'd advise you to do things the way we
recommend. The defaults are default for a reason.

--
Mike McQuaid
http://mikemcquaid.com

On Aug 17, 2011, at 9:26 AM, austin seipp wrote:

> $ git add remote github git@github.com:thoughtpolice/homebrew.git #
> NOTE: use your own SSH login for your fork

On Aug 17, 2011, at 2:25 PM, Mike McQuaid wrote:

> If you don't know what you're doing I'd advise you to do things the way 
we recommend. The defaults are default for a reason.

I absolutely agree. The Homebrew site gives minimal directions for a new 
contributor installation, and what I tried did not work.

I started over entirely from scratch, to help you debug your new user 
experience. Here is an abridged terminal session:

book % cd /usr
book % sudo mv local local3
book % sudo mkdir local
book % sudo chown dave:staff local
book % /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"
book % cd `brew --prefix`
book % pwd
/usr/local
book % brew install git
book % git add remote github git@github.com:Syzygies/homebrew.git
fatal: Not a git repository (or any of the parent directories): .git

After googling, I tried replacing 'add remote' by 'remote add':

book % git remote add github git@github.com:Syzygies/homebrew.git
fatal: Not a git repository (or any of the parent directories): .git

However (unlike 'add remote') this works with a test repo:

book % mkdir test
book % cd test
book % git init
Initialized empty Git repository in /Global/Code/homebrew/test/.git/
book % git remote add github git@github.com:Syzygies/homebrew.git

Thanks,
Dave

> I absolutely agree. The Homebrew site gives minimal directions for a new
contributor installation, and what I tried did not work.

I think your problem was you assumed that there were additional steps.

There are NO additional steps.

Homebrew is set up for contribution as soon as you install it.

I'll look over the wiki to see if it can be improved anyway.

Max

On Aug 19, 2011, at 9:23 AM, Max Howell wrote:

> I think your problem was you assumed that there were additional steps.
> 
> There are NO additional steps.

Ok, I thought I was getting git. I've read some books on git. I've 
switched to git for my other work.

My /usr/local is under git control, based on my clone of Homebrew, and 
following your directions as best I could:

	https://github.com/Syzygies/homebrew/branches/master

I'm using two approaches to keeping current:

	brew update

	git fetch upstream
	git merge upstream/master

While I have the impression that they're doing roughly the same thing, 
making one redundant, I haven't seen explicit documentation to this 
effect, and I haven't studied the 'update' portion of the 'brew code'. I'm
using both approaches because my github view isn't updating, so I'm 
flailing.

	git push github master
Everything up-to-date

So why does the github view show my master last updated Aug 22, and 233 
changes behind? I've been googling for half an hour for that 'git 
no-really,-sync-not-kidding command I must be forgetting. Or is this just 
instrumentation, the github view is gorped?

Thanks,
Dave

It's true we don't give a recommended guide for how to set up your remotes.

git fetch and git merge are not the same, they are the two separate steps 
that are git pull.

I recommend using a tool like gitx to visually see the state of your 
remotes, it works wonders for education.

> On Aug 19, 2011, at 9:23 AM, Max Howell wrote:
> 
>> I think your problem was you assumed that there were additional steps.
>> 
>> There are NO additional steps.
> 
> Ok, I thought I was getting git. I've read some books on git. I've 
switched to git for my other work.
> 
> My /usr/local is under git control, based on my clone of Homebrew, and 
following your directions as best I could:
> 
> 	https://github.com/Syzygies/homebrew/branches/master
> 
> I'm using two approaches to keeping current:
> 
> 	brew update
> 
> 	git fetch upstream
> 	git merge upstream/master
> 
> While I have the impression that they're doing roughly the same thing, 
making one redundant, I haven't seen explicit documentation to this 
effect, and I haven't studied the 'update' portion of the 'brew code'. I'm
using both approaches because my github view isn't updating, so I'm 
flailing.
> 
> 	git push github master
> Everything up-to-date
> 
> So why does the github view show my master last updated Aug 22, and 233 
changes behind? I've been googling for half an hour for that 'git 
no-really,-sync-not-kidding command I must be forgetting. Or is this just 
instrumentation, the github view is gorped?
> 
> Thanks,
> Dave
> 

Thanks.

On Sep 1, 2011, at 8:57 AM, Jack Nagel wrote:

> So they are basically the same, but if you also have a remote called 
'upstream', then they differ depending on which remote branch 
(origin/master or upstream/master) you local master is tracking.

Yes, I added 'upstream' following generic advice, not having seen anywhere
a description of which advice to skip, given that Homebrew already sets up
a lot of this.

Removing upstream, and using 'brew update' as my only mechanism for 
updating the master branch on my Mac, I'm getting more understandable 
behavior on GitHub.

On Sep 1, 2011, at 8:57 AM, Max Howell wrote:

> I recommend using a tool like gitx to visually see the state of your 
remotes, it works wonders for education.

It is a nice tool.

On Thursday, September 1, 2011, Dave Bayer <bayer@cpw.math.columbia.edu>
wrote:
> I'm using two approaches to keeping current:
>
>        brew update
>
>        git fetch upstream
>        git merge upstream/master

Homebrew uses its own remote ("origin", this is a recent change), and 'brew
update' is basically

 git pull origin refs/heads/master:refs/remotes/origin/master

which is functionally equivalent to

 git fetch origin
 git merge origin/master

So they are basically the same, but if you also have a remote called
'upstream', then they differ depending on which remote branch (origin/master
or upstream/master) you local master is tracking.

>        git push github master
> Everything up-to-date
>
> So why does the github view show my master last updated Aug 22, and 233
changes behind? I've been googling for half an hour for that 'git
no-really,-sync-not-kidding command I must be forgetting. Or is this just
instrumentation, the github view is gorped?

I'd have to know more details to answer this definitively. What branch is
your local master tracking?

Jack

On Aug 19, 2011, at 9:23 AM, Max Howell wrote:

> I think your problem was you assumed that there were additional steps.
> 
> There are NO additional steps.

Yes, I was worrying about many imaginary demons, based on assumptions 
formed using other systems. Also, as 0.8 software it's easy to imagine 
you're not as far along as you are.. I'm late to this party but used to 
trying early versions of software; one is generous in one's expectations, 
and treads carefully.

Reading how git is different helps a lot; http://book.git-scm.com/ is an 
efficient read. Concerns over putting /usr/local under git control? Hmm, I
wonder how much Linus puts under control, makes my concerns look trivial.

It would be nice if the wiki had a single start-to-finish 
new-to-Git-and-Homebrew set of instructions for zero-to-contribute. That 
would make your point, I was pulling directions from all over, hence the 
gaps. I could write out what I found I needed to know?

The other wiki contribution I was imagining was a Ruby crib sheet: If one 
learns enough Ruby to contribute by looking at sample code, what are the 
semantic issues to be aware of? The list is VERY short, but knowing that 
state changes don't persist between system calls, and what to do about it,
is a big one. I learned Ruby once in passing, so my ignorance could be 
useful.

(I've had a working macaulay2.rb formula for days, we're just still 
testing. These guys are very detail-oriented, with strong ideas as to the 
right way to do things.)

>> I think your problem was you assumed that there were additional steps.
>> 
>> There are NO additional steps.
> 
> Yes, I was worrying about many imaginary demons, based on assumptions 
formed using other systems. Also, as 0.8 software it's easy to imagine 
you're not as far along as you are.. I'm late to this party but used to 
trying early versions of software; one is generous in one's expectations, 
and treads carefully.
> 
> Reading how git is different helps a lot; http://book.git-scm.com/ is an
efficient read. Concerns over putting /usr/local under git control? Hmm, I
wonder how much Linus puts under control, makes my concerns look trivial.
> 
> It would be nice if the wiki had a single start-to-finish 
new-to-Git-and-Homebrew set of instructions for zero-to-contribute. That 
would make your point, I was pulling directions from all over, hence the 
gaps. I could write out what I found I needed to know?
> 
> The other wiki contribution I was imagining was a Ruby crib sheet: If 
one learns enough Ruby to contribute by looking at sample code, what are 
the semantic issues to be aware of? The list is VERY short, but knowing 
that state changes don't persist between system calls, and what to do 
about it, is a big one. I learned Ruby once in passing, so my ignorance 
could be useful.
> 
> (I've had a working macaulay2.rb formula for days, we're just still 
testing. These guys are very detail-oriented, with strong ideas as to the 
right way to do things.)

Yes you are right, zero-to-hero would be a useful guide to have. For sure 
what we have now expects both Git and Ruby. Would be very useful.

If you could detail what you needed it would be a good start and I could 
plan something or maybe someone else could start a wiki for it and 
eventually we'd mix it all up into an ultimate cookbook.

This could also be forked and brought up-to-date, for instance:

https://github.com/adamv/homebrew-cookbook

On Tue, Aug 23, 2011 at 10:58 AM, Max Howell <max@methylblue.com> wrote:
>>> I think your problem was you assumed that there were additional steps.
>>>
>>> There are NO additional steps.
>>
>> Yes, I was worrying about many imaginary demons, based on assumptions 
formed using other systems. Also, as 0.8 software it's easy to imagine 
you're not as far along as you are.. I'm late to this party but used to 
trying early versions of software; one is generous in one's expectations, 
and treads carefully.
>>
>> Reading how git is different helps a lot; http://book.git-scm.com/ is 
an efficient read. Concerns over putting /usr/local under git control? 
Hmm, I wonder how much Linus puts under control, makes my concerns look 
trivial.
>>
>> It would be nice if the wiki had a single start-to-finish 
new-to-Git-and-Homebrew set of instructions for zero-to-contribute. That 
would make your point, I was pulling directions from all over, hence the 
gaps. I could write out what I found I needed to know?
>>
>> The other wiki contribution I was imagining was a Ruby crib sheet: If 
one learns enough Ruby to contribute by looking at sample code, what are 
the semantic issues to be aware of? The list is VERY short, but knowing 
that state changes don't persist between system calls, and what to do 
about it, is a big one. I learned Ruby once in passing, so my ignorance 
could be useful.
>>
>> (I've had a working macaulay2.rb formula for days, we're just still 
testing. These guys are very detail-oriented, with strong ideas as to the 
right way to do things.)
>
> Yes you are right, zero-to-hero would be a useful guide to have. For 
sure what we have now expects both Git and Ruby. Would be very useful.
>
> If you could detail what you needed it would be a good start and I could
plan something or maybe someone else could start a wiki for it and 
eventually we'd mix it all up into an ultimate cookbook.
>

> book % brew install git
> book % git add remote github git@github.com:Syzygies/homebrew.git
> fatal: Not a git repository (or any of the parent directories): .git

You need to `brew update` first to set up the initial git repository.

Jack

>From my understanding of how other systems work, I'm in any case leery of
letting git take over /usr/local, which contains files that I don't want 
git or Homebrew touching. (These files coexist just fine with Homebrew 
itself.)

Note that much of Homebrew is built on top of git, so you may run into
problems depending on how robust your system of symlinks into
/usr/local is.

Jack

Thanks all for the responses to my earlier questions in the thread 
"Current thinking on formulas that need newer versions of Apple tools?" I 
ended up studying lots of code in the Library/Formula directory, as you 
and the Formula Cookbook urge.

There appears to be variable degrees of clumsiness in tackling each 
problem that comes up. As a Ruby tourist, I sympathize. For one example, 
shell state changes don't persist between system calls; a Ruby tourist may
not know this. One can change the PATH with ENV.prepend, but I also see 
code that gloms on the same PATH change to multiple system calls. As this 
is a model code base for Ruby tourists coming in to learn enough Ruby to 
contribute to Homebrew, do you want third-parties tweaking clumsy code? Or
does git offer a way to comment on code, for the maintainer to consider?

???
Today's question: I didn't see good formula examples for maintaining 
multiple, parallel versions of a program. My norm for programs I care 
about, such as GHC Haskell, is to have many versions side-by-side, 
arbitrated by the PATH variable in the same spirit as your keg-only 
installs. We also want as many Macaulay2 users as possible to test out the
development head, without messing up their day-to-day work. They need to 
be able to quickly revert back to the stable version, and Macaulay2 is not
a quick build.

It appears to me that keg_only is a static class attribute, not a command 
that can be optionally called e.g. for a --HEAD build or by request. It 
also appears to me that each install for a given formula name wipes out 
the previous install. This makes the --HEAD option a bit problematic, as 
one loses the stable version.

I'd argue that --HEAD should be keg-only by default, not wiping out the 
stable build. End of story.

Were there an install option that prevented deleting old builds, then all 
but the last build would become keg-only by default. This option would 
want to explicitly convert the previous build to keg-only, unlinking the 
previously linked build, to avoid orphan links in case what is built 
changes.

Alternatively, I could provide separate macaulay2 and macaulay2-head 
formulas, with macaulay2-head explicitly keg-only. Is this what you want? 
If there are already examples of this, I didn't see them. If you demote 
formulas like  macaulay2-head to secondary repos, then the directions need
expanding on using multiple repos, which can get brutal if one doesn't 
know one's way around git. Homebrew doesn't like a formula to simply be 
dragged into its Formula directory.

I'd prefer a single formula that can switch-hit, for optionally building 
--HEAD (now supported) and optionally installing keg-only (which I 
couldn't get to work).

Thanks

On Aug 16, 2011, at 10:29 AM, Max Howell wrote:

> Homebrew supports multiple versions of stuff installed in that as long 
as you don't `brew link` it's fine. In fact this is what a 'keg-only' 
formula is.

> In the meantime you should be able to do what you are suggesting. I know
that the Spidermonkey formula used to for instance. Try `brew versions 
spidermonkey` and there should be an old version that installed its own 
autoconf that it needed.

On 17 Aug 2011, at 15:11, Dave Bayer wrote:

> Were there an install option that prevented deleting old builds, then 
all but the last build would become keg-only by default. This option would
want to explicitly convert the previous build to keg-only, unlinking the 
previously linked build, to avoid orphan links in case what is built 
changes.

Old builds are only deleted when installing the same version again so the 
solution is just to use stable version numbers and HEAD.

--
Mike McQuaid
http://mikemcquaid.com

On Aug 17, 2011, at 7:57 AM, Mike McQuaid wrote:

> On 17 Aug 2011, at 15:11, Dave Bayer wrote:
> 
>> Were there an install option that prevented deleting old builds, then 
all but the last build would become keg-only by default. This option would
want to explicitly convert the previous build to keg-only, unlinking the 
previously linked build, to avoid orphan links in case what is built 
changes.
> 
> Old builds are only deleted when installing the same version again so 
the solution is just to use stable version numbers and HEAD.

Thanks. A different problem masked this behavior before; I am now able to 
separately install stable and HEAD versions using the same formula.

As I feared, I got orphan links. This doesn't convert one of the installs 
to keg-only, it blindly installs one over the other. As a test, I threw 
'foo' into one #{prefix}/bin and 'foo2' into the other #{prefix}/bin, as 
they were each being built. Homebrew symlinked each of 'foo' and 'foo2' in
turn into /usr/local/bin, creating a Frankenstein build. This won't hurt 
most projects, but it is a bug. As this list is a place to ask for help 
but not to report bugs, I will open a ticket unless someone objects.

"brew remove --force macaulay2" does remove all symlinks from both versions.

> As I feared, I got orphan links. This doesn't convert one of the 
installs to keg-only, it blindly installs one over the other. As a test, I
threw 'foo' into one #{prefix}/bin and 'foo2' into the other 
#{prefix}/bin, as they were each being built. Homebrew symlinked each of 
'foo' and 'foo2' in turn into /usr/local/bin, creating a Frankenstein 
build. This won't hurt most projects, but it is a bug. As this list is a 
place to ask for help but not to report bugs, I will open a ticket unless 
someone objects.

A temporary solution might be to `brew unlink foo` before installing a
newer version, though I agree that it may be a good idea to do this
automatically when installing a new version of an existing
installation.

Jack

On 16 Aug 2011, at 18:29, Max Howell wrote:

> I think we probably need an autoconf formula for the newer version. 
Keg-only like you say. I'll let adamv comment though.

Autoconf is a bad example here. As Sharpie pointed out, it isn't actually 
needed at compile-time. In addition, most Linux distributions end up 
packaging loads of versions because it isn't backwards compatible.

--
Mike McQuaid
http://mikemcquaid.com

On 16 Aug 2011, at 06:19, Dave Bayer wrote:

> The authors are friends I can engage in vigorous debate, and quite 
reasonably want to avail themselves of recent versions of standard tools. 
And I'd be the first to point out that Apple is absolutely notorious for 
having brain-dead ancient versions of tools, and never updating them. For 
one of many example domains, look at file meta-data handling: A number of 
tools now do a competent job, but never the version that Apple ships. My 
view of Apple tools is fairly widespread; I found myself wondering if 
Homebrew is grooming itself to be adopted by Apple, so it turns a blind 
eye to this widely observed phenomenon?

As the most fascist of the Homebrew contributors when it comes to 
duplicates let me explain my thoughts here.

On a Linux system people generally try and get their software compiling on
e.g. the latest version of Ubuntu. They don't depend on newer versions of 
autoconf than are provided by Ubuntu because then it becomes a pain to 
install and build. Thanks to Fink and Macports, OSX is not seem in the 
same light so it becomes a pain to install and build software on OSX 
without first building huge numbers of dependencies. This argument also 
applies to our patching, incidentally, which is why I try to encourage 
people to get our patches upstream so other people on OSX can compile the 
software without knowing some arcane knowledge.

In this case I wonder what is needed in the newer version of autoconf. I 
wonder how long it would take the developers to make it work with the 
version provided by OSX. If the answer is "not very long" then they are 
choosing their own minor convenience and forcing each of the users of 
their program on OSX to waste another 10 minutes compiling autoconf before
they can use this software. I don't think I'm alone in the belief in 
thinking this is a bit silly.

> So Macaulay2 depends on a more recent version of autoconf than Apple 
ships, and autoconf formulas have been blacklisted because of the "no 
dups" policy. What do I do? I seek adoption of macaulay2 as a vanilla 
formula, and getting a specific exemption for autoconf feels like beating 
a system that instead needs reform. Naive users should see Apple tools by 
default, but they also shouldn't have to navigate ideological debates like
this to get their package installed!

You can include autoconf as part of the Macaulay2 formula so it is just 
built and stored in the Macaulay2 Cellar as a sub formula. I don't think 
people have to navigate these debates; quite the opposite. If you compare 
us to Macports it's because of this decision that Git has no dependencies 
in Homebrew compared to a few in Macports. This means our users spend less
time compiling software they don't care about.

> So here, requesting a specific version of anything that would otherwise 
run afoul of the "no dups" policy should just work, keg-only, with the 
formulas that accomplish this tucked further from sight. Formulas needing 
these specific versions would jimmy their paths to find them, keg-only.


You'd probably be surprised how few formulae actually depend on specific 
versions of software rather than just the latest. In this case, again, 
it's just a bit harder for the formula writer but creates a better 
experience for the user.

--
Mike McQuaid
http://mikemcquaid.com

On Mon, Aug 15, 2011 at 10:19 PM, Dave Bayer <bayer@cpw.math.columbia.edu>wrote:

> Hi. I'm new to Homebrew, and enthusiastic about most of the design choices,
> though I'm wrestling with the "no dups" policy.
>
> I have a draft of a working formula for Macaulay2, a specialized computer
> algebra system:
>
>        http://www.macaulay2.com/
>        Depends on: autoconf, gnu-tar, bdw-gc, mpfr
>
> The authors are friends I can engage in vigorous debate, and quite
> reasonably want to avail themselves of recent versions of standard tools.
> And I'd be the first to point out that Apple is absolutely notorious for
> having brain-dead ancient versions of tools, and never updating them. For
> one of many example domains, look at file meta-data handling: A number of
> tools now do a competent job, but never the version that Apple ships. My
> view of Apple tools is fairly widespread; I found myself wondering if
> Homebrew is grooming itself to be adopted by Apple, so it turns a blind eye
> to this widely observed phenomenon?
>

I have never gotten the feeling that the Homebrew core team is interested in
being "adopted by Apple". Even if they were, it would be pretty hard for
one organization to exert any sort of control over a project with 2,500+
forks on GitHub---the community can easily jump ship to another fork if so
inclined.

Besides, I always thought that MacPorts was Apple's adopted package manager.

As far as I can tell, the "no dupes" policy has always been about cutting
the support burden by not having multiple copies of core libraries running
around confusing the buildsystems. The kind of errors produced when software
tries and fails to compile in an environment that provides two flavors of
the same library are very difficult to diagnose and debug.

So, basically Homebrew is choosing an evil that is known and constant over
evil that is unknown, takes hours of hair pulling to figure out, and will
probably blow up in an entirely different way for the next library that runs
into it.


So Macaulay2 depends on a more recent version of autoconf than Apple ships,
> and autoconf formulas have been blacklisted because of the "no dups" policy.
> What do I do? I seek adoption of macaulay2 as a vanilla formula, and getting
> a specific exemption for autoconf feels like beating a system that instead
> needs reform. Naive users should see Apple tools by default, but they also
> shouldn't have to navigate ideological debates like this to get their
> package installed!
>

Why do you need Autoconf to make a formula for Macaulay2? Usually source
releases include a pre-generated configure script. It is highly unusual to
be running autoconf in order to generate configure when one is just building
a release copy of some software.

If you are a developer who needs to modify the buildsystem, that is an
entirely different matter and Homebrew provides a few ways to get the newest
Autotools installed.


 Are we on the verge of a Homebrew release that quietly supports versions?
> I'm reminded of working at a famous Lake Ontario burger restaurant as a kid:
> We served medium unless one requested otherwise, but some savvy customers
> ordered medium. It was discreet code for "I know it's the lunch rush, but
> cook mine from scratch?"
>

Homebrew does have a few codewords for "cook it my way". The first is to
maintain your own custom formula that builds and installs software exactly
the way you want it. This can be work well in some situations but can be
tedious and lead to duplication of effort if the custom formula is something
well-known (like a newer version of Autoconf).

A second way to do it is to install a formula from an alternate repository
that contains a version you like better. For example, Adam's Hombrew-alt
repository doesn't follow many of the rules that Mxcl's mater fork has, so
you can get up-to-date versions of Autoconf as easily as:

    brew install
https://raw.github.com/adamv/homebrew-alt/master/duplicates/autoconf.rb

Finally, there's a tool I wrote called `brew-tap` that lets you "tap into"
other repositories in the fork network of Homebrew-Alt and easily install or
update the formula they contain. Brew tap is available in pull request
#6086:

    https://github.com/mxcl/homebrew/pull/6086


So here, requesting a specific version of anything that would otherwise run
> afoul of the "no dups" policy should just work, keg-only, with the formulas
> that accomplish this tucked further from sight. Formulas needing these
> specific versions would jimmy their paths to find them, keg-only.
>
> Even the Haskell package manager has pretty dismal version handling, though
> one of the Macaulay authors explained to me how Ruby has an excellent
> system. So I'm surprised that with its Ruby roots, Homebrew has no such
> system. I'd argue against an overly ambitious system for Homebrew, but one
> that works cleanly, at perhaps the expense of space. Requesting explicit,
> specific versions rather than version ranges wastes space, but is most
> likely to "just work".
>
> "Just works" is what drew me to Homebrew, after all. Homebrew deftly builds
> an amazing list of tools that in my experience the other approaches botch.
> Saving people time is the whole ball of wax. If this costs extra disk space,
> so be it.
>

Homebrew is a very "dumb" package manager. By "dumb", I mean that it doesn't
keep track of any metadata from which intelligence could be derived. The
list of what is installed or available for installation is basically
generated by the `ls` command. This simplicity goes a long way to making
Homebrew "just work" for the majority of use cases.

Adam is currently working on adding support for tracking metadata---but it
is hard to come up with a system that doesn't sacrifice too much simplicity.
I'm sure he would appreciate help, comments and ideas.


-Charlie