librelist archives

« back to archive

quick compression patch for attic master / 0.14

quick compression patch for attic master / 0.14

From:
Thomas Waldmann
Date:
2015-04-01 @ 21:30
See there:


https://github.com/ThomasWaldmann/attic/commit/e7673c4af9f6b6cd2f20d191917b4c45fa65f605

Note: this is a MINIMAL patch for people who rather want to stay on attic
0.14 or master branch.

The code in merge-all repo has way better compression selection AND
different and faster algorithms.

Re: [attic] quick compression patch for attic master / 0.14

From:
Manuel Faux
Date:
2015-04-16 @ 04:43
Dear all,

On Wednesday, April 15, 2015 23:29, Jonas Borgström wrote:
> Please stop telling people to test your merge-all branch without making
> sure they understand that it it's very unlikely that the repositories it
> touches will be compatible with any current or future version of Attic.
> 
> Attic will get support for other compression algorithms but will very 
> likely not be compatible with your merge-all branch.

It makes me a bit worried and sad to recognize the dispute between the primary
developer of this project and some community members. For understandable,
personal reasons the primary developer slowed down the speed of development and
cycle of releases. Since this project seem to gain sufficient attention that
people were awaiting bugfixes and important feature requests, some community
members started implementing those requested software pieces. In general I
think this is an important milestone in any open source project, since it shows
the interest of a bigger group of users and developers, than only the group of
people initially started the project. Nevertheless, it seems that the initial
developer is unhappy with that development of things and even discourages
people of using those pieces of software not published via the official
repositories, to which only the core team has write access to. It was also
explicitly
denied to allow people to submit (more or less important) bugfixes to the
official repo.

I personally am using Attic currently in situations which require those feature
requests implemented in the "unofficial repo". For my personal requirements
Attic is unusable in its pure, official release, since it lacks important
features.  And interestingly the same argument is also valid for the core Attic
team: the primary developer mentioned, that Attic's primary purpose is to
fulfill his personal needs and requirements. I can totally understand this
point of view, since I also would not like to see a project under my control
drifting way that strong, that it cannot fulfill its basic requirements anymore
I initially created the project for! Nevertheless, this is how community based
projects work. Obviously nobody (including the primary developer) could foresee
this development of things, but now we are in the situation that two different
and most notable incompatible (!) branches of Attic exist and merging them
together seems unlikely due to this dispute.

I hope this dispute does not discourage anybody of the involved people, not the
core developer and neither the active community members, since both are doing
great work in developing good, clean software. Nevertheless, I also hope that
people using any piece of branch won't experience unexpected situations, like
being more or less dependent on a dead end branch...

Best regards,
Manuel

Re: [attic] quick compression patch for attic master / 0.14

From:
Jean Jordaan
Date:
2015-04-16 @ 05:17
On Thu, Apr 16, 2015 at 11:43 AM, Manuel Faux <manuel.faux@conf.at> wrote:
>
> It makes me a bit worried and sad to recognize the dispute between the primary
> developer of this project and some community members.

Please pardon me shoving my oar in, but Pieter Hintjens has thought
and written a lot about this kind of thing. Here is a bit of a
link-bomb:
  http://hintjens.com/blog:43 (tongue in cheek)
  http://hintjens.com/blog:10 (in earnest)
  http://hintjens.com/blog:47

Git branch vs fork:
  http://hintjens.com/blog:24
  http://hintjens.com/blog:23

Perhaps his formulations & ideas may help to inform necessary
discussion on this topic.
It looks like it's something that will have to be confronted by the
community emerging around attic.

-- 
jean                                              . .. .... //\\\oo///\\

Re: [attic] quick compression patch for attic master / 0.14

From:
Dan Williams
Date:
2015-04-16 @ 07:36
 >> > Please stop telling people to test your merge-all branch without
 >> making
 >> > sure they understand that it it's very unlikely that the repositories
 >> it
 >> > touches will be compatible with any current or future version of
 >> Attic.
 >> >
 >> > Attic will get support for other compression algorithms but will very
 >> > likely not be compatible with your merge-all branch.
 >> 
 >> It makes me a bit worried and sad to recognize the dispute between the
 >> primary
 >> developer of this project and some community members.

Agreed, and this is something that concerns me greatly. I have looked at 
the new 0.15 release but there seems little there of interest and I am not
sure whether to spend the time testing it or just move directly to 
Thomas's branches.

We have a critical juncture here where the project is at risk of split. 
I'd prefer not to see two version of Attic, plus the version with the most
contributors and activity is obviously going to end up the best. I will 
follow whichever version goes in the direction most suitable for my needs,
but it would be wonderful to see this resolved and greater collaboration. 
That's what open-source software is all about.

Just my $0.02.

Dan


Re: [attic] quick compression patch for attic master / 0.14

From:
Yuri D'Elia
Date:
2015-04-16 @ 15:38
On 04/16/2015 09:36 AM, Dan Williams wrote:
>>> It makes me a bit worried and sad to recognize the dispute
>>> between the primary developer of this project and some community
>>> members.

Is it one message all it takes?
It read like a reasonable statement to make to me (maybe a bit "dry").

> Agreed, and this is something that concerns me greatly. I have looked
> at the new 0.15 release but there seems little there of interest and
> I am not sure whether to spend the time testing it or just move
> directly to Thomas's branches.

I have several projects that I developed/maintain, some definitely
lagging behind and for which I still have patches sitting there for months.

Reviewing a patch, unless it's a trivial fix, takes me quite some time,
generally thinking about all the corner cases found over time that
people generally ignore.

If you think about the development of the project going in some
direction (changes you plan to make), there might be stuff you don't
want right now, technical decisions you dislike, and also fixes that
while "working" are technically just plain wrong.

Many times I received and had to heavily adapt patches that were made
without a sense of the internal architecture of the project. You don't
want to reject the patch because it indeed fixes a bug and/or improves
it, but it's literally hours of work to adapt, fix corner cases the OP
ignored, and test. I personally don't start fixing something that
requires my to work X hours, when I know I will only have X/2.

There is also a question about maintenance. Does the patch introduce
changes you are willing to *maintain* (talking about years here!) or
it's likely something that's gonna break as soon as you touch something
else.

Just saying.
It takes time.

Re: [attic] quick compression patch for attic master / 0.14

From:
Dan Williams
Date:
2015-04-16 @ 15:57
 >> -----Original Message-----
 >> From: attic@librelist.com [mailto:attic@librelist.com] On Behalf Of Yuri
 >> D'Elia
 >> Sent: 16 April 2015 16:38
 >> To: attic@librelist.com
 >> Subject: Re: [attic] quick compression patch for attic master / 0.14
 >> 
 >> On 04/16/2015 09:36 AM, Dan Williams wrote:
 >> >>> It makes me a bit worried and sad to recognize the dispute
 >> >>> between the primary developer of this project and some community
 >> >>> members.
 >> 
 >> Is it one message all it takes?

No, it's not one message - it has been an on-going conversation on the
mailing list for some time now.



Re: [attic] quick compression patch for attic master / 0.14

From:
Randy Syring
Date:
2015-04-18 @ 17:24
On 04/16/2015 12:43 AM, Manuel Faux wrote:
> Dear all,
>
> On Wednesday, April 15, 2015 23:29, Jonas Borgström wrote:
>> Please stop telling people to test your merge-all branch without making
>> sure they understand that it it's very unlikely that the repositories it
>> touches will be compatible with any current or future version of Attic.
>>
>> Attic will get support for other compression algorithms but will very
>> likely not be compatible with your merge-all branch.
> It makes me a bit worried and sad to recognize the dispute between the primary
> developer of this project and some community members. For understandable,
> personal reasons the primary developer slowed down the speed of development and
> cycle of releases. Since this project seem to gain sufficient attention that
> people were awaiting bugfixes and important feature requests, some community
> members started implementing those requested software pieces. In general I
> think this is an important milestone in any open source project, since it shows
> the interest of a bigger group of users and developers, than only the group of
> people initially started the project. Nevertheless, it seems that the initial
> developer is unhappy with that development of things and even discourages
> people of using those pieces of software not published via the official
> repositories, to which only the core team has write access to. It was 
also explicitly
> denied to allow people to submit (more or less important) bugfixes to the
> official repo.
>
> I personally am using Attic currently in situations which require those feature
> requests implemented in the "unofficial repo". For my personal requirements
> Attic is unusable in its pure, official release, since it lacks important
> features.  And interestingly the same argument is also valid for the core Attic
> team: the primary developer mentioned, that Attic's primary purpose is to
> fulfill his personal needs and requirements. I can totally understand this
> point of view, since I also would not like to see a project under my control
> drifting way that strong, that it cannot fulfill its basic requirements anymore
> I initially created the project for! Nevertheless, this is how community based
> projects work. Obviously nobody (including the primary developer) could foresee
> this development of things, but now we are in the situation that two different
> and most notable incompatible (!) branches of Attic exist and merging them
> together seems unlikely due to this dispute.
>
> I hope this dispute does not discourage anybody of the involved people, not the
> core developer and neither the active community members, since both are doing
> great work in developing good, clean software. Nevertheless, I also hope that
> people using any piece of branch won't experience unexpected situations, like
> being more or less dependent on a dead end branch...
>
> Best regards,
> Manuel
>

Manuel captures some of my concerns about the Attic project.  I've been 
interested in it for a while, but just lurking right now b/c I've sensed 
a bit of tension and uncertainty in the development of the project.

Before I can start using Attic, or recommending it to others, I'd like 
to see a more unified front from Jonas and other interested 
contributors.  He is under no obligation to do that of course.  But I 
really hope he will and that Attic will grow beyond a single-person 
project.  The reality is that life happens to us all and good software 
is rarely the product of a single person.  If Attic is going to become 
more popular, it will need a good team behind it.

Just my $0.02.  :)

Enjoy the weekend.

*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/

Re: [attic] quick compression patch for attic master / 0.14

From:
Jonas Borgström
Date:
2015-04-15 @ 21:29
On 04/01/2015 11:30 PM, Thomas Waldmann wrote:
> See there:
>
> 
https://github.com/ThomasWaldmann/attic/commit/e7673c4af9f6b6cd2f20d191917b4c45fa65f605
>
> Note: this is a MINIMAL patch for people who rather want to stay on
> attic 0.14 or master branch.
>
> The code in merge-all repo has way better compression selection AND
> different and faster algorithms.
Please stop telling people to test your merge-all branch without making
sure they understand that it it's very unlikely that the repositories it
touches will be compatible with any current or future version of Attic.

Attic will get support for other compression algorithms but will very
likely not be compatible with your merge-all branch.

/ Jonas

Re: [attic] quick compression patch for attic master / 0.14

From:
Leo Famulari
Date:
2015-04-17 @ 19:53
I hope that the attic project will be able to integrate the energies and
enthusiasm of the community that is developing around it.

It seems that the time has come for deduplicating rolling hash
archivers. There is tarsnap, but that is not open source (although many
of its components are, and they could be very valuable). There is also
zbackup, but it does not currently support remote operations (although
it has some of its own interesting advantages, like support for a
"common" data pool shared by clients). For now, attic offers the
advantages of a BSD license and (sparsely documented) support for remote
operations. These are very valuable features. It would be a shame for
the effort spent here to be wasted.

Jonas, when you have the time, would you be willing to evaluate some of
these volunteers and consider delegating some responsibility to those
you consider qualified and trustworthy?

On Wed, Apr 15, 2015 at 11:29:52PM +0200, Jonas Borgström wrote:
> On 04/01/2015 11:30 PM, Thomas Waldmann wrote:
> > See there:
> >
> > 
https://github.com/ThomasWaldmann/attic/commit/e7673c4af9f6b6cd2f20d191917b4c45fa65f605
> >
> > Note: this is a MINIMAL patch for people who rather want to stay on
> > attic 0.14 or master branch.
> >
> > The code in merge-all repo has way better compression selection AND
> > different and faster algorithms.
> Please stop telling people to test your merge-all branch without making
> sure they understand that it it's very unlikely that the repositories it
> touches will be compatible with any current or future version of Attic.
> 
> Attic will get support for other compression algorithms but will very
> likely not be compatible with your merge-all branch.
> 
> / Jonas
> 

Re: [attic] quick compression patch for attic master / 0.14

From:
Thomas Waldmann
Date:
2015-04-15 @ 22:40
2015-04-15 23:29 GMT+02:00 Jonas Borgström <jonas@borgstrom.se>:

> On 04/01/2015 11:30 PM, Thomas Waldmann wrote:
> > See there:
> >
> >
> 
https://github.com/ThomasWaldmann/attic/commit/e7673c4af9f6b6cd2f20d191917b4c45fa65f605
> >
> > Note: this is a MINIMAL patch for people who rather want to stay on
> > attic 0.14 or master branch.
> >
> > The code in merge-all repo has way better compression selection AND
> > different and faster algorithms.
> Please stop telling people to test your merge-all branch without making
> sure they understand that it it's very unlikely that the repositories it
> touches will be compatible with any current or future version of Attic.
>
>
Maybe you've missed some of the posts, but this was for people trying to
find the data corruption issues that they have seen when backing up
terabytes of data with attic release code - which takes some days per test
run to complete. I just wanted to help them so this can be found faster.

I always pointed out that the code in this repo is experimental / for tests
only, so there shouldn't be any problems.

Re: [attic] quick compression patch for attic master / 0.14

From:
Jonas Borgström
Date:
2015-04-16 @ 21:50
On 2015-04-16 00:40, Thomas Waldmann wrote:
>
> 2015-04-15 23:29 GMT+02:00 Jonas Borgström <jonas@borgstrom.se
> <mailto:jonas@borgstrom.se>>:
>
>     On 04/01/2015 11:30 PM, Thomas Waldmann wrote:
>     > See there:
>     >
>     >
>     
https://github.com/ThomasWaldmann/attic/commit/e7673c4af9f6b6cd2f20d191917b4c45fa65f605
>     >
>     > Note: this is a MINIMAL patch for people who rather want to stay on
>     > attic 0.14 or master branch.
>     >
>     > The code in merge-all repo has way better compression selection AND
>     > different and faster algorithms.
>     Please stop telling people to test your merge-all branch without
>     making
>     sure they understand that it it's very unlikely that the
>     repositories it
>     touches will be compatible with any current or future version of
>     Attic.
>
>
> Maybe you've missed some of the posts, but this was for people trying
> to find the data corruption issues that they have seen when backing up
> terabytes of data with attic release code - which takes some days per
> test run to complete. I just wanted to help them so this can be found
> faster.
>
> I always pointed out that the code in this repo is experimental / for
> tests only, so there shouldn't be any problems.
>
To me neither the name "merge-all" or this particular post makes it
especially clear that the purpose of that branch is test-only and
produces incompatible output. I'm just afraid people might stumble upon
posts like these and start backing up their data with this code and only
to a lot later realize they are not able to restore their backups using
Attic...

For this kind of software backwards compatibility and stability is key
and something I take very seriously. For maintainability and to keep the
code complexity down the number of format changes need to be kept as low
as possible. Ideally Attic should always stay compatible with all
previous versions.

I've been planning the next (payload) format change for a while and I
like to include not only support for more compression algorithms, but
also make some encryption related changes. The new format will be
available for testing and review when it's ready, but as it looks now it
will not be compatible with the changes on the TW's "merge-all" branch.

/ Jonas

Re: [attic] quick compression patch for attic master / 0.14

From:
Date:
2015-04-17 @ 06:11
> To me neither the name "merge-all" or this particular post makes it
> especially clear that the purpose of that branch is test-only and
> produces incompatible output. I'm just afraid people might stumble upon
> posts like these and start backing up their data with this code and only
> to a lot later realize they are not able to restore their backups using
> Attic...

Well I can only speak for myself and I understood very well, that 
merge-all is "experimental" code.
Though you are right that the merge-all should contain a warning, that 
repositories touched/created with it will be incompatible with upstream 
code.

I would even go so far as to renaming the main executable 
(attic-experimental).

Nevertheless the ideas (and implementation) are well worth taking a look 
at, because it also shows the potential of attic's future. Maybe the 
implementation is not as future-proof as you (and the users) expect, but 
it's a draft and it shows what's possible.

> For this kind of software backwards compatibility and stability is key
> and something I take very seriously. For maintainability and to keep the
> code complexity down the number of format changes need to be kept as low
> as possible. Ideally Attic should always stay compatible with all
> previous versions.

You're right. Backup is a very traditional business. That's the reason why 
'tar' is still the most used archiver in the *ix world. Because it works, 
and nearly every machine can read gzipped tar.
At least once attic reaches the "1.0" it should behave the same. That of 
course calls for a format that's flexible enough that "extensions" can 
safely be ignored by client versions (or variants) that don't support them 
(like the GNU extensions to tar).

> I've been planning the next (payload) format change for a while and I
> like to include not only support for more compression algorithms, but
> also make some encryption related changes. The new format will be
> available for testing and review when it's ready, but as it looks now it
> will not be compatible with the changes on the TW's "merge-all" branch.

I don't think many people expected upstream to just merge the changes (at 
least not me). My hope is that you can get some benefit from the proposed 
changes and can integrate them into upstream. As far as I understood, the 
other branch ("merge") only contains compatible changes and bugfixes. That 
way everybody wins.

Best Regards
 Heiko

Re: [attic] quick compression patch for attic master / 0.14

From:
Leo Famulari
Date:
2015-04-17 @ 19:26
On Fri, Apr 17, 2015 at 08:11:12AM +0200, heiko.helmle@horiba.com wrote:
> I would even go so far as to renaming the main executable 
> (attic-experimental).

I think this is a good idea. It's relatively unlikely that a novice user
will be installing attic for production backups from a 3rd party github
repository, especially considering that attic is in the repos of several
major Linux distributions (although not the Red Hat family). But at
least this would help mitigate the circumstance.

Re: [attic] quick compression patch for attic master / 0.14

From:
Jonas Borgström
Date:
2015-04-20 @ 11:51
On 17/04/15 21:26, Leo Famulari wrote:
> On Fri, Apr 17, 2015 at 08:11:12AM +0200, heiko.helmle@horiba.com wrote:
>> I would even go so far as to renaming the main executable
>> (attic-experimental).
>
> I think this is a good idea. It's relatively unlikely that a novice user
> will be installing attic for production backups from a 3rd party github
> repository, especially considering that attic is in the repos of several
> major Linux distributions (although not the Red Hat family). But at
> least this would help mitigate the circumstance.

I don't want to make a big deal about this or start a fight. I'm just 
concerned about new users potentially running incompatible and 
unsupported code without fully understanding what that means.

Renaming the branch might be a good thing. But I just found out and 
realized two different things that concern me more:

The TW's merge-all branch in question is no longer hosted in his 
personal github account. Instead it's located in an 
official-looking-but-unofficial "Attic" organization account on github:

https://github.com/attic

Thomas, can you please move that branch back to your own personal 
account again to avoid confusion?

Speaking of potential confusion. I also noticed that Arch Linux now 
packages this branch and the maintainer is promoting it as:

"""
Package updated for newly released bugfix version 0.15. Also, for those 
of you interested in a more active branch, attic-merge-all-git has been 
uploaded to the AUR.
"""

https://aur.archlinux.org/packages/attic/
https://aur.archlinux.org/packages/attic-merge-all-git/

Still see no risk for confusion?


That said, Thomas I know that you and others are concerned about the PR 
backlog and frustrated by the slow review speed. Unfortunately I'm still 
not ready to stop being gatekeeper for all changes going into the code base.
I have a clear (but probably poorly communicated) vision of what I want 
Attic to become and what features I think it should have (and not have), 
and how they should be implemented.
Hopefully with time I'll become better at communicating my ideas and 
vision and I'll become more comfortable with delegating more 
responsibilities.


/ Jonas

Re: [attic] quick compression patch for attic master / 0.14

From:
Jean Jordaan
Date:
2015-04-21 @ 10:04
On Mon, Apr 20, 2015 at 6:51 PM, Jonas Borgström <jonas@borgstrom.se> wrote:
[...]
>
> The TW's merge-all branch in question is no longer hosted in his
> personal github account. Instead it's located in an
> official-looking-but-unofficial "Attic" organization account on github:
>
> https://github.com/attic
>
> Thomas, can you please move that branch back to your own personal
> account again to avoid confusion?

How about adopting Hintjens's "Pedantic Code Construction Contract (PC3)"?
  http://hintjens.com/blog:23

In particular it states:

"""
To work on an issue, a Contributor SHALL fork the project repository
and then work on their forked repository.
To submit a patch, a Contributor SHALL create a Platform pull request
back to the project.
A Contributor SHALL NOT commit changes directly to the project.
"""

-- 
jean                                              . .. .... //\\\oo///\\

Re: [attic] quick compression patch for attic master / 0.14

From:
Randy Syring
Date:
2015-04-20 @ 14:07
> Speaking of potential confusion. I also noticed that Arch Linux now
> packages this branch and the maintainer is promoting it as:
>
> """
> Package updated for newly released bugfix version 0.15. Also, for those
> of you interested in a more active branch, attic-merge-all-git has been
> uploaded to the AUR.
> """
>
> https://aur.archlinux.org/packages/attic/
> https://aur.archlinux.org/packages/attic-merge-all-git/
>
> Still see no risk for confusion?

FWIW, I see the concern.  What if the branch was renamed 
"attic-experimental" and the readme changed to indicate what the branch 
is for with a very large warning about incompatibility with 
non-experimental attic?

>
>
> That said, Thomas I know that you and others are concerned about the PR
> backlog and frustrated by the slow review speed. Unfortunately I'm still
> not ready to stop being gatekeeper for all changes going into the code base.

FWIW, I don't think Thomas or anyone else is asking you to stop being 
gatekeeper.  You may, however, want to consider working more closely 
with Thomas since he seems to have some energy and interest in 
contributing to the project.  I know that one of the hardest changes for 
an OSS developer is to go from a one man show to having others work on 
the code.  It doesn't mean you lose control or quality has to suffer.  
It does mean you grow into realizing the project can still remain 
quality without your needing to write or review every line of code.

> I have a clear (but probably poorly communicated) vision of what I want
> Attic to become and what features I think it should have (and not have),
> and how they should be implemented.
> Hopefully with time I'll become better at communicating my ideas and
> vision and I'll become more comfortable with delegating more
> responsibilities.

Attic doesn't have to become a community project.  I remember my reading 
somewhere that you started this as a personal project to experiment with 
technologies you were interested in.  It can remain that if you want it 
to and I hope no one would fault you.  It's your baby, or second baby, 
now, as the case may be.  :)

On the other hand, there is a lot of potential and some interest in 
Attic, and being more communicative  and open to contributions would 
allow attic to mature more quickly.  I would love to use Attic but do 
want to see how some of this project/maintainer stuff works itself out 
before I'm comfortable using it in production.  The stability and 
activity of the project is really important to longevity and backup 
systems and processes are big commitments.

*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/