librelist archives

« back to archive

summary of my last hacking streak

summary of my last hacking streak

From:
Antoine Beaupré
Date:
2015-10-17 @ 06:47
Hi,

This week, I took my laptop out to the countryside, without internet
(but with a HAM radio ;), to focus on peace and quiet and maybe work on
borg.

First off, I must apologise for the mess I created on the master
branch. I attempted to create a master branch in my own fork with all
the branches merged in and I mistakenly pushed that to the main repo,
which closed all the pull requests and put all my code live at once. I
have reverted the change and now only the --version pull request is
there.

I am at a loss at words at how to handle this... I guess I could rebase
those branches on top of the new messed up master, or we could reset the
master branch again. I am not sure what is the best way to proceed, but
I can't fix this cleanly myself, as I cannot force-push to master. :(

Again, apologies. Here is the rest of the mail i was hoping to send
happily an hour ago...

Pull requests
=============

I ended up with 6 feature branchs which i submitted as pull requests:

* i18n: translation support for borg
  https://github.com/borgbackup/borg/pull/287
* man-builder: failed attempt at writing a manpage formatter for
  argparse - the latter is such a tangled mess that i can't make head
  or tails of it, and i gave up: let's wait for click
  https://github.com/borgbackup/borg/issues/208
* no-inplace: operate on a "copy" during upgrades, the previous
  "inplace" mode is still available through `--inplace`
  https://github.com/borgbackup/borg/pull/280
* upgrader-index-fixes: fixes for the upgrader index problems, not
  sure how to create a unit test for this one, as repository.check()
  doesn't find the error (only create does)
  https://github.com/borgbackup/borg/pull/281
* verbosity: lots of tweaks to verbosity levels, --progress and
  --stats
  https://github.com/borgbackup/borg/pull/288
* version-arg: a --version flag! (this was mistakenly pushed directly to
  master, sorry, let me know if i need to revert)
* x-option, add -x to --do-not-cross-mountpoints setting
  https://github.com/borgbackup/borg/pull/282

Some of those conflict with each other: i18n and verbosity touch similar
areas, but the merge is fairly trivial. I have merged all the above (but
man-builder) into my master branch:

https://github.com/anarcat/borg/commits/master

(Update: this was mistakenly pushed to the main master branch, and then
reverted.)

Bug reports
===========

* tox seems to run the Attic test suite:
  https://github.com/borgbackup/borg/issues/283
* can't run tests offline: https://github.com/borgbackup/borg/issues/284
* lock.exclusive left behind:
  https://github.com/borgbackup/borg/issues/285

Other notes
===========

Upgrade notes
-------------

I ran a new conversion from Attic, this time on a fresh backup of my
laptop (i usually work on my workstation). The backup looked like:

    # attic create --exclude-caches -e /home/anarcat/mp3/ -e 
/home/pbuilder/ -e /home/anarcat/video/ -e /home/anarcat/iso/ -e  
/home/anarcat/books -e /home/anarcat/books-incoming -e .cache --stats 
--do-not-cross-mountpoints 
/media/anarcat/calyx/attic-angela::2015-10-15-first /
    Initializing cache...
    ------------------------------------------------------------------------------
    Archive name: 2015-10-15-first
    Archive fingerprint: 
5626925d54fae692e7ca5cb852e6629661d4ac5de00ad6b6494979980ec7822c
    Start time: Thu Oct 15 12:03:34 2015
    End time: Thu Oct 15 15:28:44 2015
    Duration: 3 hours 25 minutes 10.06 seconds
    Number of files: 847008
    
                           Original size      Compressed size    Deduplicated size
    This archive:               56.78 GB             38.82 GB             35.55 GB
    All archives:               56.78 GB             38.82 GB             35.55 GB
    ------------------------------------------------------------------------------

First borg run after upgrade was fast enough:

    ------------------------------------------------------------------------------
    Archive name: 2015-10-15-post-attic
    Archive fingerprint: 
44812971c502a25dcad61a7664eab3559405c6f1fd8b38bc50118f4619dbca72
    Start time: Thu Oct 15 18:36:02 2015
    End time: Thu Oct 15 18:49:15 2015
    Duration: 13 minutes 13.69 seconds
    Number of files: 847084

                           Original size      Compressed size    Deduplicated size
    This archive:               56.78 GB             38.85 GB             41.12 MB
    All archives:              113.57 GB             77.67 GB             35.59 GB

                           Unique chunks         Total chunks
    Chunk index:                 1430830              3255749
    ------------------------------------------------------------------------------

So it seems the chunks cache got reused properly this time, maybe
because the dataset is smaller.

weird integrity failure
-----------------------

When running tests with `-s`, we see some exceptions show up in the
log. It's a little confusing because we have the feeling that tests
fail because of the exception, while it seems the error is normal.

Failing test sample:

    
borg/testsuite/archiver.py::ArchiverCheckTestCase::test_missing_archive_item_chunk
Exception ignored in: <bound method Repository.__del__ of <Repository 
/tmp/tmpq8mq3loc/repository>>
    Traceback (most recent call last):
      File "/home/anarcat/src/borg/borg/repository.py", line 69, in __del__
        self.close()
      File "/home/anarcat/src/borg/borg/repository.py", line 155, in close
        self.lock.release()
      File "/home/anarcat/src/borg/borg/locking.py", line 281, in release
        self._roster.modify(EXCLUSIVE, REMOVE)
      File "/home/anarcat/src/borg/borg/locking.py", line 203, in modify
        elements.remove(self.id)
    KeyError: (('angela', 28267, 2564556544),)
    PASSED

Another:

    
borg/testsuite/archiver.py::RemoteArchiverTestCase::test_corrupted_repository
Segment entry checksum mismatch [segment 2, offset 8]
    Index object count mismatch. 147 != 0
    PASSED

Maybe those warnings are unimportant and should be ignored, but they
certainly make tests more confusing, and if those errors are expected,
maybe they shouldn't output those errors?

argv
----

It would be nice if borg would show up as "borg" instead of "python
borg" or whatever. bup does this through unpythonize_argv() which does
some pretty heavy memset() stuff to clear it up. there has to be a
better way (tm).

See: https://github.com/borgbackup/borg/issues/286

verbosity and logging
---------------------

it's harder to censor the unchanged files with stderr output: we
basically need to shove it back to stdout to grep it. that is
annoying.

basically, -v is way too verbose - it's unusable. also, as it is,
--progress doesn't show *any* progress until the file cache is loaded,
which is confusing, as we don't know if borg is waiting for a lock,
blocked or what.

the basic problem could be that --progress conflicts with the per-file
output. to resolve that, i have pushed the file listing down to the
DEBUG level for now, but the question of how to handle different
verbosity levels still stands

but i do feel confident now that the logging-refactor branch can be
merged in, and in fact most branches i have been working on have been
based on it...

oh, and --progress now knows about the terminal width, so it makes good
use of all the space to show file paths. it also shows the number of
files found so far.

--stats could similarly be improved as well...

format_file_sizes
-----------------

This could be reimplemented to cover more than terabytes, there's some
neat code from stackexchange for this...

http://stackoverflow.com/a/1094933/1174784

StableDict
----------

Why is that thing necessary, when we have OrderedDict?

public git repo was reset

From:
Antoine Beaupré
Date:
2015-10-18 @ 02:13
On 2015-10-17 02:47:09, Antoine Beaupré wrote:
> Hi,
>
> This week, I took my laptop out to the countryside, without internet
> (but with a HAM radio ;), to focus on peace and quiet and maybe work on
> borg.
>
> First off, I must apologise for the mess I created on the master
> branch. I attempted to create a master branch in my own fork with all
> the branches merged in and I mistakenly pushed that to the main repo,
> which closed all the pull requests and put all my code live at once. I
> have reverted the change and now only the --version pull request is
> there.
>
> I am at a loss at words at how to handle this... I guess I could rebase
> those branches on top of the new messed up master, or we could reset the
> master branch again. I am not sure what is the best way to proceed, but
> I can't fix this cleanly myself, as I cannot force-push to master. :(
>
> Again, apologies. Here is the rest of the mail i was hoping to send
> happily an hour ago...

So a followup on this, Thomas reset the git repository this
morning. Which means that if some of you pulled the repo between
last night (2015-10-17T02:47:09-0400) and today (12:11:23-0400), a git
remote update will force-update the remote branches, and you will need
to reset your local "master" branch, and rebase topic branches.

To reset the master branch:

git remote update
git checkout master
git reset master

To rebase a topic branch:

git remote update
git rebase upstream topic --onto origin/master

In the above, "topic" is the name of your topic branch and "upstream" is
the original branch point of your topic branch, basically where "master"
was when you created the branch. "git log --decorate" is useful to
figure that out. In fact, I use the following alias profusely for stuff
like this:

[alias]
    lg = log --graph --pretty=format:'%Cred%h%Creset 
%C(green)%G?%Creset%C(yellow)%d%Creset %s %Cgreen(%ar) 
%C(blue)<%an>%Creset' --abbrev-commit --date=relative

Sorry again for the trouble, and thanks to Thomas for fixing all of
this!

I have rerolled all my pull requests into new ones, because github
believes they were merged...

I would still welcome any feedback on the original post as well, of
course. :)

A.

-- 
Information is not knowledge. Knowledge is not wisdom.
Wisdom is not truth. Truth is not beauty.
Beauty is not love. Love is not music.
Music is the best.      - Frank Zappa