librelist archives

« back to archive

initial 0.26 cache re-sync

initial 0.26 cache re-sync

From:
Thomas Waldmann
Date:
2015-09-20 @ 12:37
Hi, just wanted to post a note about this:

When you use 0.26.0 the first time, you might see a rather long
"synchronizing chunks cache".

This is due to the way the synchronisation works in 0.26+ and will only
take that long ONCE. Subsequent re-syncs will only fetch the new
archives it discovers in the repo (not all of them).

I changed the way the single-archive indexes are kept locally, they are
now single files in ~/.cache/borg/REPOID/chunks.archive.d/.

0.25 used a compressed tar archive "chunks.archive" - but dealing with
tar and (re-)compression took way too much time and cpu and the
compression did not work as great in practice as in my experiments.

If you have a slow connection to the repository and/or or a huge number
of archives, you can save some time by manually extracting your pre-0.26
chunks.archive to that location.

This manual procedure is OPTIONAL, if you do not do it, borg will kill
the compressed tar automatically and then fetch all single-archive
indexes from the (remote?) repo.

Make sure you have lots of disk space free in .cache/borg:

cd ~/.cache/borg/REPOID
mkdir chunks.archive.d
cd chunks.archive.d
tar xJvf ../chunks.archive
# if you have a older python, it might be also xjvf or xzvf.

# at the end, check that permissions/mode are as you see
# for the other files in .cache/borg:
cd ..
chown -R borg.borg chunks.archive.d
chmod -R go-rwX chunks.archive.d

# after successfully extracting the chunks.archive. remove it
# 0.26 does not use it any more (and would also remove it the
# first time a cache resync happens):
rm chunks.archive

In case something does not work, you can still kill chunks.archive.d and
let borg do it (slowly).

Cheers,

Thomas

-- 

GPG ID: FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

Re: [borgbackup] initial 0.26 cache re-sync

From:
Leo Famulari
Date:
2015-09-20 @ 17:32
I'm really glad to see this work being done. Am I right to assume that
the motivation behind this is to make it easier to backup multiple
systems to the same repository, achieving deduplication of "system"
files like /etc, /usr, /lib, et cetera?

On Sun, Sep 20, 2015, at 08:37, Thomas Waldmann wrote:
> Hi, just wanted to post a note about this:
> 
> When you use 0.26.0 the first time, you might see a rather long
> "synchronizing chunks cache".
> 
> This is due to the way the synchronisation works in 0.26+ and will only
> take that long ONCE. Subsequent re-syncs will only fetch the new
> archives it discovers in the repo (not all of them).
> 
> I changed the way the single-archive indexes are kept locally, they are
> now single files in ~/.cache/borg/REPOID/chunks.archive.d/.
> 
> 0.25 used a compressed tar archive "chunks.archive" - but dealing with
> tar and (re-)compression took way too much time and cpu and the
> compression did not work as great in practice as in my experiments.
> 
> If you have a slow connection to the repository and/or or a huge number
> of archives, you can save some time by manually extracting your pre-0.26
> chunks.archive to that location.
> 
> This manual procedure is OPTIONAL, if you do not do it, borg will kill
> the compressed tar automatically and then fetch all single-archive
> indexes from the (remote?) repo.
> 
> Make sure you have lots of disk space free in .cache/borg:
> 
> cd ~/.cache/borg/REPOID
> mkdir chunks.archive.d
> cd chunks.archive.d
> tar xJvf ../chunks.archive
> # if you have a older python, it might be also xjvf or xzvf.
> 
> # at the end, check that permissions/mode are as you see
> # for the other files in .cache/borg:
> cd ..
> chown -R borg.borg chunks.archive.d
> chmod -R go-rwX chunks.archive.d
> 
> # after successfully extracting the chunks.archive. remove it
> # 0.26 does not use it any more (and would also remove it the
> # first time a cache resync happens):
> rm chunks.archive
> 
> In case something does not work, you can still kill chunks.archive.d and
> let borg do it (slowly).
> 
> Cheers,
> 
> Thomas
> 
> -- 
> 
> GPG ID: FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
> 

Re: [borgbackup] initial 0.26 cache re-sync

From:
Thomas Waldmann
Date:
2015-09-20 @ 17:39
On 09/20/2015 07:32 PM, Leo Famulari wrote:
> I'm really glad to see this work being done. Am I right to assume that
> the motivation behind this is to make it easier to backup multiple
> systems to the same repository, achieving deduplication of "system"
> files like /etc, /usr, /lib, et cetera?

Well, this is nothing new, this has worked since long.
You can even move some user data files from one machine to another and
still have them deduplicated against the past backups.

New is that I am trying to make the cache resync faster.

Once borg notices that it's local cache is out of sync with the repo
(because another machine did a backup meanwhile), it needs to
resynchronize the cache.

---
GPG ID: FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393