librelist archives

« back to archive

New user questions

New user questions

From:
Jonathan
Date:
2015-02-24 @ 18:40
Hi,

I recently learnt about Attic and after using it for a few days I'm really
impressed! Great project!

After using attic I've compiled a couple of questions.

- How does attic handle open files?
I assume that attic will backup the opened files as they are on disk,
so any changes not saved will not be in the backup.

- I'm currently using a key file without a passphrase. I'd like to use a 
passphrase, but
I don't know if there is any added security as I'll have to specify it as 
an environment
variable. Attic doesn't use asymmetric encryption, so I'd like to know how other
security conscious users are dealing with encryption.

- What's the best way to check if `attic check` fails in a script?
At the moment my script looks like this,

```
#!/bin/sh
REPOSITORY=/attic/somehost.attic

attic create --stats \
  $REPOSITORY::$(date +%Y-%m-%d-%H:%M:%S) \
  /something

echo "****** Checking repository integrity ******"
attic check --repository-only $REPOSITORY

echo "***** Pruning old archives ******"
attic prune -v $REPOSITORY --keep-hourly=24 --keep-daily=7 --keep-weekly=4
--keep-monthly=6
```

Ideally if `attic check` fails I don't want `attic prune` to run.
What's the best way to check for failure on integrity checking?


Thanks.
Jonathan

Re: [attic] New user questions

From:
Date:
2015-02-25 @ 07:08
> - How does attic handle open files?
> I assume that attic will backup the opened files as they are on disk,
> so any changes not saved will not be in the backup.

yes, and it also doesn't handle files changing during backup. So a file 
changing - while backup is reading it - will often result in a corrupted 
backup.

So ideally two things should happen:
* you might want to consider making a snapshot of your filesystem (btrfs 
or lvm have tools for that) and backup the snapshot
* attic could gain the feature of detecting file changes during backup and 
spit out a warning (like tar does)

> - What's the best way to check if `attic check` fails in a script?
> At the moment my script looks like this,
...snip....
> Ideally if `attic check` fails I don't want `attic prune` to run.
> What's the best way to check for failure on integrity checking?

easiest way: if you put "set -e" in the script, it will stop on any failed 
command. If "attic check" sets an error code !=0 your script would stop 
after the check.

Best Regards
 Heiko

Sv: [attic] New user questions

From:
Petter Gunnerud
Date:
2015-02-26 @ 14:17
>> - What's the best way to check if `attic check` fails in a script?
>...snip.... 
>> Ideally if `attic check` fails I don't want `attic
>prune` to run.
>> What's the best way to check for failure on integrity checking? 
>
>easiest way: if you put "set -e" in the
>script, it will stop on any failed command. If "attic check"
>sets an error code !=0 your script would stop after the check. 


wouldn't a simple 


# attic check && attic prune

do the trick?

PG

Re: Sv: [attic] New user questions

From:
Jonathan
Date:
2015-02-27 @ 17:14
I thought maybe I would have to write an if statement to check the output of
`attic check` looking for "Repository check complete, no problems found.".

Something like,

```
integrity_check=$(attic check --repository-only $REPOSITORY)

if integrity_check == "Repository check complete, no problems found."; then
  attic prune

  // rsync ....
else
  exit 1
fi
```

----- Original Message -----
From: "Petter Gunnerud" <pgspm@yahoo.no>
To: attic@librelist.com
Sent: Thursday, February 26, 2015 3:17:27 PM
Subject: Sv: [attic] New user questions


>> - What's the best way to check if `attic check` fails in a script?
>...snip.... 
>> Ideally if `attic check` fails I don't want `attic
>prune` to run.
>> What's the best way to check for failure on integrity checking? 
>
>easiest way: if you put "set -e" in the
>script, it will stop on any failed command. If "attic check"
>sets an error code !=0 your script would stop after the check. 


wouldn't a simple 


# attic check && attic prune

do the trick?

PG

Re: [attic] New user questions

From:
Jonathan
Date:
2015-02-27 @ 17:00
Hi Heiko,

> yes, and it also doesn't handle files changing during backup. So a file 
changing - while backup is reading it - will often result in a corrupted 
backup. 

> So ideally two things should happen: 
> * you might want to consider making a snapshot of your filesystem (btrfs
or lvm have tools for that) and backup the snapshot 
> * attic could gain the feature of detecting file changes during backup 
and spit out a warning (like tar does) 

This is a bummer :(

I didn't expect attic to corrupt the backups if files are open. I was 
planning on using attic as an alternative to arq backup on Mac OS X
as it is 100% open source and I was hoping to do continuous hourly backups
of my /Users/user directory similar to arq. However,
if this results in corrupted backups I'll have to rethink my backup 
strategy using attic.



----- Original Message -----
From: "heiko helmle" <heiko.helmle@horiba.com>
To: attic@librelist.com
Sent: Wednesday, February 25, 2015 8:08:28 AM
Subject: Re: [attic] New user questions

> - How does attic handle open files? 
> I assume that attic will backup the opened files as they are on disk, 
> so any changes not saved will not be in the backup. 

yes, and it also doesn't handle files changing during backup. So a file 
changing - while backup is reading it - will often result in a corrupted 
backup. 

So ideally two things should happen: 
* you might want to consider making a snapshot of your filesystem (btrfs 
or lvm have tools for that) and backup the snapshot 
* attic could gain the feature of detecting file changes during backup and
spit out a warning (like tar does) 

> - What's the best way to check if `attic check` fails in a script? 
> At the moment my script looks like this, 
...snip.... 
> Ideally if `attic check` fails I don't want `attic prune` to run. 
> What's the best way to check for failure on integrity checking? 

easiest way: if you put "set -e" in the script, it will stop on any failed
command. If "attic check" sets an error code !=0 your script would stop 
after the check. 

Best Regards 
Heiko 

Re: [attic] New user questions

From:
Dan Christensen
Date:
2015-02-27 @ 20:26
Jonathan <dev@puppeq.com> writes:

> I didn't expect attic to corrupt the backups if files are open. 

In addition to what others have added, it is not open files that
are a problem, but files that change exactly when attic tries
to back them up.  I suspect this is very unlikely to cause a 
problem in practice, and is no different from how most backup
programs work.

Dan

Re: [attic] New user questions

From:
Date:
2015-03-02 @ 06:38
> 
> > I didn't expect attic to corrupt the backups if files are open. 
> 
> In addition to what others have added, it is not open files that
> are a problem, but files that change exactly when attic tries
> to back them up.  I suspect this is very unlikely to cause a 
> problem in practice, and is no different from how most backup
> programs work.

Well it highly depends on the type of (changing) file that is backed up.
* log  files and files that are usually only appended to - no problem
* databases files, VM images and so on WILL suffer if you don't ensure 
consistency with some mechanism
  (many databases can be put into "log only" mode during backup and VMs 
would need to be paused (or snapshotted)
* for normal files (documents) it depends - most times it's not a problem 
but you can get unlucky

that's why i always recommend backing up a snapshot. Unfortunately I don't 
think how (if) that works on OSX.

Best Regards
 Heiko

Re: [attic] New user questions

From:
Henrik Christensen
Date:
2015-03-02 @ 07:39
The best way to make sure all files and database are intact would always
be to make a snapshot of the system first and then take a backup of the
snapshot, or using proprietary backup software. I use EXT4 with LVM on my
servers, a snapshot takes under a second, then I backup the data from the
snapshot and remove the snapshot again right after. With databases you
will have to make a backup with a database tool before the snapshot, but
that is easily done with a scripts. There is no to very little performance
difference on servers running with LVM, I know others would say there is
but I have not had any problems with performance as long as you delete the
snapshots again. You could also use, like it have been noted  before ZFS
or BTRFS witch both are now very reliable files systems, with ZFS as
production ready for Linux. Both files systems are highly advanced and
have internal snapshot systems that are better than LVM.

 

Someone correct me if I am wrong, but I would believe this is the 'best
practice' way to do backup.

 

Best regards

Henrik

 

From: attic@librelist.com [mailto:attic@librelist.com] On Behalf Of
heiko.helmle@horiba.com
Sent: 2. marts 2015 07:39
To: attic@librelist.com
Subject: Re: [attic] New user questions

 


> 
> > I didn't expect attic to corrupt the backups if files are open. 
> 
> In addition to what others have added, it is not open files that
> are a problem, but files that change exactly when attic tries
> to back them up.  I suspect this is very unlikely to cause a 
> problem in practice, and is no different from how most backup
> programs work. 

Well it highly depends on the type of (changing) file that is backed up. 
* log  files and files that are usually only appended to - no problem 
* databases files, VM images and so on WILL suffer if you don't ensure
consistency with some mechanism 
  (many databases can be put into "log only" mode during backup and VMs
would need to be paused (or snapshotted) 
* for normal files (documents) it depends - most times it's not a problem
but you can get unlucky 

that's why i always recommend backing up a snapshot. Unfortunately I don't
think how (if) that works on OSX. 

Best Regards 
 Heiko

Re: [attic] New user questions

From:
Yuri D'Elia
Date:
2015-02-27 @ 17:15
On 02/27/2015 06:00 PM, Jonathan wrote:
>> So ideally two things should happen: * you might want to consider
>> making a snapshot of your filesystem (btrfs or lvm have tools for
>> that) and backup the snapshot * attic could gain the feature of
>> detecting file changes during backup and spit out a warning (like
>> tar does)
> 
> This is a bummer :(

I would be surprised if any backup system could guarantee file
consistency without active cooperation or mandatory locking. attic is no
better or worse than tar (or any other backup program in this field).

> I didn't expect attic to corrupt the backups if files are open. I was
> planning on using attic as an alternative to arq backup on Mac OS X 
> as it is 100% open source and I was hoping to do continuous hourly
> backups of my /Users/user directory similar to arq. However, if this
> results in corrupted backups I'll have to rethink my backup strategy
> using attic.

I would be surprised if arq did actually guarantee file-level integrity.