librelist archives

« back to archive

feature request: archive stdin or fifo

feature request: archive stdin or fifo

From:
Matthias Böttcher
Date:
2014-01-02 @ 14:55
Hi,

first of all thanks to Jonas for developing attic. I am using attic
since 4 weeks on Debian and that's a great tool for doing backup with
deduplication. I am excited about the dedup ratio (original size vs.
unique data) achieved in my single repository although throwing in
different backup sources (tar, mysqldump) from different machines.

I would like to send a feature request here because i dont't have a
github account:
Like (all) the other Unix/Linux tools attic should be able to open
stdin or a fifo for backup and dedup the stream until eof. And of
course it should be possible to extract these archives to stdout.

Greetings,
Matthias.

Re: [attic] feature request: archive stdin or fifo

From:
Jonas Borgström
Date:
2014-01-03 @ 12:28
On 2014-01-02 15:55, Matthias Böttcher wrote:
> Hi,
>
> first of all thanks to Jonas for developing attic. I am using attic
> since 4 weeks on Debian and that's a great tool for doing backup with
> deduplication. I am excited about the dedup ratio (original size vs.
> unique data) achieved in my single repository although throwing in
> different backup sources (tar, mysqldump) from different machines.
Thanks, it's always nice to hear from users!

>
> I would like to send a feature request here because i dont't have a
> github account:
> Like (all) the other Unix/Linux tools attic should be able to open
> stdin or a fifo for backup and dedup the stream until eof. And of
> course it should be possible to extract these archives to stdout.
That sounds like a useful feature. I've created a github issue for it here:

https://github.com/jborg/attic/issues/22

Is myqldump your main use case for this?

/ Jonas

Re: [attic] feature request: archive stdin or fifo

From:
Matthias Böttcher
Date:
2014-01-03 @ 16:20
Hi Jonas,

2014/1/3 Jonas Borgström <jonas@borgstrom.se>:

> That sounds like a useful feature. I've created a github issue for it here:
> https://github.com/jborg/attic/issues/22

Thanks.

> Is myqldump your main use case for this?

The main use case for me at the moment is to archive mysqldumps and
prebuild tars with attic.

If attic could read and archive stdin or fifo (or device files) until
eof, I would use attic to archive
- mysqldumps in a pipe
- whole file system partitions, esp. ntfs ("every byte of a file system")
- the output of ntfsclone in a pipe
- ... and anything the can be send to a pipe...

Greetings,
Matthias

Re: [attic] feature request: archive stdin or fifo

From:
Petros Moisiadis
Date:
2014-01-05 @ 11:54
On 01/03/2014 06:20 PM, Matthias Böttcher wrote:
> Hi Jonas,
>
> 2014/1/3 Jonas Borgström <jonas@borgstrom.se>:
>
>> That sounds like a useful feature. I've created a github issue for it here:
>> https://github.com/jborg/attic/issues/22
> Thanks.
>
>> Is myqldump your main use case for this?
> The main use case for me at the moment is to archive mysqldumps and
> prebuild tars with attic.
>
> If attic could read and archive stdin or fifo (or device files) until
> eof, I would use attic to archive
> - mysqldumps in a pipe
> - whole file system partitions, esp. ntfs ("every byte of a file system")
> - the output of ntfsclone in a pipe
> - ... and anything the can be send to a pipe...
>
> Greetings,
> Matthias

The ability to open some files specially is indeed a very useful feature.
Another use case is to take consistent backups of live systems based on
LVM snapshots, by first creating the snapshots and then have attic open
them as block device files.

As an intuitive (imho) interface to this feature, I suggest to use path
prefixes when giving paths to the create and extract commands. So, the
format of paths would be: "[prefix:]PATH", where 'prefix:' could be one
of "blockdev:", "fifo:" and "stdin:". A "blockdev:" prefix would
instruct attic to open the path as block device. A "fifo:" prefix would
instruct attic to open the path as a fifo pipe. A standalone "stdin:"
prefix (without a path) would instruct attic to add data coming from
stdin to the archive. For example,|

|

    |$ attic create /path/to/attic/repo::archivename
    ||blockdev||:||/dev/vgname/lvmsnapshot||
    ||$ attic extract ||/path/to/attic/repo::archivename
    ||blockdev||:||dev/vgname/lvmsnapshot||
    |

Although that would be highly uncommon, if a literal "prefix:PATH" path
does exist at the current working directory, it should not be treated as
a special file.
||
||Also the extract command could be expanded to take one extra optional
argument: "-m --mode" (or something similar) with possible values:
"relative", "stdout" and "inplace".
In "relative" mode, the default one, attic would extract in the same
fashion as it does now: to a path relative to the current working
directory. In "inplace" mode, attic would extract at the full path,
possibly overwriting existing files (this could prompt for confirmation
or do its job quietly if a --force-overwrite option is passed). In
"stdout" mode, attic would extract to standard output.

For example, the following attic invocations, would create an archive
from mysqldump piped to attic's stdin and then restore the dump with
attic extracting archive's stdin to stdout and piping to mysql

    |$ ||mysqldump dbname | attic create
    /path/to/attic/repo::archivename ||stdin:||||
    ||||$ ||attic extract -m stdout ||/path/to/attic/repo::archivename
    ||stdin||||:||||| mysql dbname|

||What do you think?

Re: [attic] feature request: archive stdin or fifo

From:
Jonas Borgström
Date:
2014-01-05 @ 13:58
On 2014-01-05 12:54, Petros Moisiadis wrote:
> On 01/03/2014 06:20 PM, Matthias Böttcher wrote:
>> Hi Jonas,
>>
>> 2014/1/3 Jonas Borgström <jonas@borgstrom.se>:
>>
>>> That sounds like a useful feature. I've created a github issue for it here:
>>> https://github.com/jborg/attic/issues/22
>> Thanks.
>>
>>> Is myqldump your main use case for this?
>> The main use case for me at the moment is to archive mysqldumps and
>> prebuild tars with attic.
>>
>> If attic could read and archive stdin or fifo (or device files) until
>> eof, I would use attic to archive
>> - mysqldumps in a pipe
>> - whole file system partitions, esp. ntfs ("every byte of a file system")
>> - the output of ntfsclone in a pipe
>> - ... and anything the can be send to a pipe...
>>
>> Greetings,
>> Matthias
> 
> The ability to open some files specially is indeed a very useful feature.
> Another use case is to take consistent backups of live systems based on
> LVM snapshots, by first creating the snapshots and then have attic open
> them as block device files.
> 
> As an intuitive (imho) interface to this feature, I suggest to use path
> prefixes when giving paths to the create and extract commands. So, the
> format of paths would be: "[prefix:]PATH", where 'prefix:' could be one
> of "blockdev:", "fifo:" and "stdin:". A "blockdev:" prefix would
> instruct attic to open the path as block device. A "fifo:" prefix would
> instruct attic to open the path as a fifo pipe. A standalone "stdin:"
> prefix (without a path) would instruct attic to add data coming from
> stdin to the archive. For example,|
> 
> |
> 
>     |$ attic create /path/to/attic/repo::archivename
>     ||blockdev||:||/dev/vgname/lvmsnapshot||
>     ||$ attic extract ||/path/to/attic/repo::archivename
>     ||blockdev||:||dev/vgname/lvmsnapshot||
>     |
> 
> Although that would be highly uncommon, if a literal "prefix:PATH" path
> does exist at the current working directory, it should not be treated as
> a special file.
> ||
> ||Also the extract command could be expanded to take one extra optional
> argument: "-m --mode" (or something similar) with possible values:
> "relative", "stdout" and "inplace".
> In "relative" mode, the default one, attic would extract in the same
> fashion as it does now: to a path relative to the current working
> directory. In "inplace" mode, attic would extract at the full path,
> possibly overwriting existing files (this could prompt for confirmation
> or do its job quietly if a --force-overwrite option is passed). In
> "stdout" mode, attic would extract to standard output.
> 
> For example, the following attic invocations, would create an archive
> from mysqldump piped to attic's stdin and then restore the dump with
> attic extracting archive's stdin to stdout and piping to mysql
> 
>     |$ ||mysqldump dbname | attic create
>     /path/to/attic/repo::archivename ||stdin:||||
>     ||||$ ||attic extract -m stdout ||/path/to/attic/repo::archivename
>     ||stdin||||:||||| mysql dbname|
> 
> ||What do you think?

It could work but seems unnecessarily complex.

I think we should try to mimic the way similar tool have solved this.
Unfortunately most/all comparable tools (tar, zip, duplicity, tarsnap,
rsync) do _not_ support this...

On the other hand, one common unix convention is to use '-' to specify
stdin on the command line.

So we could support the following syntax

mysqldump mydb | attic create repo::my-db-backup -

To tell attic to create a new type of "stdin/stream archive" that
contains the contents read from stdin and nothing else. It will not be
possible to mix stdin and regular files in the same archive.

Extracting a "stdin/stream archive" will always write the contents to
stdout:

attic extract repo::my-db-backup | mysql

To archive a block/char/fifo device we could do this:

cat /dev/vg0/mysnapshot | attic create repo::my-snapshot -

Or add options --raw (or a better name) to avoid having to use cat:

attic create repo::my-snapshot --raw /dev/vg0/mysnapshot

What do you think?

/ Jonas







Re: [attic] feature request: archive stdin or fifo

From:
Petros Moisiadis
Date:
2014-01-05 @ 20:00
On 01/05/2014 03:58 PM, Jonas Borgström wrote:
> On 2014-01-05 12:54, Petros Moisiadis wrote:
>> On 01/03/2014 06:20 PM, Matthias Böttcher wrote:
>>> Hi Jonas,
>>>
>>> 2014/1/3 Jonas Borgström <jonas@borgstrom.se>:
>>>
>>>> That sounds like a useful feature. I've created a github issue for it here:
>>>> https://github.com/jborg/attic/issues/22
>>> Thanks.
>>>
>>>> Is myqldump your main use case for this?
>>> The main use case for me at the moment is to archive mysqldumps and
>>> prebuild tars with attic.
>>>
>>> If attic could read and archive stdin or fifo (or device files) until
>>> eof, I would use attic to archive
>>> - mysqldumps in a pipe
>>> - whole file system partitions, esp. ntfs ("every byte of a file system")
>>> - the output of ntfsclone in a pipe
>>> - ... and anything the can be send to a pipe...
>>>
>>> Greetings,
>>> Matthias
>> The ability to open some files specially is indeed a very useful feature.
>> Another use case is to take consistent backups of live systems based on
>> LVM snapshots, by first creating the snapshots and then have attic open
>> them as block device files.
>>
>> As an intuitive (imho) interface to this feature, I suggest to use path
>> prefixes when giving paths to the create and extract commands. So, the
>> format of paths would be: "[prefix:]PATH", where 'prefix:' could be one
>> of "blockdev:", "fifo:" and "stdin:". A "blockdev:" prefix would
>> instruct attic to open the path as block device. A "fifo:" prefix would
>> instruct attic to open the path as a fifo pipe. A standalone "stdin:"
>> prefix (without a path) would instruct attic to add data coming from
>> stdin to the archive. For example,|
>>
>> |
>>
>>     |$ attic create /path/to/attic/repo::archivename
>>     ||blockdev||:||/dev/vgname/lvmsnapshot||
>>     ||$ attic extract ||/path/to/attic/repo::archivename
>>     ||blockdev||:||dev/vgname/lvmsnapshot||
>>     |
>>
>> Although that would be highly uncommon, if a literal "prefix:PATH" path
>> does exist at the current working directory, it should not be treated as
>> a special file.
>> ||
>> ||Also the extract command could be expanded to take one extra optional
>> argument: "-m --mode" (or something similar) with possible values:
>> "relative", "stdout" and "inplace".
>> In "relative" mode, the default one, attic would extract in the same
>> fashion as it does now: to a path relative to the current working
>> directory. In "inplace" mode, attic would extract at the full path,
>> possibly overwriting existing files (this could prompt for confirmation
>> or do its job quietly if a --force-overwrite option is passed). In
>> "stdout" mode, attic would extract to standard output.
>>
>> For example, the following attic invocations, would create an archive
>> from mysqldump piped to attic's stdin and then restore the dump with
>> attic extracting archive's stdin to stdout and piping to mysql
>>
>>     |$ ||mysqldump dbname | attic create
>>     /path/to/attic/repo::archivename ||stdin:||||
>>     ||||$ ||attic extract -m stdout ||/path/to/attic/repo::archivename
>>     ||stdin||||:||||| mysql dbname|
>>
>> ||What do you think?
> It could work but seems unnecessarily complex.
>
> I think we should try to mimic the way similar tool have solved this.
> Unfortunately most/all comparable tools (tar, zip, duplicity, tarsnap,
> rsync) do _not_ support this...
>
> On the other hand, one common unix convention is to use '-' to specify
> stdin on the command line.
>
> So we could support the following syntax
>
> mysqldump mydb | attic create repo::my-db-backup -
>
> To tell attic to create a new type of "stdin/stream archive" that
> contains the contents read from stdin and nothing else. It will not be
> possible to mix stdin and regular files in the same archive.
>
> Extracting a "stdin/stream archive" will always write the contents to
> stdout:
>
> attic extract repo::my-db-backup | mysql
>
> To archive a block/char/fifo device we could do this:
>
> cat /dev/vg0/mysnapshot | attic create repo::my-snapshot -
>
> Or add options --raw (or a better name) to avoid having to use cat:
>
> attic create repo::my-snapshot --raw /dev/vg0/mysnapshot
>
> What do you think?
>
> / Jonas
>

I am not that keen of the idea to have one archive per special file and
have to invoke attic several times for a number of special files. For
instance, I would not feel very comfortable with having to invoke attic
separately for each block device on a virtualization server.

So, I would suggest the implementation of two new archive types:
1) The 'stdin' archive type that is created when reading from stdin with
'-'.
2) The 'raw' archive type that is created when reading from one or more
'raw' files with --raw option.
For example:
attic create repo::my-vms --raw /dev/vg0/vm1-snapshot
/dev/vg0/vm2-snapshot /dev/vg0/vm3-snapshot

Then, the extract operation would need to check for the correct
invocation syntax based on the archive type:
1) If extracting from a 'normal' archive it would behave as it behaves now
2) If extracting from a 'stdin' archive it would write data to stdout
and would not allow the specification of any path
3) If extracting from a 'raw' archive it would write data to stdout and
would only allow the specification of exactly one path

What do you think?

Re: [attic] feature request: archive stdin or fifo

From:
Jonas Borgström
Date:
2014-01-05 @ 20:32
On 2014-01-05 21:00, Petros Moisiadis wrote:
> On 01/05/2014 03:58 PM, Jonas Borgström wrote:
>> On 2014-01-05 12:54, Petros Moisiadis wrote:
>>> On 01/03/2014 06:20 PM, Matthias Böttcher wrote:
>>>> Hi Jonas,
>>>>
>>>> 2014/1/3 Jonas Borgström <jonas@borgstrom.se>:
>>>>
>>>>> That sounds like a useful feature. I've created a github issue for it here:
>>>>> https://github.com/jborg/attic/issues/22
>>>> Thanks.
>>>>
>>>>> Is myqldump your main use case for this?
>>>> The main use case for me at the moment is to archive mysqldumps and
>>>> prebuild tars with attic.
>>>>
>>>> If attic could read and archive stdin or fifo (or device files) until
>>>> eof, I would use attic to archive
>>>> - mysqldumps in a pipe
>>>> - whole file system partitions, esp. ntfs ("every byte of a file system")
>>>> - the output of ntfsclone in a pipe
>>>> - ... and anything the can be send to a pipe...
>>>>
>>>> Greetings,
>>>> Matthias
>>> The ability to open some files specially is indeed a very useful feature.
>>> Another use case is to take consistent backups of live systems based on
>>> LVM snapshots, by first creating the snapshots and then have attic open
>>> them as block device files.
>>>
>>> As an intuitive (imho) interface to this feature, I suggest to use path
>>> prefixes when giving paths to the create and extract commands. So, the
>>> format of paths would be: "[prefix:]PATH", where 'prefix:' could be one
>>> of "blockdev:", "fifo:" and "stdin:". A "blockdev:" prefix would
>>> instruct attic to open the path as block device. A "fifo:" prefix would
>>> instruct attic to open the path as a fifo pipe. A standalone "stdin:"
>>> prefix (without a path) would instruct attic to add data coming from
>>> stdin to the archive. For example,|
>>>
>>> |
>>>
>>>     |$ attic create /path/to/attic/repo::archivename
>>>     ||blockdev||:||/dev/vgname/lvmsnapshot||
>>>     ||$ attic extract ||/path/to/attic/repo::archivename
>>>     ||blockdev||:||dev/vgname/lvmsnapshot||
>>>     |
>>>
>>> Although that would be highly uncommon, if a literal "prefix:PATH" path
>>> does exist at the current working directory, it should not be treated as
>>> a special file.
>>> ||
>>> ||Also the extract command could be expanded to take one extra optional
>>> argument: "-m --mode" (or something similar) with possible values:
>>> "relative", "stdout" and "inplace".
>>> In "relative" mode, the default one, attic would extract in the same
>>> fashion as it does now: to a path relative to the current working
>>> directory. In "inplace" mode, attic would extract at the full path,
>>> possibly overwriting existing files (this could prompt for confirmation
>>> or do its job quietly if a --force-overwrite option is passed). In
>>> "stdout" mode, attic would extract to standard output.
>>>
>>> For example, the following attic invocations, would create an archive
>>> from mysqldump piped to attic's stdin and then restore the dump with
>>> attic extracting archive's stdin to stdout and piping to mysql
>>>
>>>     |$ ||mysqldump dbname | attic create
>>>     /path/to/attic/repo::archivename ||stdin:||||
>>>     ||||$ ||attic extract -m stdout ||/path/to/attic/repo::archivename
>>>     ||stdin||||:||||| mysql dbname|
>>>
>>> ||What do you think?
>> It could work but seems unnecessarily complex.
>>
>> I think we should try to mimic the way similar tool have solved this.
>> Unfortunately most/all comparable tools (tar, zip, duplicity, tarsnap,
>> rsync) do _not_ support this...
>>
>> On the other hand, one common unix convention is to use '-' to specify
>> stdin on the command line.
>>
>> So we could support the following syntax
>>
>> mysqldump mydb | attic create repo::my-db-backup -
>>
>> To tell attic to create a new type of "stdin/stream archive" that
>> contains the contents read from stdin and nothing else. It will not be
>> possible to mix stdin and regular files in the same archive.
>>
>> Extracting a "stdin/stream archive" will always write the contents to
>> stdout:
>>
>> attic extract repo::my-db-backup | mysql
>>
>> To archive a block/char/fifo device we could do this:
>>
>> cat /dev/vg0/mysnapshot | attic create repo::my-snapshot -
>>
>> Or add options --raw (or a better name) to avoid having to use cat:
>>
>> attic create repo::my-snapshot --raw /dev/vg0/mysnapshot
>>
>> What do you think?
>>
>> / Jonas
>>
> 
> I am not that keen of the idea to have one archive per special file and
> have to invoke attic several times for a number of special files. For
> instance, I would not feel very comfortable with having to invoke attic
> separately for each block device on a virtualization server.
> 
> So, I would suggest the implementation of two new archive types:
> 1) The 'stdin' archive type that is created when reading from stdin with
> '-'.
> 2) The 'raw' archive type that is created when reading from one or more
> 'raw' files with --raw option.
> For example:
> attic create repo::my-vms --raw /dev/vg0/vm1-snapshot
> /dev/vg0/vm2-snapshot /dev/vg0/vm3-snapshot
> 
> Then, the extract operation would need to check for the correct
> invocation syntax based on the archive type:
> 1) If extracting from a 'normal' archive it would behave as it behaves now
> 2) If extracting from a 'stdin' archive it would write data to stdout
> and would not allow the specification of any path
> 3) If extracting from a 'raw' archive it would write data to stdout and
> would only allow the specification of exactly one path

It seems a bit asymmetric to support archiving multiple devices with a
single "create" invocation but require multiple "extract" invocations to
restore them.

How about this instead:

Add a "--copy-devices" option to the create command that will instruct
attic to record the contents of all block devices encountered (in
addition to the major/minor numbers).

This would backup a bunch of snapshots:

attic create --copy-devices repo::my-vms /dev/vg0/*-snapshot

And this would restore them:

attic extract --write-devices repo::my-vms

The "--write-devices" option wouldn't be strictly needed but I think
it's better not to write to block devices unless the user explicitly
asks us to do so.

/ Jonas

Re: [attic] feature request: archive stdin or fifo

From:
Petros Moisiadis
Date:
2014-01-05 @ 20:58
On 01/05/2014 10:32 PM, Jonas Borgström wrote:
> On 2014-01-05 21:00, Petros Moisiadis wrote:
>> On 01/05/2014 03:58 PM, Jonas Borgström wrote:
>>> On 2014-01-05 12:54, Petros Moisiadis wrote:
>>>> On 01/03/2014 06:20 PM, Matthias Böttcher wrote:
>>>>> Hi Jonas,
>>>>>
>>>>> 2014/1/3 Jonas Borgström <jonas@borgstrom.se>:
>>>>>
>>>>>> That sounds like a useful feature. I've created a github issue for it here:
>>>>>> https://github.com/jborg/attic/issues/22
>>>>> Thanks.
>>>>>
>>>>>> Is myqldump your main use case for this?
>>>>> The main use case for me at the moment is to archive mysqldumps and
>>>>> prebuild tars with attic.
>>>>>
>>>>> If attic could read and archive stdin or fifo (or device files) until
>>>>> eof, I would use attic to archive
>>>>> - mysqldumps in a pipe
>>>>> - whole file system partitions, esp. ntfs ("every byte of a file system")
>>>>> - the output of ntfsclone in a pipe
>>>>> - ... and anything the can be send to a pipe...
>>>>>
>>>>> Greetings,
>>>>> Matthias
>>>> The ability to open some files specially is indeed a very useful feature.
>>>> Another use case is to take consistent backups of live systems based on
>>>> LVM snapshots, by first creating the snapshots and then have attic open
>>>> them as block device files.
>>>>
>>>> As an intuitive (imho) interface to this feature, I suggest to use path
>>>> prefixes when giving paths to the create and extract commands. So, the
>>>> format of paths would be: "[prefix:]PATH", where 'prefix:' could be one
>>>> of "blockdev:", "fifo:" and "stdin:". A "blockdev:" prefix would
>>>> instruct attic to open the path as block device. A "fifo:" prefix would
>>>> instruct attic to open the path as a fifo pipe. A standalone "stdin:"
>>>> prefix (without a path) would instruct attic to add data coming from
>>>> stdin to the archive. For example,|
>>>>
>>>> |
>>>>
>>>>     |$ attic create /path/to/attic/repo::archivename
>>>>     ||blockdev||:||/dev/vgname/lvmsnapshot||
>>>>     ||$ attic extract ||/path/to/attic/repo::archivename
>>>>     ||blockdev||:||dev/vgname/lvmsnapshot||
>>>>     |
>>>>
>>>> Although that would be highly uncommon, if a literal "prefix:PATH" path
>>>> does exist at the current working directory, it should not be treated as
>>>> a special file.
>>>> ||
>>>> ||Also the extract command could be expanded to take one extra optional
>>>> argument: "-m --mode" (or something similar) with possible values:
>>>> "relative", "stdout" and "inplace".
>>>> In "relative" mode, the default one, attic would extract in the same
>>>> fashion as it does now: to a path relative to the current working
>>>> directory. In "inplace" mode, attic would extract at the full path,
>>>> possibly overwriting existing files (this could prompt for confirmation
>>>> or do its job quietly if a --force-overwrite option is passed). In
>>>> "stdout" mode, attic would extract to standard output.
>>>>
>>>> For example, the following attic invocations, would create an archive
>>>> from mysqldump piped to attic's stdin and then restore the dump with
>>>> attic extracting archive's stdin to stdout and piping to mysql
>>>>
>>>>     |$ ||mysqldump dbname | attic create
>>>>     /path/to/attic/repo::archivename ||stdin:||||
>>>>     ||||$ ||attic extract -m stdout ||/path/to/attic/repo::archivename
>>>>     ||stdin||||:||||| mysql dbname|
>>>>
>>>> ||What do you think?
>>> It could work but seems unnecessarily complex.
>>>
>>> I think we should try to mimic the way similar tool have solved this.
>>> Unfortunately most/all comparable tools (tar, zip, duplicity, tarsnap,
>>> rsync) do _not_ support this...
>>>
>>> On the other hand, one common unix convention is to use '-' to specify
>>> stdin on the command line.
>>>
>>> So we could support the following syntax
>>>
>>> mysqldump mydb | attic create repo::my-db-backup -
>>>
>>> To tell attic to create a new type of "stdin/stream archive" that
>>> contains the contents read from stdin and nothing else. It will not be
>>> possible to mix stdin and regular files in the same archive.
>>>
>>> Extracting a "stdin/stream archive" will always write the contents to
>>> stdout:
>>>
>>> attic extract repo::my-db-backup | mysql
>>>
>>> To archive a block/char/fifo device we could do this:
>>>
>>> cat /dev/vg0/mysnapshot | attic create repo::my-snapshot -
>>>
>>> Or add options --raw (or a better name) to avoid having to use cat:
>>>
>>> attic create repo::my-snapshot --raw /dev/vg0/mysnapshot
>>>
>>> What do you think?
>>>
>>> / Jonas
>>>
>> I am not that keen of the idea to have one archive per special file and
>> have to invoke attic several times for a number of special files. For
>> instance, I would not feel very comfortable with having to invoke attic
>> separately for each block device on a virtualization server.
>>
>> So, I would suggest the implementation of two new archive types:
>> 1) The 'stdin' archive type that is created when reading from stdin with
>> '-'.
>> 2) The 'raw' archive type that is created when reading from one or more
>> 'raw' files with --raw option.
>> For example:
>> attic create repo::my-vms --raw /dev/vg0/vm1-snapshot
>> /dev/vg0/vm2-snapshot /dev/vg0/vm3-snapshot
>>
>> Then, the extract operation would need to check for the correct
>> invocation syntax based on the archive type:
>> 1) If extracting from a 'normal' archive it would behave as it behaves now
>> 2) If extracting from a 'stdin' archive it would write data to stdout
>> and would not allow the specification of any path
>> 3) If extracting from a 'raw' archive it would write data to stdout and
>> would only allow the specification of exactly one path
> It seems a bit asymmetric to support archiving multiple devices with a
> single "create" invocation but require multiple "extract" invocations to
> restore them.
>
> How about this instead:
>
> Add a "--copy-devices" option to the create command that will instruct
> attic to record the contents of all block devices encountered (in
> addition to the major/minor numbers).
>
> This would backup a bunch of snapshots:
>
> attic create --copy-devices repo::my-vms /dev/vg0/*-snapshot
>
> And this would restore them:
>
> attic extract --write-devices repo::my-vms
>
> The "--write-devices" option wouldn't be strictly needed but I think
> it's better not to write to block devices unless the user explicitly
> asks us to do so.
>
> / Jonas
>
>

That would work great if you want all devices in the archive to be
restored in-place, but will it be still possible to extract a single raw
file to some other place (via stdout redirection) ?

For example:
attic extract repo::my-vms /dev/vg0/vm1-snapshot  > /tmp/vm1-recover

Would the above work as expected?

Re: [attic] feature request: archive stdin or fifo

From:
Jonas Borgström
Date:
2014-01-05 @ 21:05
On 2014-01-05 21:58, Petros Moisiadis wrote:
> On 01/05/2014 10:32 PM, Jonas Borgström wrote:
>> On 2014-01-05 21:00, Petros Moisiadis wrote:
>>> On 01/05/2014 03:58 PM, Jonas Borgström wrote:
>>>> On 2014-01-05 12:54, Petros Moisiadis wrote:
>>>>> On 01/03/2014 06:20 PM, Matthias Böttcher wrote:
>>>>>> Hi Jonas,
>>>>>>
>>>>>> 2014/1/3 Jonas Borgström <jonas@borgstrom.se>:
>>>>>>
>>>>>>> That sounds like a useful feature. I've created a github issue for
it here:
>>>>>>> https://github.com/jborg/attic/issues/22
>>>>>> Thanks.
>>>>>>
>>>>>>> Is myqldump your main use case for this?
>>>>>> The main use case for me at the moment is to archive mysqldumps and
>>>>>> prebuild tars with attic.
>>>>>>
>>>>>> If attic could read and archive stdin or fifo (or device files) until
>>>>>> eof, I would use attic to archive
>>>>>> - mysqldumps in a pipe
>>>>>> - whole file system partitions, esp. ntfs ("every byte of a file system")
>>>>>> - the output of ntfsclone in a pipe
>>>>>> - ... and anything the can be send to a pipe...
>>>>>>
>>>>>> Greetings,
>>>>>> Matthias
>>>>> The ability to open some files specially is indeed a very useful feature.
>>>>> Another use case is to take consistent backups of live systems based on
>>>>> LVM snapshots, by first creating the snapshots and then have attic open
>>>>> them as block device files.
>>>>>
>>>>> As an intuitive (imho) interface to this feature, I suggest to use path
>>>>> prefixes when giving paths to the create and extract commands. So, the
>>>>> format of paths would be: "[prefix:]PATH", where 'prefix:' could be one
>>>>> of "blockdev:", "fifo:" and "stdin:". A "blockdev:" prefix would
>>>>> instruct attic to open the path as block device. A "fifo:" prefix would
>>>>> instruct attic to open the path as a fifo pipe. A standalone "stdin:"
>>>>> prefix (without a path) would instruct attic to add data coming from
>>>>> stdin to the archive. For example,|
>>>>>
>>>>> |
>>>>>
>>>>>     |$ attic create /path/to/attic/repo::archivename
>>>>>     ||blockdev||:||/dev/vgname/lvmsnapshot||
>>>>>     ||$ attic extract ||/path/to/attic/repo::archivename
>>>>>     ||blockdev||:||dev/vgname/lvmsnapshot||
>>>>>     |
>>>>>
>>>>> Although that would be highly uncommon, if a literal "prefix:PATH" path
>>>>> does exist at the current working directory, it should not be treated as
>>>>> a special file.
>>>>> ||
>>>>> ||Also the extract command could be expanded to take one extra optional
>>>>> argument: "-m --mode" (or something similar) with possible values:
>>>>> "relative", "stdout" and "inplace".
>>>>> In "relative" mode, the default one, attic would extract in the same
>>>>> fashion as it does now: to a path relative to the current working
>>>>> directory. In "inplace" mode, attic would extract at the full path,
>>>>> possibly overwriting existing files (this could prompt for confirmation
>>>>> or do its job quietly if a --force-overwrite option is passed). In
>>>>> "stdout" mode, attic would extract to standard output.
>>>>>
>>>>> For example, the following attic invocations, would create an archive
>>>>> from mysqldump piped to attic's stdin and then restore the dump with
>>>>> attic extracting archive's stdin to stdout and piping to mysql
>>>>>
>>>>>     |$ ||mysqldump dbname | attic create
>>>>>     /path/to/attic/repo::archivename ||stdin:||||
>>>>>     ||||$ ||attic extract -m stdout ||/path/to/attic/repo::archivename
>>>>>     ||stdin||||:||||| mysql dbname|
>>>>>
>>>>> ||What do you think?
>>>> It could work but seems unnecessarily complex.
>>>>
>>>> I think we should try to mimic the way similar tool have solved this.
>>>> Unfortunately most/all comparable tools (tar, zip, duplicity, tarsnap,
>>>> rsync) do _not_ support this...
>>>>
>>>> On the other hand, one common unix convention is to use '-' to specify
>>>> stdin on the command line.
>>>>
>>>> So we could support the following syntax
>>>>
>>>> mysqldump mydb | attic create repo::my-db-backup -
>>>>
>>>> To tell attic to create a new type of "stdin/stream archive" that
>>>> contains the contents read from stdin and nothing else. It will not be
>>>> possible to mix stdin and regular files in the same archive.
>>>>
>>>> Extracting a "stdin/stream archive" will always write the contents to
>>>> stdout:
>>>>
>>>> attic extract repo::my-db-backup | mysql
>>>>
>>>> To archive a block/char/fifo device we could do this:
>>>>
>>>> cat /dev/vg0/mysnapshot | attic create repo::my-snapshot -
>>>>
>>>> Or add options --raw (or a better name) to avoid having to use cat:
>>>>
>>>> attic create repo::my-snapshot --raw /dev/vg0/mysnapshot
>>>>
>>>> What do you think?
>>>>
>>>> / Jonas
>>>>
>>> I am not that keen of the idea to have one archive per special file and
>>> have to invoke attic several times for a number of special files. For
>>> instance, I would not feel very comfortable with having to invoke attic
>>> separately for each block device on a virtualization server.
>>>
>>> So, I would suggest the implementation of two new archive types:
>>> 1) The 'stdin' archive type that is created when reading from stdin with
>>> '-'.
>>> 2) The 'raw' archive type that is created when reading from one or more
>>> 'raw' files with --raw option.
>>> For example:
>>> attic create repo::my-vms --raw /dev/vg0/vm1-snapshot
>>> /dev/vg0/vm2-snapshot /dev/vg0/vm3-snapshot
>>>
>>> Then, the extract operation would need to check for the correct
>>> invocation syntax based on the archive type:
>>> 1) If extracting from a 'normal' archive it would behave as it behaves now
>>> 2) If extracting from a 'stdin' archive it would write data to stdout
>>> and would not allow the specification of any path
>>> 3) If extracting from a 'raw' archive it would write data to stdout and
>>> would only allow the specification of exactly one path
>> It seems a bit asymmetric to support archiving multiple devices with a
>> single "create" invocation but require multiple "extract" invocations to
>> restore them.
>>
>> How about this instead:
>>
>> Add a "--copy-devices" option to the create command that will instruct
>> attic to record the contents of all block devices encountered (in
>> addition to the major/minor numbers).
>>
>> This would backup a bunch of snapshots:
>>
>> attic create --copy-devices repo::my-vms /dev/vg0/*-snapshot
>>
>> And this would restore them:
>>
>> attic extract --write-devices repo::my-vms
>>
>> The "--write-devices" option wouldn't be strictly needed but I think
>> it's better not to write to block devices unless the user explicitly
>> asks us to do so.
>>
>> / Jonas
>>
>>
> 
> That would work great if you want all devices in the archive to be
> restored in-place, but will it be still possible to extract a single raw
> file to some other place (via stdout redirection) ?
>
> For example:
> attic extract repo::my-vms /dev/vg0/vm1-snapshot  > /tmp/vm1-recover
>
> Would the above work as expected?
>

Extracting a single file/device to stdout could be useful even when
working with regular files. Something like this could be a nice addition:

attic extract repo::my-archive --stdout some/file/or/device > /dev/whatever

/ Jonas


Re: [attic] feature request: archive stdin or fifo

From:
Matthias Böttcher
Date:
2014-01-05 @ 20:59
2014/1/5 Jonas Borgström <jonas@borgstrom.se>:

> attic extract --write-devices repo::my-vms
>
> The "--write-devices" option wouldn't be strictly needed but I think
> it's better not to write to block devices unless the user explicitly
> asks us to do so.

An explicit to give option for "attic extract ..." is required to
proctect the users for unwanted overwriting data of block devices. The
"--raw" option like in "attic create .." should do this.

Matthias.

Re: [attic] feature request: archive stdin or fifo

From:
Matthias Böttcher
Date:
2014-01-05 @ 20:46
2014/1/5 Jonas Borgström <jonas@borgstrom.se>:
> It could work but seems unnecessarily complex.
>
> I think we should try to mimic the way similar tool have solved this.
> Unfortunately most/all comparable tools (tar, zip, duplicity, tarsnap,
> rsync) do _not_ support this...
>
> On the other hand, one common unix convention is to use '-' to specify
> stdin on the command line.

That's what I want.

> So we could support the following syntax
>
> mysqldump mydb | attic create repo::my-db-backup -
>
> To tell attic to create a new type of "stdin/stream archive" that
> contains the contents read from stdin and nothing else. It will not be
> possible to mix stdin and regular files in the same archive.

Only one input type can be put in one archive.
The metadata for a archive from stdin will be empty except creation
date/time and original size.

The information about the archive source can not be stored. That takes
me to another feature request:
An optional comment field should be stored in the archive.

> Extracting a "stdin/stream archive" will always write the contents to
> stdout:
>
> attic extract repo::my-db-backup | mysql
>
> To archive a block/char/fifo device we could do this:
>
> cat /dev/vg0/mysnapshot | attic create repo::my-snapshot -
>
> Or add options --raw (or a better name) to avoid having to use cat:
>
> attic create repo::my-snapshot --raw /dev/vg0/mysnapshot
>
> What do you think?

An archive created with "--raw" from a special file brings the advantages:
- no pipe, error handling of source reading is done by attic
- the information about the archive source is stored in the archive

And if I really need to archive stdin:
  output_to_stdout_process | attic create repo::myarchive --raw -

Try to keep it small and simple.

Greetings
Matthias