librelist archives

« back to archive

attic with davfs2

attic with davfs2

From:
Heinz Haderer
Date:
2014-08-17 @ 09:54
Hello list,

since storage provider often support WebDAV, I decided to give attic a try
using davfs2. However, I was not successfully.

Using version 0.13 I get
# attic create --stats /home/heinz/dav/dav/test.attic::first /home/heinz/test
hashindex: /home/heinz/dav/dav/test.attic/index.0: mmap failed
Traceback (most recent call last):
  File "/usr/local/bin/attic", line 3, in <module>
    main()
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
715, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
705, in run
    return args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
99, in do_create
    manifest, key = Manifest.load(repository)
  File "/usr/local/lib/python3.4/dist-packages/attic/helpers.py", line 96, in load
    cdata = repository.get(cls.MANIFEST_ID)
  File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", line 
336, in get
    self.index = self.get_read_only_index(self.get_transaction_id())
  File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", line 
134, in get_read_only_index
    return NSIndex((os.path.join(self.path, 'index.%d') % 
transaction_id).encode('utf-8'), readonly=True)
  File "hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__ 
(attic/hashindex.c:1040)
Exception: Failed to open b'/home/heinz/dav/dav/test.attic/index.0'



Using the latest version on GitHub (21e03af56b) the error message is
# attic create --stats /home/heinz/dav/dav/test.attic::first /home/heinz/test
hashindex: /home/heinz/dav/dav/test.attic/index.0: fread failed
Traceback (most recent call last):
  File "/usr/local/bin/attic", line 3, in <module>
    main()
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
725, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
715, in run
    return args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
99, in do_create
    manifest, key = Manifest.load(repository)
  File "/usr/local/lib/python3.4/dist-packages/attic/helpers.py", line 96, in load
    cdata = repository.get(cls.MANIFEST_ID)
  File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", line 
332, in get
    self.index = self.open_index(self.get_transaction_id())
  File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", line 
134, in open_index
    return NSIndex.read((os.path.join(self.path, 'index.%d') % 
transaction_id).encode('utf-8'))
  File "hashindex.pyx", line 47, in attic.hashindex.IndexBase.read 
(attic/hashindex.c:1209)
  File "hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__ 
(attic/hashindex.c:1037)
Exception: hashindex_read failed


Just to clarify, the documentation mentions:

However, be aware that sshfs doesn’t fully implement POSIX locks, so you 
must be sure to not have two processes trying to access the same 
repository at the same time.

Is this satisfied if I just type the commands like above?


Best regards,
Heinz

Re: [attic] attic with davfs2

From:
Jonas Borgström
Date:
2014-08-21 @ 21:14
On 2014-08-17 11:54, Heinz Haderer wrote:
> Hello list,
> 
> since storage provider often support WebDAV, I decided to give attic a 
try using davfs2. However, I was not successfully.
> 
> Using version 0.13 I get
> # attic create --stats /home/heinz/dav/dav/test.attic::first /home/heinz/test
> hashindex: /home/heinz/dav/dav/test.attic/index.0: mmap failed
> Traceback (most recent call last):
>   File "/usr/local/bin/attic", line 3, in <module>
>     main()
>   File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
715, in main
>     exit_code = archiver.run(sys.argv[1:])
>   File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
705, in run
>     return args.func(args)
>   File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
99, in do_create
>     manifest, key = Manifest.load(repository)
>   File "/usr/local/lib/python3.4/dist-packages/attic/helpers.py", line 
96, in load
>     cdata = repository.get(cls.MANIFEST_ID)
>   File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", 
line 336, in get
>     self.index = self.get_read_only_index(self.get_transaction_id())
>   File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", 
line 134, in get_read_only_index
>     return NSIndex((os.path.join(self.path, 'index.%d') % 
transaction_id).encode('utf-8'), readonly=True)
>   File "hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__ 
(attic/hashindex.c:1040)
> Exception: Failed to open b'/home/heinz/dav/dav/test.attic/index.0'
> 
> 
> 
> Using the latest version on GitHub (21e03af56b) the error message is
> # attic create --stats /home/heinz/dav/dav/test.attic::first /home/heinz/test
> hashindex: /home/heinz/dav/dav/test.attic/index.0: fread failed
> Traceback (most recent call last):
>   File "/usr/local/bin/attic", line 3, in <module>
>     main()
>   File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
725, in main
>     exit_code = archiver.run(sys.argv[1:])
>   File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
715, in run
>     return args.func(args)
>   File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 
99, in do_create
>     manifest, key = Manifest.load(repository)
>   File "/usr/local/lib/python3.4/dist-packages/attic/helpers.py", line 
96, in load
>     cdata = repository.get(cls.MANIFEST_ID)
>   File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", 
line 332, in get
>     self.index = self.open_index(self.get_transaction_id())
>   File "/usr/local/lib/python3.4/dist-packages/attic/repository.py", 
line 134, in open_index
>     return NSIndex.read((os.path.join(self.path, 'index.%d') % 
transaction_id).encode('utf-8'))
>   File "hashindex.pyx", line 47, in attic.hashindex.IndexBase.read 
(attic/hashindex.c:1209)
>   File "hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__ 
(attic/hashindex.c:1037)
> Exception: hashindex_read failed

As you've noticed the git version has switched to a different hashindex
implementation that uses a basic fread() call to read the entire file
into memory instead of the slightly more exotic mmap(). So this makes it
even more surprising that the git version does not work.

Here's some questions:

1. Is the filesystem otherwise working correctly as far as you can tell?

2. Does the create command succeed if you first copy the repository back
to a local filesystem, or is it corrupt/damanged?

3. Which webdav provider do you use?

4. Can you re-run the command using "strace". That output might answer
the question why fread() fail.

> Just to clarify, the documentation mentions:
> 
> However, be aware that sshfs doesn’t fully implement POSIX locks, so you
must be sure to not have two processes trying to access the same 
repository at the same time.
> 
> Is this satisfied if I just type the commands like above?

Without a working POSIX locks Attic itself can't protect repositories
from concurrent modification. But as long as you are sure you only run
one attic command at a time you're safe.

/ Jonas

Re: [attic] attic with davfs2

From:
Heinz Haderer
Date:
2014-08-22 @ 21:55
1. Is the filesystem otherwise working correctly as far as you can tell?


Apart from the delay, basic operations like creating a file, copying, 
deleting etc. are working correctly.

2. Does the create command succeed if you first copy the repository back
to a local filesystem, or is it corrupt/damanged?

No, it doesn't. I noticed that the file size of index.0 and hints.0 of a 
repository which is initialized into the davfs-mounted folder is zero.


3. Which webdav provider do you use?

https://dav.box.com




4. Can you re-run the command using "strace". That output might answer
the question why fread() fail.

Yes, I did some further investigation. I figured out that the renaming of 
index.tmp and hints.tmp causes the problem. This seems to be an issue of 
davfs (see http://sourceforge.net/p/dav/bugs/72/#fc76 ). In my case, I 
could find the tmp files in the corresponding lost+found folder. However, 
using "delay_upload 0" forces davfs to upload the files immediately after 
closing (see http://savannah.nongnu.org/support/?107615 ). This seems to 
fix this issue.

Thanks a lot for pointing me into the right direction.

Best regards,
Heinz