librelist archives

« back to archive

Does attic have an upper data size limit?

Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-06 @ 16:37
I have about 2.5TB of data that I'm trying to run through attic. I've tried
with 0.14 and with git master. Both times, as the index reaches about
2.5GB, the backup job fails with:

hashindex: data.attic/index.349400: Incorrect file
length                              Traceback (most recent call
last):
File "/usr/local/bin/attic", line 4, in
<module>
__import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
'attic')                      File
"/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line
652, in run_script    self.require(requires)[0].run_script(script_name,
ns)                                           File
"/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line
1573, in
run_script
exec(code, namespace,
namespace)
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
line 3, in <module>
main()
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 727, in main
exit_code =
archiver.run(sys.argv[1:])
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 717, in run
return
args.func(args)
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 128, in do_create
self._process(archive, cache, args.excludes, args.exclude_caches,
skip_inodes, path,
restrict_dev)
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process
os.path.join(path, filename),
restrict_dev)                                                     File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process
os.path.join(path, filename),
restrict_dev)                                                     File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process
os.path.join(path, filename),
restrict_dev)                                                     File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process
os.path.join(path, filename),
restrict_dev)                                                     File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 163, in _process
archive.process_file(path, st,
cache)                                                           File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
line 416, in process_file
self.add_item(item)
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
line 179, in add_item
self.write_checkpoint()
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
line 185, in write_checkpoint
self.cache.chunk_decref(self.id,
self.stats)                                                    File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
line 202, in chunk_decref
self.repository.delete(id,
wait=False)                                                          File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
line 365, in delete
self.prepare_txn(self.get_transaction_id())
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
line 143, in prepare_txn
self.index =
self.open_index(transaction_id)
File

"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
line 137, in open_index
return NSIndex.read((os.path.join(self.path, 'index.%d') %
transaction_id).encode('utf-8'))     File "attic/hashindex.pyx", line 47,
in attic.hashindex.IndexBase.read (attic/hashindex.c:1128)   File
"attic/hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__
(attic/hashindex.c:955)
Exception: hashindex_read failed

I've tried this twice on different machines, once with a remote target and
once locally.

Seems similar to that described here:

http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4

I haven't found any other reference to hashindex file length issues.

Any insight appreciated.

Sv: [attic] Does attic have an upper data size limit?

From:
Petter Gunnerud
Date:
2015-01-06 @ 17:36
My cache is close to 4GB. My data is about 800GB (1200GB if expanding 
sparse files). I've used attic for about 6 weeks, 4 runs/week.I haven't 
run into any size issues yet, except that each run takes 10 hours. 

Attic version is 0.13

      Fra: William Eshagh <william@eshagh.com>
 Til: attic@librelist.com 
 Sendt: Tirsdag, 6. januar 2015 17.37
 Emne: [attic] Does attic have an upper data size limit?
   
I have about 2.5TB of data that I'm trying to run through attic. I've 
tried with 0.14 and with git master. Both times, as the index reaches 
about 2.5GB, the backup job fails with:hashindex: data.attic/index.349400:
Incorrect file length                              Traceback (most recent 
call 
last):                                                                  
File "/usr/local/bin/attic", line 4, in 
<module>                                                    
__import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a', 
'attic')                      File 
"/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line 
652, in run_script    self.require(requires)[0].run_script(script_name, 
ns)                                           File 
"/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line 
1573, in run_script                    
 exec(code, namespace, 
namespace)                                                                
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
line 3, in <module>                                                
main()                                                                                         
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 727, in main                                                        
exit_code = 
archiver.run(sys.argv[1:])                                                         
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 717, in run                                                        
return 
args.func(args)                                                                         
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 128, in do_create                                                  
self._process(archive, cache, args.excludes, args.exclude_caches, 
skip_inodes, path, restrict_dev)             
 File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process                                                   
os.path.join(path, filename), 
restrict_dev)                                                     File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process                                                   
os.path.join(path, filename), 
restrict_dev)                                                     File 
"/usr/lo 
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process                                                   
os.path.join(path, filename), 
restrict_dev)                                                     File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process                                                   
os.path.join(path, filename), 
restrict_dev)                                                     File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 177, in _process    
 File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
line 163, in _process                                                   
archive.process_file(path, st, 
cache)                                                           File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
line 416, in process_file                                                
self.add_item(item)                                                                            
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
line 179, in add_item                                                     
self.write_checkpoint()                                                                        
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
line 185, in write_checkpoint                                            
self.cache.chunk_decref(self.id, 
self.stats)                                                    File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
line 202, in chunk_decref                                                 
self.repository.delete(id, wait=False)    
 File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
line 365, in delete                                                   
self.prepare_txn(self.get_transaction_id())                                                    
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
line 143, in prepare_txn                                              
self.index = 
self.open_index(transaction_id)                                                   
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free 
bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in 
open_index                                               return 
NSIndex.read((os.path.join(self.path, 'index.%d') % 
transaction_id).encode('utf-8'))     File "attic/hashindex.pyx", line 47, 
in attic.hashindex.IndexBase.read (attic/hashindex.c:1128)   File 
"attic/hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__ 
(attic/hashindex.c:955)                                                                                              
Exception: hashindex_read failedI've tried this twice on different 
machines, once with a remote target and once locally.      Seems similar 
to that described here:       
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4I
haven't found any other reference to hashindex file length issues.Any 
insight appreciated.                           

  

Re: Sv: [attic] Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-06 @ 20:55
Thanks Peter, so it may not be not hash index size. Anyone have a
repository of 2-3T or more?
On Jan 6, 2015 9:37 AM, "Petter Gunnerud" <pgspm@yahoo.no> wrote:

> My cache is close to 4GB. My data is about 800GB (1200GB if expanding
> sparse files). I've used attic for about 6 weeks, 4 runs/week.
> I haven't run into any size issues yet, except that each run takes 10
> hours.
>
> Attic version is 0.13
>
>   ------------------------------
>  *Fra:* William Eshagh <william@eshagh.com>
> *Til:* attic@librelist.com
> *Sendt:* Tirsdag, 6. januar 2015 17.37
> *Emne:* [attic] Does attic have an upper data size limit?
>
> I have about 2.5TB of data that I'm trying to run through attic. I've
> tried with 0.14 and with git master. Both times, as the index reaches about
> 2.5GB, the backup job fails with:
> hashindex: data.attic/index.349400: Incorrect file
> length                              Traceback (most recent call
> last):
> File "/usr/local/bin/attic", line 4, in
> <module>                                             &n bsp;
> __import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
> 'attic')                      File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line
> 652, in run_script    self.require(requires)[0].run_script(script_name,
> ns)                                           File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line
> 1573, in run_script
>
> exec(code, namespace,
> namespace)
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd
> 64.egg/EGG-INFO/scripts/attic", line 3, in
> <module>
> main()
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/arch
> iver.py", line 727, in main
>                  exit_code =
> archiver.run(sys.argv[1:])
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 717, in run
> return args.func(args)
>                                                                       File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 128, in do_create
> self._process(archive, cache, args.excludes, args.exclude_caches,
> skip_inodes, path, restrict_dev)        ;
>
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
> os.path.join(path, filename), restrict_dev) &
> nbsp;                                                   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
> os.path.join(path, filename),
> restrict_dev)                                 &n bsp;
> File "/usr/lo
> 
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
> os.path.join(path, filename),
> restrict_dev)                                                     File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process          
> ;                                         os.path.join(path, filename),
> restrict_dev)                                                     File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 163, in _process
> archive.process_file(path, st,
> cache)                                                           File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 416, in process_file  &
> nbsp;
> self.add_item(item)
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 179, in add_item
> self.write_checkpoint()
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 185, in write_checkpoint              &n
> bsp;                             self.cache.chunk_decref(self.id,
> self.stats)                                                    File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> line 202, in chunk_decref
> self.repository.delete(id, wait=False)
>                                                        File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 365, in delete
> self.prepare_txn(self.get_transaction_id())                              &
> nbsp;                      File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 143, in prepare_txn
> self.index =
> self.open_index(transaction_id)
> File
> "/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free
> bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in
> open_index                                               return
> NSIndex.read((os.path.join(self.path, 'index.%d') %
> transaction_id).encode('utf-8'))     File "attic/hashindex.pyx", line 47,
> in attic.hashindex.IndexBase.read (attic/hashindex.c:1128)   File
> "attic/hashindex.pyx", line 35, in attic.hashindex.IndexBase.__cinit__
> (attic/hashindex.c:955)
 
> ;                                         Exception: hashindex_read failed
> I've tried this twice on different machines, once with a remote target and
> once locally.
> Seems similar to that described here:
> 
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4
> I haven't found any other reference to hashindex file length issues.
> Any insight appreciated.
>
>
>

Re: Sv: [attic] Does attic have an upper data size limit?

From:
Jonas Borgström
Date:
2015-01-06 @ 21:40
I though that I fixed all 2GB issues with this commit:

https://github.com/jborg/attic/commit/7e15abd29c94ef2d499b05520cd805c9f4dba798

But it looks like I missed one place:

Hopefully this should fix your crash:
https://github.com/jborg/attic/commit/4ab4ecc7afbfdfb081b28670b33af87b8dd2e4ff

To answer your initial question. Attic has no hard upper data size limit
but the deduplication is quite memory hungry and the memory usage
depends on the amount of deduplicated (unique) data stored in the
repository.

/ Jonas

On 2015-01-06 21:55, William Eshagh wrote:
>
> Thanks Peter, so it may not be not hash index size. Anyone have a
> repository of 2-3T or more?
>
> On Jan 6, 2015 9:37 AM, "Petter Gunnerud" <pgspm@yahoo.no
> <mailto:pgspm@yahoo.no>> wrote:
>
>     My cache is close to 4GB. My data is about 800GB (1200GB if
>     expanding sparse files). I've used attic for about 6 weeks, 4
>     runs/week.
>     I haven't run into any size issues yet, except that each run takes
>     10 hours.
>
>     Attic version is 0.13
>
>     ------------------------------------------------------------------------
>     *Fra:* William Eshagh <william@eshagh.com <mailto:william@eshagh.com>>
>     *Til:* attic@librelist.com <mailto:attic@librelist.com>
>     *Sendt:* Tirsdag, 6. januar 2015 17.37
>     *Emne:* [attic] Does attic have an upper data size limit?
>
>     I have about 2.5TB of data that I'm trying to run through attic.
>     I've tried with 0.14 and with git master. Both times, as the index
>     reaches about 2.5GB, the backup job fails with:
>     hashindex: data.attic/index.349400: Incorrect file
>     length                              Traceback (most recent call
>     last):                                                                 
>     File "/usr/local/bin/attic", line 4, in
>     <module>                                             &n bsp;     
>     __import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
>     'attic')                      File
>     "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line
>     652, in run_script   
>     self.require(requires)[0].run_script(script_name,
>     ns)                                           File
>     "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py", line
>     1573, in run_script                   
>

>     exec(code, namespace,
>     namespace)                                                               
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd
>     64.egg/EGG-INFO/scripts/attic", line 3, in
>     <module>                                               
>     main()

>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/arch
>     iver.py", line 727, in main                                      
>                      exit_code =
>     archiver.run(sys.argv[1:])

>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 717, in
>     run                                                        return
>     args.func(args)    
>                                                                          
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 128, in
>     do_create                                                 
>     self._process(archive, cache, args.excludes, args.exclude_caches,
>     skip_inodes, path, restrict_dev)        ;     
>

>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 177, in
>     _process                                                  
>     os.path.join(path, filename), restrict_dev) &
>     nbsp;                                                   File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 177, in
>     _process                                                  
>     os.path.join(path, filename),
>     restrict_dev)                                 &n
>     bsp;                   File "/usr/lo
>     
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 177, in
>     _process                                                  
>     os.path.join(path, filename),
>     restrict_dev)                                                    
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 177, in _process          
>     ;                                         os.path.join(path,
>     filename),
>     restrict_dev)                                                    
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 177, in _process   
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>     line 163, in
>     _process                                                  
>     archive.process_file(path, st,
>     cache)                                                          
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>     line 416, in process_file  &
>     nbsp;                                            
>     self.add_item(item)

>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>     line 179, in add_item 
>                                                       
>     self.write_checkpoint()

>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>     line 185, in write_checkpoint              &n
>     bsp;                             self.cache.chunk_decref(self.id
>     <http://self.id/>,
>     self.stats)                                                   
>     File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
>     line 202, in
>     chunk_decref                                                  
>     self.repository.delete(id, wait=False)   
>                                                            File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>     line 365, in
>     delete                                                  
>     self.prepare_txn(self.get_transaction_id())                              &
>     nbsp;                      File
>     
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>     line 143, in
>     prepare_txn                                             
>     self.index =
>     self.open_index(transaction_id)

>     File
>     "/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free
>     bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in
>     open_index                                               return
>     NSIndex.read((os.path.join(self.path, 'index.%d') %
>     transaction_id).encode('utf-8'))     File "attic/hashindex.pyx",
>     line 47, in attic.hashindex.IndexBase.read
>     (attic/hashindex.c:1128)   File "attic/hashindex.pyx", line 35, in
>     attic.hashindex.IndexBase.__cinit__
>     (attic/hashindex.c:955)
 
>     ;                                         Exception:
>     hashindex_read failed
>     I've tried this twice on different machines, once with a remote
>     target and once locally.     
>     Seems similar to that described here:      
>     
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4
>     I haven't found any other reference to hashindex file length issues.
>     Any insight appreciated.                          
>
>

Re: Sv: [attic] Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-07 @ 04:52
Thanks, trying it now. It'll take a few days to rerun the backup anew.
On Jan 6, 2015 1:40 PM, "Jonas Borgström" <jonas@borgstrom.se> wrote:

> I though that I fixed all 2GB issues with this commit:
>
>
> https://github.com/jborg/attic/commit/7e15abd29c94ef2d499b05520cd805c9f4dba798
>
> But it looks like I missed one place:
>
> Hopefully this should fix your crash:
>
> https://github.com/jborg/attic/commit/4ab4ecc7afbfdfb081b28670b33af87b8dd2e4ff
>
> To answer your initial question. Attic has no hard upper data size limit
> but the deduplication is quite memory hungry and the memory usage
> depends on the amount of deduplicated (unique) data stored in the
> repository.
>
> / Jonas
>
> On 2015-01-06 21:55, William Eshagh wrote:
> >
> > Thanks Peter, so it may not be not hash index size. Anyone have a
> > repository of 2-3T or more?
> >
> > On Jan 6, 2015 9:37 AM, "Petter Gunnerud" <pgspm@yahoo.no
> > <mailto:pgspm@yahoo.no>> wrote:
> >
> >     My cache is close to 4GB. My data is about 800GB (1200GB if
> >     expanding sparse files). I've used attic for about 6 weeks, 4
> >     runs/week.
> >     I haven't run into any size issues yet, except that each run takes
> >     10 hours.
> >
> >     Attic version is 0.13
> >
> >
>  ------------------------------------------------------------------------
> >     *Fra:* William Eshagh <william@eshagh.com <mailto:william@eshagh.com
> >>
> >     *Til:* attic@librelist.com <mailto:attic@librelist.com>
> >     *Sendt:* Tirsdag, 6. januar 2015 17.37
> >     *Emne:* [attic] Does attic have an upper data size limit?
> >
> >     I have about 2.5TB of data that I'm trying to run through attic.
> >     I've tried with 0.14 and with git master. Both times, as the index
> >     reaches about 2.5GB, the backup job fails with:
> >     hashindex: data.attic/index.349400: Incorrect file
> >     length                              Traceback (most recent call
> >     last):
> >     File "/usr/local/bin/attic", line 4, in
> >     <module>                                             &n bsp;
> >     __import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
> >     'attic')                      File
> >     "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line
> >     652, in run_script
> >     self.require(requires)[0].run_script(script_name,
> >     ns)                                           File
> >     "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line
> >     1573, in run_script
> >
> >     exec(code, namespace,
> >     namespace)
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd
> >     64.egg/EGG-INFO/scripts/attic", line 3, in
> >     <module>
> >     main()
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/arch
> >     iver.py", line 727, in main
> >                      exit_code =
> >     archiver.run(sys.argv[1:])
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 717, in
> >     run                                                        return
> >     args.func(args)
> >
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 128, in
> >     do_create
> >     self._process(archive, cache, args.excludes, args.exclude_caches,
> >     skip_inodes, path, restrict_dev)        ;
> >
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 177, in
> >     _process
> >     os.path.join(path, filename), restrict_dev) &
> >     nbsp;                                                   File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 177, in
> >     _process
> >     os.path.join(path, filename),
> >     restrict_dev)                                 &n
> >     bsp;                   File "/usr/lo
> >
>  
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 177, in
> >     _process
> >     os.path.join(path, filename),
> >     restrict_dev)
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 177, in _process          
> >     ;                                         os.path.join(path,
> >     filename),
> >     restrict_dev)
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 177, in _process
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >     line 163, in
> >     _process
> >     archive.process_file(path, st,
> >     cache)
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> >     line 416, in process_file  &
> >     nbsp;
> >     self.add_item(item)
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> >     line 179, in add_item
> >
> >     self.write_checkpoint()
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> >     line 185, in write_checkpoint              &n
> >     bsp;                             self.cache.chunk_decref(self.id
> >     <http://self.id/>,
> >     self.stats)
> >     File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> >     line 202, in
> >     chunk_decref
> >     self.repository.delete(id, wait=False)
> >                                                            File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> >     line 365, in
> >     delete
> >     self.prepare_txn(self.get_transaction_id())
>     &
> >     nbsp;                      File
> >
>  
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> >     line 143, in
> >     prepare_txn
> >     self.index =
> >     self.open_index(transaction_id)
> >     File
> >
>  "/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free
> >     bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in
> >     open_index                                               return
> >     NSIndex.read((os.path.join(self.path, 'index.%d') %
> >     transaction_id).encode('utf-8'))     File "attic/hashindex.pyx",
> >     line 47, in attic.hashindex.IndexBase.read
> >     (attic/hashindex.c:1128)   File "attic/hashindex.pyx", line 35, in
> >     attic.hashindex.IndexBase.__cinit__
> >     (attic/hashindex.c:955)
>         
> >     ;                                         Exception:
> >     hashindex_read failed
> >     I've tried this twice on different machines, once with a remote
> >     target and once locally.
> >     Seems similar to that described here:
> >
> 
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4
> >     I haven't found any other reference to hashindex file length issues.
> >     Any insight appreciated.
> >
> >
>
>
>

Re: Sv: [attic] Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-09 @ 03:34
could not decode message

Re: Sv: [attic] Does attic have an upper data size limit?

From:
Petros Moisiadis
Date:
2015-01-09 @ 08:25
I am not sure if this is going to fix William's problem (William, do you
happen to run attic on a 32-bit machine?), but according to

http://stackoverflow.com/questions/9073667/where-to-find-the-complete-definition-of-off-t-type/14351239#14351239,
maybe it is more reliable to use off64_t instead of off_t.

On 01/09/2015 05:34 AM, William Eshagh wrote:
> Hit the same issue, right at about the same point in the backup:
>
> hashindex: data.attic/index.349366: Incorrect file length
> Traceback (most recent call last):
>   File "/usr/local/bin/attic", line 4, in <module>
>     __import__('pkg_resources').run_script('Attic===0.14-5-g4ab4ecc',
> 'attic')
>   File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line 652, in run_script
>     self.require(requires)[0].run_script(script_name, ns)
>   File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line 1573, in run_script
>     exec(code, namespace, namespace)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
> line 3, in <module>
>     main()
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 727, in main
>     exit_code = archiver.run(sys.argv[1:])
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 717, in run
>     return args.func(args)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 128, in do_create
>     self._process(archive, cache, args.excludes, args.exclude_caches,
> skip_inodes, path, restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 163, in _process
>     archive.process_file(path, st, cache)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 416, in process_file
>     self.add_item(item)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 179, in add_item
>     self.write_checkpoint()
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 185, in write_checkpoint
>     self.cache.chunk_decref(self.id, self.stats)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> line 202, in chunk_decref
>     self.repository.delete(id, wait=False)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 365, in delete
>     self.prepare_txn(self.get_transaction_id())
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 143, in prepare_txn
>     self.index = self.open_index(transaction_id)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 137, in open_index
>     return NSIndex.read((os.path.join(self.path, 'index.%d') %
> transaction_id).encode('utf-8'))
>   File "attic/hashindex.pyx", line 47, in
> attic.hashindex.IndexBase.read (attic/hashindex.c:1128)
>   File "attic/hashindex.pyx", line 35, in
> attic.hashindex.IndexBase.__cinit__ (attic/hashindex.c:955)
> Exception: hashindex_read failed
>
> The index file is the same size as last time, 2.5GB. Could something
> else be contributing to the hashindex issue?
>
> On 01/06/2015 08:52 PM, William Eshagh wrote:
>>
>> Thanks, trying it now. It'll take a few days to rerun the backup anew.
>>
>> On Jan 6, 2015 1:40 PM, "Jonas Borgström" <jonas@borgstrom.se
>> <mailto:jonas@borgstrom.se>> wrote:
>>
>>     I though that I fixed all 2GB issues with this commit:
>>
>>     
https://github.com/jborg/attic/commit/7e15abd29c94ef2d499b05520cd805c9f4dba798
>>
>>     But it looks like I missed one place:
>>
>>     Hopefully this should fix your crash:
>>     
https://github.com/jborg/attic/commit/4ab4ecc7afbfdfb081b28670b33af87b8dd2e4ff
>>
>>     To answer your initial question. Attic has no hard upper data
>>     size limit
>>     but the deduplication is quite memory hungry and the memory usage
>>     depends on the amount of deduplicated (unique) data stored in the
>>     repository.
>>
>>     / Jonas
>>
>>     On 2015-01-06 21:55, William Eshagh wrote:
>>     >
>>     > Thanks Peter, so it may not be not hash index size. Anyone have a
>>     > repository of 2-3T or more?
>>     >
>>     > On Jan 6, 2015 9:37 AM, "Petter Gunnerud" <pgspm@yahoo.no
>>     <mailto:pgspm@yahoo.no>
>>     > <mailto:pgspm@yahoo.no <mailto:pgspm@yahoo.no>>> wrote:
>>     >
>>     >     My cache is close to 4GB. My data is about 800GB (1200GB if
>>     >     expanding sparse files). I've used attic for about 6 weeks, 4
>>     >     runs/week.
>>     >     I haven't run into any size issues yet, except that each
>>     run takes
>>     >     10 hours.
>>     >
>>     >     Attic version is 0.13
>>     >
>>     >   
>>      ------------------------------------------------------------------------
>>     >     *Fra:* William Eshagh <william@eshagh.com
>>     <mailto:william@eshagh.com> <mailto:william@eshagh.com
>>     <mailto:william@eshagh.com>>>
>>     >     *Til:* attic@librelist.com <mailto:attic@librelist.com>
>>     <mailto:attic@librelist.com <mailto:attic@librelist.com>>
>>     >     *Sendt:* Tirsdag, 6. januar 2015 17.37
>>     >     *Emne:* [attic] Does attic have an upper data size limit?
>>     >
>>     >     I have about 2.5TB of data that I'm trying to run through
>>     attic.
>>     >     I've tried with 0.14 and with git master. Both times, as
>>     the index
>>     >     reaches about 2.5GB, the backup job fails with:
>>     >     hashindex: data.attic/index.349400: Incorrect file
>>     >     length                              Traceback (most recent call
>>     >     last):
>>     >     File "/usr/local/bin/attic", line 4, in
>>     >     <module>                                             &n bsp;
>>     >   
>>      __import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
>>     >     'attic')                      File
>>     >   
>>      "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>>     line
>>     >     652, in run_script
>>     >     self.require(requires)[0].run_script(script_name,
>>     >     ns)                                           File
>>     >   
>>      "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>>     line
>>     >     1573, in run_script
>>     >
>>     >     exec(code, namespace,
>>     >     namespace)
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd
>>     >     64.egg/EGG-INFO/scripts/attic", line 3, in
>>     >     <module>
>>     >     main()
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/arch
>>     >     iver.py", line 727, in main
>>     >                      exit_code =
>>     >     archiver.run(sys.argv[1:])
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 717, in
>>     >     run                                                       
>>     return
>>     >     args.func(args)
>>     >
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 128, in
>>     >     do_create
>>     >     self._process(archive, cache, args.excludes,
>>     args.exclude_caches,
>>     >     skip_inodes, path, restrict_dev)        ;
>>     >
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 177, in
>>     >     _process
>>     >     os.path.join(path, filename), restrict_dev) &
>>     >     nbsp;                                                   File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 177, in
>>     >     _process
>>     >     os.path.join(path, filename),
>>     >     restrict_dev)                                 &n
>>     >     bsp;                   File "/usr/lo
>>     >   
>>      
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 177, in
>>     >     _process
>>     >     os.path.join(path, filename),
>>     >     restrict_dev)
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 177, in _process          
>>     >     ;                                         os.path.join(path,
>>     >     filename),
>>     >     restrict_dev)
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 177, in _process
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>     >     line 163, in
>>     >     _process
>>     >     archive.process_file(path, st,
>>     >     cache)
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>>     >     line 416, in process_file  &
>>     >     nbsp;
>>     >     self.add_item(item)
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>>     >     line 179, in add_item
>>     >
>>     >     self.write_checkpoint()
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>>     >     line 185, in write_checkpoint              &n
>>     >     bsp;                           
>>      self.cache.chunk_decref(self.id <http://self.id>
>>     >     <http://self.id/>,
>>     >     self.stats)
>>     >     File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
>>     >     line 202, in
>>     >     chunk_decref
>>     >     self.repository.delete(id, wait=False)
>>     >                                                            File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>>     >     line 365, in
>>     >     delete
>>     >     self.prepare_txn(self.get_transaction_id())               
>>                   &
>>     >     nbsp;                      File
>>     >   
>>      
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>>     >     line 143, in
>>     >     prepare_txn
>>     >     self.index =
>>     >     self.open_index(transaction_id)
>>     >     File
>>     >   
>>      "/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free
>>     >     bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in
>>     >     open_index                                               return
>>     >     NSIndex.read((os.path.join(self.path, 'index.%d') %
>>     >     transaction_id).encode('utf-8'))     File
>>     "attic/hashindex.pyx",
>>     >     line 47, in attic.hashindex.IndexBase.read
>>     >     (attic/hashindex.c:1128)   File "attic/hashindex.pyx", line
>>     35, in
>>     >     attic.hashindex.IndexBase.__cinit__
>>     >     (attic/hashindex.c:955)                                   
>>                       
>>     >     ;                                         Exception:
>>     >     hashindex_read failed
>>     >     I've tried this twice on different machines, once with a remote
>>     >     target and once locally.
>>     >     Seems similar to that described here:
>>     >   
>>      
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4
>>     >     I haven't found any other reference to hashindex file
>>     length issues.
>>     >     Any insight appreciated.
>>     >
>>     >
>>
>>
>

Re: Sv: [attic] Does attic have an upper data size limit?

From:
Thiago Coutinho
Date:
2015-01-09 @ 10:40
The error suggest he is using x86_64 on FreeBSD:

...
File 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
line 3, in <module>
...

2015-01-09 6:25 GMT-02:00 Petros Moisiadis <ernest0x@yahoo.gr>:
> I am not sure if this is going to fix William's problem (William, do you
> happen to run attic on a 32-bit machine?), but according to
> 
http://stackoverflow.com/questions/9073667/where-to-find-the-complete-definition-of-off-t-type/14351239#14351239,
> maybe it is more reliable to use off64_t instead of off_t.
>
> On 01/09/2015 05:34 AM, William Eshagh wrote:
>
> Hit the same issue, right at about the same point in the backup:
>
> hashindex: data.attic/index.349366: Incorrect file length
> Traceback (most recent call last):
>   File "/usr/local/bin/attic", line 4, in <module>
>     __import__('pkg_resources').run_script('Attic===0.14-5-g4ab4ecc',
> 'attic')
>   File "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line 652, in run_script
>     self.require(requires)[0].run_script(script_name, ns)
>   File "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line 1573, in run_script
>     exec(code, namespace, namespace)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
> line 3, in <module>
>     main()
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 727, in main
>     exit_code = archiver.run(sys.argv[1:])
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 717, in run
>     return args.func(args)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 128, in do_create
>     self._process(archive, cache, args.excludes, args.exclude_caches,
> skip_inodes, path, restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 163, in _process
>     archive.process_file(path, st, cache)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 416, in process_file
>     self.add_item(item)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 179, in add_item
>     self.write_checkpoint()
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 185, in write_checkpoint
>     self.cache.chunk_decref(self.id, self.stats)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> line 202, in chunk_decref
>     self.repository.delete(id, wait=False)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 365, in delete
>     self.prepare_txn(self.get_transaction_id())
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 143, in prepare_txn
>     self.index = self.open_index(transaction_id)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 137, in open_index
>     return NSIndex.read((os.path.join(self.path, 'index.%d') %
> transaction_id).encode('utf-8'))
>   File "attic/hashindex.pyx", line 47, in attic.hashindex.IndexBase.read
> (attic/hashindex.c:1128)
>   File "attic/hashindex.pyx", line 35, in
> attic.hashindex.IndexBase.__cinit__ (attic/hashindex.c:955)
> Exception: hashindex_read failed
>
> The index file is the same size as last time, 2.5GB. Could something else be
> contributing to the hashindex issue?
>
> On 01/06/2015 08:52 PM, William Eshagh wrote:
>
> Thanks, trying it now. It'll take a few days to rerun the backup anew.
>
> On Jan 6, 2015 1:40 PM, "Jonas Borgström" <jonas@borgstrom.se> wrote:
>>
>> I though that I fixed all 2GB issues with this commit:
>>
>>
>> https://github.com/jborg/attic/commit/7e15abd29c94ef2d499b05520cd805c9f4dba798
>>
>> But it looks like I missed one place:
>>
>> Hopefully this should fix your crash:
>>
>> https://github.com/jborg/attic/commit/4ab4ecc7afbfdfb081b28670b33af87b8dd2e4ff
>>
>> To answer your initial question. Attic has no hard upper data size limit
>> but the deduplication is quite memory hungry and the memory usage
>> depends on the amount of deduplicated (unique) data stored in the
>> repository.
>>
>> / Jonas
>>
>> On 2015-01-06 21:55, William Eshagh wrote:
>> >
>> > Thanks Peter, so it may not be not hash index size. Anyone have a
>> > repository of 2-3T or more?
>> >
>> > On Jan 6, 2015 9:37 AM, "Petter Gunnerud" <pgspm@yahoo.no
>> > <mailto:pgspm@yahoo.no>> wrote:
>> >
>> >     My cache is close to 4GB. My data is about 800GB (1200GB if
>> >     expanding sparse files). I've used attic for about 6 weeks, 4
>> >     runs/week.
>> >     I haven't run into any size issues yet, except that each run takes
>> >     10 hours.
>> >
>> >     Attic version is 0.13
>> >
>> >
>> > ------------------------------------------------------------------------
>> >     *Fra:* William Eshagh <william@eshagh.com
>> > <mailto:william@eshagh.com>>
>> >     *Til:* attic@librelist.com <mailto:attic@librelist.com>
>> >     *Sendt:* Tirsdag, 6. januar 2015 17.37
>> >     *Emne:* [attic] Does attic have an upper data size limit?
>> >
>> >     I have about 2.5TB of data that I'm trying to run through attic.
>> >     I've tried with 0.14 and with git master. Both times, as the index
>> >     reaches about 2.5GB, the backup job fails with:
>> >     hashindex: data.attic/index.349400: Incorrect file
>> >     length                              Traceback (most recent call
>> >     last):
>> >     File "/usr/local/bin/attic", line 4, in
>> >     <module>                                             &n bsp;
>> >     __import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
>> >     'attic')                      File
>> >     "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>> > line
>> >     652, in run_script
>> >     self.require(requires)[0].run_script(script_name,
>> >     ns)                                           File
>> >     "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>> > line
>> >     1573, in run_script
>> >
>> >     exec(code, namespace,
>> >     namespace)
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd
>> >     64.egg/EGG-INFO/scripts/attic", line 3, in
>> >     <module>
>> >     main()
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/arch
>> >     iver.py", line 727, in main
>> >                      exit_code =
>> >     archiver.run(sys.argv[1:])
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 717, in
>> >     run                                                        return
>> >     args.func(args)
>> >
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 128, in
>> >     do_create
>> >     self._process(archive, cache, args.excludes, args.exclude_caches,
>> >     skip_inodes, path, restrict_dev)        ;
>> >
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 177, in
>> >     _process
>> >     os.path.join(path, filename), restrict_dev) &
>> >     nbsp;                                                   File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 177, in
>> >     _process
>> >     os.path.join(path, filename),
>> >     restrict_dev)                                 &n
>> >     bsp;                   File "/usr/lo
>> >
>> > 
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 177, in
>> >     _process
>> >     os.path.join(path, filename),
>> >     restrict_dev)
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 177, in _process          
>> >     ;                                         os.path.join(path,
>> >     filename),
>> >     restrict_dev)
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 177, in _process
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> >     line 163, in
>> >     _process
>> >     archive.process_file(path, st,
>> >     cache)
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>> >     line 416, in process_file  &
>> >     nbsp;
>> >     self.add_item(item)
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>> >     line 179, in add_item
>> >
>> >     self.write_checkpoint()
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>> >     line 185, in write_checkpoint              &n
>> >     bsp;                             self.cache.chunk_decref(self.id
>> >     <http://self.id/>,
>> >     self.stats)
>> >     File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
>> >     line 202, in
>> >     chunk_decref
>> >     self.repository.delete(id, wait=False)
>> >                                                            File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>> >     line 365, in
>> >     delete
>> >     self.prepare_txn(self.get_transaction_id())
>> > &
>> >     nbsp;                      File
>> >
>> > 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>> >     line 143, in
>> >     prepare_txn
>> >     self.index =
>> >     self.open_index(transaction_id)
>> >     File
>> >
>> > "/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free
>> >     bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in
>> >     open_index                                               return
>> >     NSIndex.read((os.path.join(self.path, 'index.%d') %
>> >     transaction_id).encode('utf-8'))     File "attic/hashindex.pyx",
>> >     line 47, in attic.hashindex.IndexBase.read
>> >     (attic/hashindex.c:1128)   File "attic/hashindex.pyx", line 35, in
>> >     attic.hashindex.IndexBase.__cinit__
>> >     (attic/hashindex.c:955)
>> >  
>> >     ;                                         Exception:
>> >     hashindex_read failed
>> >     I've tried this twice on different machines, once with a remote
>> >     target and once locally.
>> >     Seems similar to that described here:
>> >
>> > 
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4
>> >     I haven't found any other reference to hashindex file length issues.
>> >     Any insight appreciated.
>> >
>> >
>>
>>
>
>



-- 
Thiago Coutinho

"O povo não deveria temer o governo. O governo é quem deveria temer o povo."
V de Vingança

Re: Sv: [attic] Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-09 @ 16:22
Yes, 64bit FreeBSD
On Jan 9, 2015 2:41 AM, "Thiago Coutinho" <root@thiagoc.net> wrote:

> The error suggest he is using x86_64 on FreeBSD:
>
> ...
> File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
> line 3, in <module>
> ...
>
> 2015-01-09 6:25 GMT-02:00 Petros Moisiadis <ernest0x@yahoo.gr>:
> > I am not sure if this is going to fix William's problem (William, do you
> > happen to run attic on a 32-bit machine?), but according to
> >
> 
http://stackoverflow.com/questions/9073667/where-to-find-the-complete-definition-of-off-t-type/14351239#14351239
> ,
> > maybe it is more reliable to use off64_t instead of off_t.
> >
> > On 01/09/2015 05:34 AM, William Eshagh wrote:
> >
> > Hit the same issue, right at about the same point in the backup:
> >
> > hashindex: data.attic/index.349366: Incorrect file length
> > Traceback (most recent call last):
> >   File "/usr/local/bin/attic", line 4, in <module>
> >     __import__('pkg_resources').run_script('Attic===0.14-5-g4ab4ecc',
> > 'attic')
> >   File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> > line 652, in run_script
> >     self.require(requires)[0].run_script(script_name, ns)
> >   File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> > line 1573, in run_script
> >     exec(code, namespace, namespace)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
> > line 3, in <module>
> >     main()
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 727, in main
> >     exit_code = archiver.run(sys.argv[1:])
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 717, in run
> >     return args.func(args)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 128, in do_create
> >     self._process(archive, cache, args.excludes, args.exclude_caches,
> > skip_inodes, path, restrict_dev)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 177, in _process
> >     os.path.join(path, filename), restrict_dev)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 177, in _process
> >     os.path.join(path, filename), restrict_dev)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 177, in _process
> >     os.path.join(path, filename), restrict_dev)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 177, in _process
> >     os.path.join(path, filename), restrict_dev)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 177, in _process
> >     os.path.join(path, filename), restrict_dev)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> > line 163, in _process
> >     archive.process_file(path, st, cache)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> > line 416, in process_file
> >     self.add_item(item)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> > line 179, in add_item
> >     self.write_checkpoint()
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> > line 185, in write_checkpoint
> >     self.cache.chunk_decref(self.id, self.stats)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> > line 202, in chunk_decref
> >     self.repository.delete(id, wait=False)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> > line 365, in delete
> >     self.prepare_txn(self.get_transaction_id())
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> > line 143, in prepare_txn
> >     self.index = self.open_index(transaction_id)
> >   File
> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> > line 137, in open_index
> >     return NSIndex.read((os.path.join(self.path, 'index.%d') %
> > transaction_id).encode('utf-8'))
> >   File "attic/hashindex.pyx", line 47, in attic.hashindex.IndexBase.read
> > (attic/hashindex.c:1128)
> >   File "attic/hashindex.pyx", line 35, in
> > attic.hashindex.IndexBase.__cinit__ (attic/hashindex.c:955)
> > Exception: hashindex_read failed
> >
> > The index file is the same size as last time, 2.5GB. Could something
> else be
> > contributing to the hashindex issue?
> >
> > On 01/06/2015 08:52 PM, William Eshagh wrote:
> >
> > Thanks, trying it now. It'll take a few days to rerun the backup anew.
> >
> > On Jan 6, 2015 1:40 PM, "Jonas Borgström" <jonas@borgstrom.se> wrote:
> >>
> >> I though that I fixed all 2GB issues with this commit:
> >>
> >>
> >>
> https://github.com/jborg/attic/commit/7e15abd29c94ef2d499b05520cd805c9f4dba798
> >>
> >> But it looks like I missed one place:
> >>
> >> Hopefully this should fix your crash:
> >>
> >>
> https://github.com/jborg/attic/commit/4ab4ecc7afbfdfb081b28670b33af87b8dd2e4ff
> >>
> >> To answer your initial question. Attic has no hard upper data size limit
> >> but the deduplication is quite memory hungry and the memory usage
> >> depends on the amount of deduplicated (unique) data stored in the
> >> repository.
> >>
> >> / Jonas
> >>
> >> On 2015-01-06 21:55, William Eshagh wrote:
> >> >
> >> > Thanks Peter, so it may not be not hash index size. Anyone have a
> >> > repository of 2-3T or more?
> >> >
> >> > On Jan 6, 2015 9:37 AM, "Petter Gunnerud" <pgspm@yahoo.no
> >> > <mailto:pgspm@yahoo.no>> wrote:
> >> >
> >> >     My cache is close to 4GB. My data is about 800GB (1200GB if
> >> >     expanding sparse files). I've used attic for about 6 weeks, 4
> >> >     runs/week.
> >> >     I haven't run into any size issues yet, except that each run takes
> >> >     10 hours.
> >> >
> >> >     Attic version is 0.13
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------
> >> >     *Fra:* William Eshagh <william@eshagh.com
> >> > <mailto:william@eshagh.com>>
> >> >     *Til:* attic@librelist.com <mailto:attic@librelist.com>
> >> >     *Sendt:* Tirsdag, 6. januar 2015 17.37
> >> >     *Emne:* [attic] Does attic have an upper data size limit?
> >> >
> >> >     I have about 2.5TB of data that I'm trying to run through attic.
> >> >     I've tried with 0.14 and with git master. Both times, as the index
> >> >     reaches about 2.5GB, the backup job fails with:
> >> >     hashindex: data.attic/index.349400: Incorrect file
> >> >     length                              Traceback (most recent call
> >> >     last):
> >> >     File "/usr/local/bin/attic", line 4, in
> >> >     <module>                                             &n bsp;
> >> >     __import__('pkg_resources').run_script('Attic===0.14-3-g77bd65a',
> >> >     'attic')                      File
> >> >
>  "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> >> > line
> >> >     652, in run_script
> >> >     self.require(requires)[0].run_script(script_name,
> >> >     ns)                                           File
> >> >
>  "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> >> > line
> >> >     1573, in run_script
> >> >
> >> >     exec(code, namespace,
> >> >     namespace)
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd
> >> >     64.egg/EGG-INFO/scripts/attic", line 3, in
> >> >     <module>
> >> >     main()
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/arch
> >> >     iver.py", line 727, in main
> >> >                      exit_code =
> >> >     archiver.run(sys.argv[1:])
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 717, in
> >> >     run                                                        return
> >> >     args.func(args)
> >> >
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 128, in
> >> >     do_create
> >> >     self._process(archive, cache, args.excludes, args.exclude_caches,
> >> >     skip_inodes, path, restrict_dev)        ;
> >> >
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 177, in
> >> >     _process
> >> >     os.path.join(path, filename), restrict_dev) &
> >> >     nbsp;                                                   File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 177, in
> >> >     _process
> >> >     os.path.join(path, filename),
> >> >     restrict_dev)                                 &n
> >> >     bsp;                   File "/usr/lo
> >> >
> >> >
> 
cal/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 177, in
> >> >     _process
> >> >     os.path.join(path, filename),
> >> >     restrict_dev)
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 177, in _process          
> >> >     ;                                         os.path.join(path,
> >> >     filename),
> >> >     restrict_dev)
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 177, in _process
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> >> >     line 163, in
> >> >     _process
> >> >     archive.process_file(path, st,
> >> >     cache)
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> >> >     line 416, in process_file  &
> >> >     nbsp;
> >> >     self.add_item(item)
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> >> >     line 179, in add_item
> >> >
> >> >     self.write_checkpoint()
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> >> >     line 185, in write_checkpoint              &n
> >> >     bsp;                             self.cache.chunk_decref(self.id
> >> >     <http://self.id/>,
> >> >     self.stats)
> >> >     File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> >> >     line 202, in
> >> >     chunk_decref
> >> >     self.repository.delete(id, wait=False)
> >> >                                                            File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> >> >     line 365, in
> >> >     delete
> >> >     self.prepare_txn(self.get_transaction_id())
> >> > &
> >> >     nbsp;                      File
> >> >
> >> >
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> >> >     line 143, in
> >> >     prepare_txn
> >> >     self.index =
> >> >     self.open_index(transaction_id)
> >> >     File
> >> >
> >> >
> "/usr/local/lib/python3.3/site-packages/Attic-0.14_3_g77bd65a-py3.3-free
> >> >     bsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py", line 137, in
> >> >     open_index                                               return
> >> >     NSIndex.read((os.path.join(self.path, 'index.%d') %
> >> >     transaction_id).encode('utf-8'))     File "attic/hashindex.pyx",
> >> >     line 47, in attic.hashindex.IndexBase.read
> >> >     (attic/hashindex.c:1128)   File "attic/hashindex.pyx", line 35, in
> >> >     attic.hashindex.IndexBase.__cinit__
> >> >     (attic/hashindex.c:955)
> >> >  
> >> >     ;                                         Exception:
> >> >     hashindex_read failed
> >> >     I've tried this twice on different machines, once with a remote
> >> >     target and once locally.
> >> >     Seems similar to that described here:
> >> >
> >> >
> 
http://librelist.com/browser//attic/2014/11/19/crash-resuming-an-interrupted-backup/#42ead0e02028650448ba4f387c768bd4
> >> >     I haven't found any other reference to hashindex file length
> issues.
> >> >     Any insight appreciated.
> >> >
> >> >
> >>
> >>
> >
> >
>
>
>
> --
> Thiago Coutinho
>
> "O povo não deveria temer o governo. O governo é quem deveria temer o
> povo."
> V de Vingança
>

Re: Sv: [attic] Does attic have an upper data size limit?

From:
Jonas Borgström
Date:
2015-01-10 @ 19:19
On 2015-01-09 04:34, William Eshagh wrote:
> Hit the same issue, right at about the same point in the backup:
>
> hashindex: data.attic/index.349366: Incorrect file length
> Traceback (most recent call last):
>   File "/usr/local/bin/attic", line 4, in <module>
>     __import__('pkg_resources').run_script('Attic===0.14-5-g4ab4ecc',
> 'attic')
>   File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line 652, in run_script
>     self.require(requires)[0].run_script(script_name, ns)
>   File
> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
> line 1573, in run_script
>     exec(code, namespace, namespace)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
> line 3, in <module>
>     main()
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 727, in main
>     exit_code = archiver.run(sys.argv[1:])
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 717, in run
>     return args.func(args)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 128, in do_create
>     self._process(archive, cache, args.excludes, args.exclude_caches,
> skip_inodes, path, restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 177, in _process
>     os.path.join(path, filename), restrict_dev)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
> line 163, in _process
>     archive.process_file(path, st, cache)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 416, in process_file
>     self.add_item(item)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 179, in add_item
>     self.write_checkpoint()
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
> line 185, in write_checkpoint
>     self.cache.chunk_decref(self.id, self.stats)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
> line 202, in chunk_decref
>     self.repository.delete(id, wait=False)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 365, in delete
>     self.prepare_txn(self.get_transaction_id())
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 143, in prepare_txn
>     self.index = self.open_index(transaction_id)
>   File
> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
> line 137, in open_index
>     return NSIndex.read((os.path.join(self.path, 'index.%d') %
> transaction_id).encode('utf-8'))
>   File "attic/hashindex.pyx", line 47, in
> attic.hashindex.IndexBase.read (attic/hashindex.c:1128)
>   File "attic/hashindex.pyx", line 35, in
> attic.hashindex.IndexBase.__cinit__ (attic/hashindex.c:955)
> Exception: hashindex_read failed
>
> The index file is the same size as last time, 2.5GB. Could something
> else be contributing to the hashindex issue?
That's strange, the patch should really fix this.
Can you verify that the python binary module called
"hashindex.cpython-34m.so" or something similar was actually recompiled
after the update (compare the timestamp with _hashindex.c).

You can force setup.py to recompile all modules by running "touch
attic/*.pyx" and then reinstall.

/ Jonas

Re: Sv: [attic] Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-12 @ 05:26
Not sure if it had been rebuilt, but its been rebuilt now. Just kicked
off another full backup and will update in a few days after it completes.

Thanks Jonas!

On 01/10/2015 11:19 AM, Jonas Borgström wrote:
> On 2015-01-09 04:34, William Eshagh wrote:
>> Hit the same issue, right at about the same point in the backup:
>>
>> hashindex: data.attic/index.349366: Incorrect file length
>> Traceback (most recent call last):
>>   File "/usr/local/bin/attic", line 4, in <module>
>>     __import__('pkg_resources').run_script('Attic===0.14-5-g4ab4ecc',
>> 'attic')
>>   File
>> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>> line 652, in run_script
>>     self.require(requires)[0].run_script(script_name, ns)
>>   File
>> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>> line 1573, in run_script
>>     exec(code, namespace, namespace)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
>> line 3, in <module>
>>     main()
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 727, in main
>>     exit_code = archiver.run(sys.argv[1:])
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 717, in run
>>     return args.func(args)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 128, in do_create
>>     self._process(archive, cache, args.excludes, args.exclude_caches,
>> skip_inodes, path, restrict_dev)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 177, in _process
>>     os.path.join(path, filename), restrict_dev)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 177, in _process
>>     os.path.join(path, filename), restrict_dev)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 177, in _process
>>     os.path.join(path, filename), restrict_dev)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 177, in _process
>>     os.path.join(path, filename), restrict_dev)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 177, in _process
>>     os.path.join(path, filename), restrict_dev)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>> line 163, in _process
>>     archive.process_file(path, st, cache)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>> line 416, in process_file
>>     self.add_item(item)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>> line 179, in add_item
>>     self.write_checkpoint()
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>> line 185, in write_checkpoint
>>     self.cache.chunk_decref(self.id, self.stats)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
>> line 202, in chunk_decref
>>     self.repository.delete(id, wait=False)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>> line 365, in delete
>>     self.prepare_txn(self.get_transaction_id())
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>> line 143, in prepare_txn
>>     self.index = self.open_index(transaction_id)
>>   File
>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>> line 137, in open_index
>>     return NSIndex.read((os.path.join(self.path, 'index.%d') %
>> transaction_id).encode('utf-8'))
>>   File "attic/hashindex.pyx", line 47, in
>> attic.hashindex.IndexBase.read (attic/hashindex.c:1128)
>>   File "attic/hashindex.pyx", line 35, in
>> attic.hashindex.IndexBase.__cinit__ (attic/hashindex.c:955)
>> Exception: hashindex_read failed
>>
>> The index file is the same size as last time, 2.5GB. Could something
>> else be contributing to the hashindex issue?
> That's strange, the patch should really fix this.
> Can you verify that the python binary module called
> "hashindex.cpython-34m.so" or something similar was actually recompiled
> after the update (compare the timestamp with _hashindex.c).
>
> You can force setup.py to recompile all modules by running "touch
> attic/*.pyx" and then reinstall.
>
> / Jonas
>
>

Re: Sv: [attic] Does attic have an upper data size limit?

From:
William Eshagh
Date:
2015-01-14 @ 03:54
Success!

------------------------------------------------------------------------------
Archive name: 20150111
Archive fingerprint:
8bed1da1850c98d0458791cf333cfdc6eece4f81e27a2112346c33a3c18b7f24
Start time: Sun Jan 11 19:08:59 2015
End time: Tue Jan 13 19:20:10 2015
Duration: 2 days 11 minutes 11.18 seconds
Number of files: 1194402

                       Original size      Compressed size   
Deduplicated size
This archive:                3.20 TB              2.88 TB             
2.37 TB
All archives:                3.17 TB              2.88 TB             
2.37 TB
------------------------------------------------------------------------------

Thanks Jonas, recompile was the issue.


On 01/11/2015 09:26 PM, William Eshagh wrote:
> Not sure if it had been rebuilt, but its been rebuilt now. Just kicked
> off another full backup and will update in a few days after it completes.
>
> Thanks Jonas!
>
> On 01/10/2015 11:19 AM, Jonas Borgström wrote:
>> On 2015-01-09 04:34, William Eshagh wrote:
>>> Hit the same issue, right at about the same point in the backup:
>>>
>>> hashindex: data.attic/index.349366: Incorrect file length
>>> Traceback (most recent call last):
>>>   File "/usr/local/bin/attic", line 4, in <module>
>>>     __import__('pkg_resources').run_script('Attic===0.14-5-g4ab4ecc',
>>> 'attic')
>>>   File
>>> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>>> line 652, in run_script
>>>     self.require(requires)[0].run_script(script_name, ns)
>>>   File
>>> "/usr/local/lib/python3.3/site-packages/pkg_resources/__init__.py",
>>> line 1573, in run_script
>>>     exec(code, namespace, namespace)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/EGG-INFO/scripts/attic",
>>> line 3, in <module>
>>>     main()
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 727, in main
>>>     exit_code = archiver.run(sys.argv[1:])
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 717, in run
>>>     return args.func(args)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 128, in do_create
>>>     self._process(archive, cache, args.excludes, args.exclude_caches,
>>> skip_inodes, path, restrict_dev)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 177, in _process
>>>     os.path.join(path, filename), restrict_dev)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 177, in _process
>>>     os.path.join(path, filename), restrict_dev)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 177, in _process
>>>     os.path.join(path, filename), restrict_dev)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 177, in _process
>>>     os.path.join(path, filename), restrict_dev)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 177, in _process
>>>     os.path.join(path, filename), restrict_dev)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archiver.py",
>>> line 163, in _process
>>>     archive.process_file(path, st, cache)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>>> line 416, in process_file
>>>     self.add_item(item)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>>> line 179, in add_item
>>>     self.write_checkpoint()
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/archive.py",
>>> line 185, in write_checkpoint
>>>     self.cache.chunk_decref(self.id, self.stats)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/cache.py",
>>> line 202, in chunk_decref
>>>     self.repository.delete(id, wait=False)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>>> line 365, in delete
>>>     self.prepare_txn(self.get_transaction_id())
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>>> line 143, in prepare_txn
>>>     self.index = self.open_index(transaction_id)
>>>   File
>>> 
"/usr/local/lib/python3.3/site-packages/Attic-0.14_5_g4ab4ecc-py3.3-freebsd-9.3-RELEASE-p5-amd64.egg/attic/repository.py",
>>> line 137, in open_index
>>>     return NSIndex.read((os.path.join(self.path, 'index.%d') %
>>> transaction_id).encode('utf-8'))
>>>   File "attic/hashindex.pyx", line 47, in
>>> attic.hashindex.IndexBase.read (attic/hashindex.c:1128)
>>>   File "attic/hashindex.pyx", line 35, in
>>> attic.hashindex.IndexBase.__cinit__ (attic/hashindex.c:955)
>>> Exception: hashindex_read failed
>>>
>>> The index file is the same size as last time, 2.5GB. Could something
>>> else be contributing to the hashindex issue?
>> That's strange, the patch should really fix this.
>> Can you verify that the python binary module called
>> "hashindex.cpython-34m.so" or something similar was actually recompiled
>> after the update (compare the timestamp with _hashindex.c).
>>
>> You can force setup.py to recompile all modules by running "touch
>> attic/*.pyx" and then reinstall.
>>
>> / Jonas
>>
>>
>