librelist archives

« back to archive

crash resuming an interrupted backup

crash resuming an interrupted backup

From:
Joel Martin
Date:
2014-11-19 @ 06:44
version 0.13 release

I was doing my initial backup and there was a reboot in the middle of it,
resuming afterwards resulted in the stack trace below.  It's my first
backup with attic so I'm likely holding it wrong but I can't figure out how.

here's what I've done.

export ATTIC_PASSPHRASE='
mypassphrase'
nohup attic create --stats   \
    /mnt/attic/omvdata.attic::omv-`date +%Y-%m-%d` \
    /media/c615b345-5b9c-498d-84b8-610b1fce3cfe     \
    --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Consume/tmp  \
    --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Virtual_Machines \
    2> start.stderr > start.stdout

# next day I forgot i was backing up and rebooted the machine, argh.
# but there's a checkpoint file, yay.

attic list /mnt/obnam_a/attic/omvdata.attic
Enter passphrase for /mnt/attic/omvdata.attic:
omv-2014-11-17.checkpoint            Tue Nov 18 11:23:43 2014

export ATTIC_PASSPHRASE='mypassphrase'
nohup attic create --stats /mnt/attic/omvdata.attic::omv-2014-11-17 \
    /media/c615b345-5b9c-498d-84b8-610b1fce3cfe \
    --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Consume/tmp \
    --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Virtual_Machines \
    2> restart.stderr > restart.stdout

# about 1.5 hours later

hashindex: /mnt/attic/omvdata.attic/index.tmp.tmp: Incorrect file length
Traceback (most recent call last):
  File "/usr/local/bin/attic", line 3, in <module>
    main()
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
715, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
705, in run
    return args.func(args)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
128, in do_create
    self._process(archive, cache, args.excludes, args.exclude_caches,
skip_inodes, path, restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
177, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
177, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
177, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
177, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
177, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
177, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.2/dist-packages/attic/archiver.py", line
163, in _process
    archive.process_file(path, st, cache)
  File "/usr/local/lib/python3.2/dist-packages/attic/archive.py", line 402,
in process_file
    chunks.append(cache.add_chunk(self.key.id_hash(chunk), chunk,
self.stats))
  File "/usr/local/lib/python3.2/dist-packages/attic/cache.py", line 180,
in add_chunk
    self.repository.put(id, data, wait=False)
  File "/usr/local/lib/python3.2/dist-packages/attic/repository.py", line
362, in put
    self.index[id] = segment, offset
  File "hashindex.pyx", line 104, in attic.hashindex.NSIndex.__setitem__
(attic/hashindex.c:2463)


attic list /mnt/attic/omvdata.attic
Enter passphrase for /mnt/attic/omvdata.attic:
omv-2014-11-17.checkpoint            Tue Nov 18 11:23:43 2014
omv-2014-11-17.checkpoint.1          Tue Nov 18 14:32:05 2014

Re: [attic] crash resuming an interrupted backup

From:
Jonas Borgström
Date:
2014-11-20 @ 20:56
On 2014-11-19 07:44, Joel Martin wrote:
> version 0.13 release
>
> I was doing my initial backup and there was a reboot in the middle of
> it, resuming afterwards resulted in the stack trace below.  It's my
> first backup with attic so I'm likely holding it wrong but I can't
> figure out how.
>
> here's what I've done.
>
> export ATTIC_PASSPHRASE='
> mypassphrase'
> nohup attic create --stats   \
>     /mnt/attic/omvdata.attic::omv-`date +%Y-%m-%d` \
>     /media/c615b345-5b9c-498d-84b8-610b1fce3cfe     \
>     --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Consume/tmp  \
>     --exclude
> /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Virtual_Machines \
>     2> start.stderr > start.stdout
>
> # next day I forgot i was backing up and rebooted the machine, argh.
> # but there's a checkpoint file, yay.
>
> attic list /mnt/obnam_a/attic/omvdata.attic
> Enter passphrase for /mnt/attic/omvdata.attic:
> omv-2014-11-17.checkpoint            Tue Nov 18 11:23:43 2014
>
> export ATTIC_PASSPHRASE='mypassphrase'
> nohup attic create --stats /mnt/attic/omvdata.attic::omv-2014-11-17 \
>     /media/c615b345-5b9c-498d-84b8-610b1fce3cfe \
>     --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Consume/tmp \
>     --exclude
> /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Virtual_Machines \
>     2> restart.stderr > restart.stdout
>
> # about 1.5 hours later
>
> hashindex: /mnt/attic/omvdata.attic/index.tmp.tmp: Incorrect file length

It looks attic attempted to extend the index size but that somehow failed.
If this file still there? How large is it? It would also be interesting
if you could send me that file (or at least the first 100 bytes of it).

You didn't happen to fill the filesystem?

/ Jonas

Re: [attic] crash resuming an interrupted backup

From:
Joel Martin
Date:
2014-11-20 @ 21:22
ls -l /mnt/attic/omvdata.attic/
total 5244964
-rw-r--r--  1 root root        148 Nov 17 14:19 config
drwxr-xr-x 40 root root       4096 Nov 18 14:26 data
-rw-r--r--  1 root root    2095139 Nov 18 14:32 hints.371539
-rw-r--r--  1 root root 1342177298 Nov 18 14:32 index.371539
-rw-r--r--  1 root root 1342177298 Nov 18 14:34 index.tmp
-rw-r--r--  1 root root 2684354578 Nov 18 14:34 index.tmp.tmp
-rw-r--r--  1 root root         28 Nov 17 14:19 README

df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       5.5T  1.9T  3.4T  36% /mnt/attic

100 bytes sent from index.tmp and index.tmp.tmp

On Thu, Nov 20, 2014 at 12:56 PM, Jonas Borgström <jonas@borgstrom.se>
wrote:

> On 2014-11-19 07:44, Joel Martin wrote:
> > version 0.13 release
> >
> > I was doing my initial backup and there was a reboot in the middle of
> > it, resuming afterwards resulted in the stack trace below.  It's my
> > first backup with attic so I'm likely holding it wrong but I can't
> > figure out how.
> >
> > here's what I've done.
> >
> > export ATTIC_PASSPHRASE='
> > mypassphrase'
> > nohup attic create --stats   \
> >     /mnt/attic/omvdata.attic::omv-`date +%Y-%m-%d` \
> >     /media/c615b345-5b9c-498d-84b8-610b1fce3cfe     \
> >     --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Consume/tmp  \
> >     --exclude
> > /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Virtual_Machines \
> >     2> start.stderr > start.stdout
> >
> > # next day I forgot i was backing up and rebooted the machine, argh.
> > # but there's a checkpoint file, yay.
> >
> > attic list /mnt/obnam_a/attic/omvdata.attic
> > Enter passphrase for /mnt/attic/omvdata.attic:
> > omv-2014-11-17.checkpoint            Tue Nov 18 11:23:43 2014
> >
> > export ATTIC_PASSPHRASE='mypassphrase'
> > nohup attic create --stats /mnt/attic/omvdata.attic::omv-2014-11-17 \
> >     /media/c615b345-5b9c-498d-84b8-610b1fce3cfe \
> >     --exclude /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Consume/tmp \
> >     --exclude
> > /media/c615b345-5b9c-498d-84b8-610b1fce3cfe/Virtual_Machines \
> >     2> restart.stderr > restart.stdout
> >
> > # about 1.5 hours later
> >
> > hashindex: /mnt/attic/omvdata.attic/index.tmp.tmp: Incorrect file length
>
> It looks attic attempted to extend the index size but that somehow failed.
> If this file still there? How large is it? It would also be interesting
> if you could send me that file (or at least the first 100 bytes of it).
>
> You didn't happen to fill the filesystem?
>
> / Jonas
>
>

Re: [attic] crash resuming an interrupted backup

From:
Jonas Borgström
Date:
2014-11-20 @ 21:58
could not decode message

Re: [attic] crash resuming an interrupted backup

From:
Joel Martin
Date:
2014-11-21 @ 21:32
It's only about 3 tb, there is a plex library on it so a few hundred
thousand files as plex like its files

the patch segfaults, first time I ran it the next command also pruned the
database fwiw, reran to get backtrace

patch -p1 < ../_hashindex.c.patch
easy_install3 .

# double checking after first segfault, looks like yes, i did apply and
install the patch
diff _hashindex.c ../../../Attic-0.13/attic/_hashindex.c
198c198
<     if(length != sizeof(HashHeader) + _le32toh(header->num_buckets) *
(header->key_size + header->value_size)) {
---
>     if(length != sizeof(HashHeader) +
((off_t)_le32toh(header->num_buckets)) * (header->key_size +
header->value_size)) {

-rwxr-xr-x 1 root staff 297879 Nov 20 22:47
/usr/local/lib/python3.2/dist-packages/Attic-0.13-py3.2-linux-x86_64.egg/attic/
hashindex.cpython-32mu.so

Program received signal SIGSEGV, Segmentation fault.
0x00007f130f5380f0 in hashindex_lookup (index=index@entry=0x8840c20,
key=key@entry=0x7f12112820b2) at attic/_hashindex.c:89
89            if(BUCKET_IS_EMPTY(index, idx))
(gdb) backtrace
#0  0x00007f130f5380f0 in hashindex_lookup (index=index@entry=0x8840c20,
key=key@entry=0x7f12112820b2) at attic/_hashindex.c:89
#1  0x00007f130f538dd8 in hashindex_set (index=0x8840c20,
key=0x7f12112820b2, value=0x7f12112820d2) at attic/_hashindex.c:325
#2  0x00007f130f538f51 in hashindex_resize (index=index@entry=0x173580d0,
capacity=67108864) at attic/_hashindex.c:128
#3  0x00007f130f538ebf in hashindex_set (index=0x173580d0, key=0x41bf1170,
value=0x7fff582b5f10) at attic/_hashindex.c:330
#4  0x00007f130f53ce39 in __pyx_pf_5attic_9hashindex_7NSIndex_2__setitem__
(__pyx_v_value=<optimized out>, __pyx_v_key=0x41bf1150,
__pyx_v_self=<optimized out>) at attic/hashindex.c:2449
#5  __pyx_pw_5attic_9hashindex_7NSIndex_3__setitem__
(__pyx_v_value=<optimized out>, __pyx_v_key=0x41bf1150,
__pyx_v_self=0x7f13100978a0) at attic/hashindex.c:2377
#6  __pyx_mp_ass_subscript_5attic_9hashindex_NSIndex (o=0x7f13100978a0,
i=0x41bf1150, v=<optimized out>) at attic/hashindex.c:4055
#7  0x00000000004b748b in PyEval_EvalFrameEx ()
#8  0x00000000004cdee7 in PyEval_EvalCodeEx ()
#9  0x00000000004ba764 in PyEval_EvalFrameEx ()
#10 0x00000000004ba36e in PyEval_EvalFrameEx ()
#11 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#12 0x00000000004ba764 in PyEval_EvalFrameEx ()
#13 0x00000000004ba36e in PyEval_EvalFrameEx ()
#14 0x00000000004ba36e in PyEval_EvalFrameEx ()
#15 0x00000000004ba36e in PyEval_EvalFrameEx ()
#16 0x00000000004ba36e in PyEval_EvalFrameEx ()
#17 0x00000000004ba36e in PyEval_EvalFrameEx ()
#18 0x00000000004ba36e in PyEval_EvalFrameEx ()
#19 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#20 0x00000000004ba764 in PyEval_EvalFrameEx ()
#21 0x00000000004ba36e in PyEval_EvalFrameEx ()
#22 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#23 0x0000000000557e2d in ?? ()
#24 0x00000000004b666f in PyEval_EvalFrameEx ()
#25 0x00000000004ba36e in PyEval_EvalFrameEx ()
#26 0x00000000004ba36e in PyEval_EvalFrameEx ()
#27 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#28 0x00000000005464bf in ?? ()
#29 0x000000000045420c in PyRun_FileExFlags ()
#30 0x000000000045452c in PyRun_SimpleFileExFlags ()
#31 0x00000000005587b5 in Py_Main ()
#32 0x00000000004616eb in main ()



On Thu, Nov 20, 2014 at 1:58 PM, Jonas Borgström <jonas@borgstrom.se> wrote:

> On 2014-11-20 22:22, Joel Martin wrote:
> > ls -l /mnt/attic/omvdata.attic/
> > total 5244964
> > -rw-r--r--  1 root root        148 Nov 17 14:19 config
> > drwxr-xr-x 40 root root       4096 Nov 18 14:26 data
> > -rw-r--r--  1 root root    2095139 Nov 18 14:32 hints.371539
> > -rw-r--r--  1 root root 1342177298 Nov 18 14:32 index.371539
> > -rw-r--r--  1 root root 1342177298 Nov 18 14:34 index.tmp
> > -rw-r--r--  1 root root 2684354578 Nov 18 14:34 index.tmp.tmp
>
> Looks like there's an issue with index files larger than 2GB. May I ask
> how big your repository is? :)
>
> Anyway, the GIT version of Attic uses a new hashindex implementation
> which should not be affected by this.
>
> I've also attached a patch which hopefully fixes this for 0.13 as well.
>
> / Jonas
>
>

Re: [attic] crash resuming an interrupted backup

From:
Joel Martin
Date:
2014-11-22 @ 07:59
should have added, i'm just keeping this around in case you want to debug,
it is my first backup so no harm in starting over.

On Fri, Nov 21, 2014 at 1:32 PM, Joel Martin <blurbling@gmail.com> wrote:

> It's only about 3 tb, there is a plex library on it so a few hundred
> thousand files as plex like its files
>
> the patch segfaults, first time I ran it the next command also pruned the
> database fwiw, reran to get backtrace
>
> patch -p1 < ../_hashindex.c.patch
> easy_install3 .
>
> # double checking after first segfault, looks like yes, i did apply and
> install the patch
> diff _hashindex.c ../../../Attic-0.13/attic/_hashindex.c
> 198c198
> <     if(length != sizeof(HashHeader) + _le32toh(header->num_buckets) *
> (header->key_size + header->value_size)) {
> ---
> >     if(length != sizeof(HashHeader) +
> ((off_t)_le32toh(header->num_buckets)) * (header->key_size +
> header->value_size)) {
>
> -rwxr-xr-x 1 root staff 297879 Nov 20 22:47
> /usr/local/lib/python3.2/dist-packages/Attic-0.13-py3.2-linux-x86_64.egg/attic/
> hashindex.cpython-32mu.so
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007f130f5380f0 in hashindex_lookup (index=index@entry=0x8840c20,
> key=key@entry=0x7f12112820b2) at attic/_hashindex.c:89
> 89            if(BUCKET_IS_EMPTY(index, idx))
> (gdb) backtrace
> #0  0x00007f130f5380f0 in hashindex_lookup (index=index@entry=0x8840c20,
> key=key@entry=0x7f12112820b2) at attic/_hashindex.c:89
> #1  0x00007f130f538dd8 in hashindex_set (index=0x8840c20,
> key=0x7f12112820b2, value=0x7f12112820d2) at attic/_hashindex.c:325
> #2  0x00007f130f538f51 in hashindex_resize (index=index@entry=0x173580d0,
> capacity=67108864) at attic/_hashindex.c:128
> #3  0x00007f130f538ebf in hashindex_set (index=0x173580d0, key=0x41bf1170,
> value=0x7fff582b5f10) at attic/_hashindex.c:330
> #4  0x00007f130f53ce39 in __pyx_pf_5attic_9hashindex_7NSIndex_2__setitem__
> (__pyx_v_value=<optimized out>, __pyx_v_key=0x41bf1150,
> __pyx_v_self=<optimized out>) at attic/hashindex.c:2449
> #5  __pyx_pw_5attic_9hashindex_7NSIndex_3__setitem__
> (__pyx_v_value=<optimized out>, __pyx_v_key=0x41bf1150,
> __pyx_v_self=0x7f13100978a0) at attic/hashindex.c:2377
> #6  __pyx_mp_ass_subscript_5attic_9hashindex_NSIndex (o=0x7f13100978a0,
> i=0x41bf1150, v=<optimized out>) at attic/hashindex.c:4055
> #7  0x00000000004b748b in PyEval_EvalFrameEx ()
> #8  0x00000000004cdee7 in PyEval_EvalCodeEx ()
> #9  0x00000000004ba764 in PyEval_EvalFrameEx ()
> #10 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #11 0x00000000004cdee7 in PyEval_EvalCodeEx ()
> #12 0x00000000004ba764 in PyEval_EvalFrameEx ()
> #13 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #14 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #15 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #16 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #17 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #18 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #19 0x00000000004cdee7 in PyEval_EvalCodeEx ()
> #20 0x00000000004ba764 in PyEval_EvalFrameEx ()
> #21 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #22 0x00000000004cdee7 in PyEval_EvalCodeEx ()
> #23 0x0000000000557e2d in ?? ()
> #24 0x00000000004b666f in PyEval_EvalFrameEx ()
> #25 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #26 0x00000000004ba36e in PyEval_EvalFrameEx ()
> #27 0x00000000004cdee7 in PyEval_EvalCodeEx ()
> #28 0x00000000005464bf in ?? ()
> #29 0x000000000045420c in PyRun_FileExFlags ()
> #30 0x000000000045452c in PyRun_SimpleFileExFlags ()
> #31 0x00000000005587b5 in Py_Main ()
> #32 0x00000000004616eb in main ()
>
>
>
> On Thu, Nov 20, 2014 at 1:58 PM, Jonas Borgström <jonas@borgstrom.se>
> wrote:
>
>> On 2014-11-20 22:22, Joel Martin wrote:
>> > ls -l /mnt/attic/omvdata.attic/
>> > total 5244964
>> > -rw-r--r--  1 root root        148 Nov 17 14:19 config
>> > drwxr-xr-x 40 root root       4096 Nov 18 14:26 data
>> > -rw-r--r--  1 root root    2095139 Nov 18 14:32 hints.371539
>> > -rw-r--r--  1 root root 1342177298 Nov 18 14:32 index.371539
>> > -rw-r--r--  1 root root 1342177298 Nov 18 14:34 index.tmp
>> > -rw-r--r--  1 root root 2684354578 Nov 18 14:34 index.tmp.tmp
>>
>> Looks like there's an issue with index files larger than 2GB. May I ask
>> how big your repository is? :)
>>
>> Anyway, the GIT version of Attic uses a new hashindex implementation
>> which should not be affected by this.
>>
>> I've also attached a patch which hopefully fixes this for 0.13 as well.
>>
>> / Jonas
>>
>>
>

Re: [attic] crash resuming an interrupted backup

From:
Jonas Borgström
Date:
2014-11-22 @ 13:33
could not decode message

Re: [attic] crash resuming an interrupted backup

From:
Joel Martin
Date:
2014-11-23 @ 02:21
Patched version seems happy, I'm probably going to rethink this if large
directories are expensive for deduplicating.  the server has 32gb and I
haven't seen attic go over 2 or 3 but there's little sense in comparing an
itunes library with a lightroom library for deduplication.  Think i'll
restart with it split into sensible divisions.
thanks for the quick patch.

On Sat, Nov 22, 2014 at 5:33 AM, Jonas Borgström <jonas@borgstrom.se> wrote:

> On 2014-11-22 08:59, Joel Martin wrote:
> > should have added, i'm just keeping this around in case you want to
> > debug, it is my first backup so no harm in starting over.
>
> Sorry about that. The patch I sent you was incomplete. I've attached a
> new version.
>
> It's usually a good idea to do "touch attic/*.pyx" before installing
> after applying a patch to make sure everything is recompiled correctly.
>
> >
> > On Fri, Nov 21, 2014 at 1:32 PM, Joel Martin <blurbling@gmail.com
> > <mailto:blurbling@gmail.com>> wrote:
> >
> >     It's only about 3 tb, there is a plex library on it so a few
> >     hundred thousand files as plex like its files
> >
> That's probably some kind of record anyway, otherwise other people would
> have run into the same issue. It will be interesting to see what other
> kinds of problems you will run into. I hope you have a lot of ram.
> Deduplicating 3TB of data will consume a lot of memory for sure.
>
> / Jonas
>
>

Re: [attic] crash resuming an interrupted backup

From:
Joel Martin
Date:
2014-11-23 @ 04:31
argh, jinxed myself!   let it run for 5 hours then noticed it segfaulted,
did touch the pyx's and diff showed i applied the patch.

I'm going to restart from scratch with a better strategy, that's needed
regardless of efficiency so I don't outgrow my target disk.

Program received signal SIGSEGV, Segmentation fault.
0x00007f9bc581a0f0 in hashindex_lookup (index=index@entry=0xde31820,
key=key@entry=0x7f9ac62310b2) at attic/_hashindex.c:89
89            if(BUCKET_IS_EMPTY(index, idx))
(gdb) backtrace
#0  0x00007f9bc581a0f0 in hashindex_lookup (index=index@entry=0xde31820,
key=key@entry=0x7f9ac62310b2) at attic/_hashindex.c:89
#1  0x00007f9bc581add8 in hashindex_set (index=0xde31820,
key=0x7f9ac62310b2, value=0x7f9ac62310d2) at attic/_hashindex.c:325
#2  0x00007f9bc581af51 in hashindex_resize (index=index@entry=0x1768bae0,
capacity=67108864) at attic/_hashindex.c:128
#3  0x00007f9bc581aebf in hashindex_set (index=0x1768bae0, key=0x1cdc5248,
value=0x7fff05603040) at attic/_hashindex.c:330
#4  0x00007f9bc581ee39 in __pyx_pf_5attic_9hashindex_7NSIndex_2__setitem__
(__pyx_v_value=<optimized out>, __pyx_v_key=0x1cdc5228,
__pyx_v_self=<optimized out>) at attic/hashindex.c:2449
#5  __pyx_pw_5attic_9hashindex_7NSIndex_3__setitem__
(__pyx_v_value=<optimized out>, __pyx_v_key=0x1cdc5228,
__pyx_v_self=0x7f9bc6379c60) at attic/hashindex.c:2377
#6  __pyx_mp_ass_subscript_5attic_9hashindex_NSIndex (o=0x7f9bc6379c60,
i=0x1cdc5228, v=<optimized out>) at attic/hashindex.c:4055
#7  0x00000000004b748b in PyEval_EvalFrameEx ()
#8  0x00000000004cdee7 in PyEval_EvalCodeEx ()
#9  0x00000000004ba764 in PyEval_EvalFrameEx ()
#10 0x00000000004ba36e in PyEval_EvalFrameEx ()
#11 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#12 0x00000000004ba764 in PyEval_EvalFrameEx ()
#13 0x00000000004ba36e in PyEval_EvalFrameEx ()
#14 0x00000000004ba36e in PyEval_EvalFrameEx ()
#15 0x00000000004ba36e in PyEval_EvalFrameEx ()
#16 0x00000000004ba36e in PyEval_EvalFrameEx ()
#17 0x00000000004ba36e in PyEval_EvalFrameEx ()
#18 0x00000000004ba36e in PyEval_EvalFrameEx ()
#19 0x00000000004ba36e in PyEval_EvalFrameEx ()
#20 0x00000000004ba36e in PyEval_EvalFrameEx ()
#21 0x00000000004ba36e in PyEval_EvalFrameEx ()
#22 0x00000000004ba36e in PyEval_EvalFrameEx ()
#23 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#24 0x00000000004ba764 in PyEval_EvalFrameEx ()
#25 0x00000000004ba36e in PyEval_EvalFrameEx ()
#26 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#27 0x0000000000557e2d in ?? ()
#28 0x00000000004b666f in PyEval_EvalFrameEx ()
#29 0x00000000004ba36e in PyEval_EvalFrameEx ()
#30 0x00000000004ba36e in PyEval_EvalFrameEx ()
#31 0x00000000004cdee7 in PyEval_EvalCodeEx ()
#32 0x00000000005464bf in ?? ()
#33 0x000000000045420c in PyRun_FileExFlags ()
#34 0x000000000045452c in PyRun_SimpleFileExFlags ()
#35 0x00000000005587b5 in Py_Main ()
#36 0x00000000004616eb in main ()


On Sat, Nov 22, 2014 at 6:21 PM, Joel Martin <blurbling@gmail.com> wrote:

> Patched version seems happy, I'm probably going to rethink this if large
> directories are expensive for deduplicating.  the server has 32gb and I
> haven't seen attic go over 2 or 3 but there's little sense in comparing an
> itunes library with a lightroom library for deduplication.  Think i'll
> restart with it split into sensible divisions.
> thanks for the quick patch.
>
> On Sat, Nov 22, 2014 at 5:33 AM, Jonas Borgström <jonas@borgstrom.se>
> wrote:
>
>> On 2014-11-22 08:59, Joel Martin wrote:
>> > should have added, i'm just keeping this around in case you want to
>> > debug, it is my first backup so no harm in starting over.
>>
>> Sorry about that. The patch I sent you was incomplete. I've attached a
>> new version.
>>
>> It's usually a good idea to do "touch attic/*.pyx" before installing
>> after applying a patch to make sure everything is recompiled correctly.
>>
>> >
>> > On Fri, Nov 21, 2014 at 1:32 PM, Joel Martin <blurbling@gmail.com
>> > <mailto:blurbling@gmail.com>> wrote:
>> >
>> >     It's only about 3 tb, there is a plex library on it so a few
>> >     hundred thousand files as plex like its files
>> >
>> That's probably some kind of record anyway, otherwise other people would
>> have run into the same issue. It will be interesting to see what other
>> kinds of problems you will run into. I hope you have a lot of ram.
>> Deduplicating 3TB of data will consume a lot of memory for sure.
>>
>> / Jonas
>>
>>
>