librelist archives

« back to archive

[Errno 35] Resource temporarily unavailable

[Errno 35] Resource temporarily unavailable

From:
Simon Cahuk
Date:
2014-06-11 @ 23:50
Hi!

I want to backup my files on my Mac to a remote host running FreeBSD 10 and
Attic.

I don't get any error with attic init but attic create has some issues:

attic: projects/textmate/Frameworks/layout/tests/t_basic_tree_numeric.cc:
[Errno 35] Resource temporarily unavailable

------------------------------------------------------------------------------

Archive name: test1

Archive fingerprint:
baea7fed2f36f2f9fc9f1c87941c5c8e931351b7d0b34ab7884d66487ce27951

Start time: Wed Jun 11 22:06:01 2014

End time: Wed Jun 11 23:17:20 2014

Duration: 1 hours 11 minutes 19.77 seconds

Number of files: 4962


                       Original size      Compressed size    Deduplicated
size

This archive:              475.36 MB            397.16 MB            390.44
MB

All archives:              475.36 MB            397.16 MB            390.44
MB

------------------------------------------------------------------------------

attic: Exiting with failure status due to previous errors


Thanks,


Simon

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Jonas Borgström
Date:
2014-06-12 @ 07:03
On 12/06/14 01:50, Simon Cahuk wrote:
> Hi!
> 
> I want to backup my files on my Mac to a remote host running FreeBSD 10
> and Attic.
> 
> I don't get any error with attic init but attic create has some issues: 
> 
> attic:
> projects/textmate/Frameworks/layout/tests/t_basic_tree_numeric.cc:
> [Errno 35] Resource temporarily unavailable

Interesting, I have a couple of questions that would help me figure this
out:

- What version of Attic do you have?
- What version of Python do you have?
- Can you reliably reproduce this? And is it always the same file?
- Is the t_basic_tree_numeric.cc file currently open or "special" in any
other way?
- More than 1h to transfer 390MB sounds slow (1Mbit/s), what's you
upstream bandwidth?

/ Jonas

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Simon Cahuk
Date:
2014-06-12 @ 10:01
It is the latest git version on both local and remote.
Python 3.4 on both local and remote.
t_basic_tree_numeric.cc is one of the files I wanted to backup. I get the
error for the whole set of files I want to upload (for every single file).
Yes, I can reproduce the error with every new attic --create
Yes, I have 1 mbit/s upload, so the upload time is normal.

To expand the story: I have also a local attic backup (not for the same set
of files) and I couldn't reproduce this error. I backup two of my folders
on my Mac and then upload the two attic repositories to rsync.net since the
latter does not support Attic.


On Thu, Jun 12, 2014 at 9:03 AM, Jonas Borgström <jonas@borgstrom.se> wrote:

> On 12/06/14 01:50, Simon Cahuk wrote:
> > Hi!
> >
> > I want to backup my files on my Mac to a remote host running FreeBSD 10
> > and Attic.
> >
> > I don't get any error with attic init but attic create has some issues:
> >
> > attic:
> > projects/textmate/Frameworks/layout/tests/t_basic_tree_numeric.cc:
> > [Errno 35] Resource temporarily unavailable
>
> Interesting, I have a couple of questions that would help me figure this
> out:
>
> - What version of Attic do you have?
> - What version of Python do you have?
> - Can you reliably reproduce this? And is it always the same file?
> - Is the t_basic_tree_numeric.cc file currently open or "special" in any
> other way?
> - More than 1h to transfer 390MB sounds slow (1Mbit/s), what's you
> upstream bandwidth?
>
> / Jonas
>
>

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Jonas Borgström
Date:
2014-06-12 @ 10:29
could not decode message

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Simon Cahuk
Date:
2014-06-12 @ 16:03
diff --git a/attic/archiver.py b/attic/archiver.py
index c17b487..7103746 100644
--- a/attic/archiver.py
+++ b/attic/archiver.py
@@ -148,6 +148,7 @@ Type "Yes I am sure" if you understand this and want 
to continue.\n""")
             st = os.lstat(path)
         except OSError as e:
             self.print_error('%s: %s', path, e)
+            raise
             return
         if (st.st_ino, st.st_dev) in skip_inodes:
             return
@@ -163,6 +164,7 @@ Type "Yes I am sure" if you understand this and want 
to continue.\n""")
                 archive.process_file(path, st, cache)
             except IOError as e:
                 self.print_error('%s: %s', path, e)
+                raise
         elif stat.S_ISDIR(st.st_mode):
             if exclude_caches and is_cachedir(path):
                 return
@@ -171,6 +173,7 @@ Type "Yes I am sure" if you understand this and want 
to continue.\n""")
                 entries = os.listdir(path)
             except OSError as e:
                 self.print_error('%s: %s', path, e)
+                raise
             else:
                 for filename in sorted(entries):
                     self._process(archive, cache, excludes, 
exclude_caches, skip_inodes,

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Simon Cahuk
Date:
2014-06-12 @ 17:05
Here is a log after applying your patch:

attic: projects/atom/spec/fixtures/git/working-dir/git.git/index: [Errno
35] Resource temporarily unavailable

Traceback (most recent call last):

  File "/usr/local/bin/attic", line 5, in <module>

    pkg_resources.run_script('Attic==0.12-35-g95162ce-dirty', 'attic')

  File "/usr/local/lib/python3.4/site-packages/pkg_resources.py", line 488,
in run_script

    self.require(requires)[0].run_script(script_name, ns)

  File "/usr/local/lib/python3.4/site-packages/pkg_resources.py", line
1338, in run_script

    execfile(script_filename, namespace, namespace)

  File "/usr/local/lib/python3.4/site-packages/pkg_resources.py", line 55,
in execfile

    exec(compile(open(fn).read(), fn, 'exec'), globs, locs)

  File

"/usr/local/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/EGG-INFO/scripts/attic",
line 3, in <module>

    main()

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 718, in main

    exit_code = archiver.run(sys.argv[1:])

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 708, in run

    return args.func(args)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 128, in do_create

    self._process(archive, cache, args.excludes, args.exclude_caches,
skip_inodes, path, restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 180, in _process

    os.path.join(path, filename), restrict_dev)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archiver.py",
line 164, in _process

    archive.process_file(path, st, cache)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/archive.py",
line 402, in process_file

    chunks.append(cache.add_chunk(self.key.id_hash(chunk), chunk,
self.stats))

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/cache.py",
line 180, in add_chunk

    self.repository.put(id, data, wait=False)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/remote.py",
line 227, in put

    return self.call('put', id_, data, wait=wait)

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/remote.py",
line 126, in call

    for resp in self.call_many(cmd, [args], **kw):

  File

"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/remote.py",
line 198, in call_many

    self.to_send = self.to_send[os.write(self.stdin_fd, self.to_send):]

BlockingIOError: [Errno 35] Resource temporarily unavailable

Traceback (most recent call last):

  File "/usr/local/bin/attic", line 3, in <module>

    main()

  File "/usr/local/lib/python3.4/site-packages/attic/archiver.py", line
715, in main

    exit_code = archiver.run(sys.argv[1:])

  File "/usr/local/lib/python3.4/site-packages/attic/archiver.py", line
705, in run

    return args.func(args)

  File "/usr/local/lib/python3.4/site-packages/attic/archiver.py", line 54,
in do_serve

    return
RepositoryServer(restrict_to_paths=args.restrict_to_paths).serve()

  File "/usr/local/lib/python3.4/site-packages/attic/remote.py", line 58,
in serve

    sys.stdout.flush()

BrokenPipeError: [Errno 32] Broken pipe

Exception ignored in: <_io.TextIOWrapper name='<stdout>'
encoding='US-ASCII'>

BrokenPipeError: [Errno 32] Broken pipe


On Thu, Jun 12, 2014 at 6:03 PM, Simon Cahuk <simon.cahuk@gmail.com> wrote:

> Thanks for the patch, applied it, already uploading.
> Will report what happens.
>
> I thing I know what caused the error (in some cases): metadata.
>
> For example:
>
> ls -l@ Downloads/latest/piwik/plugins/UserCountry/images/flags/do.png
>
>
> shows
>
>
> -rw-r--r--@ 1 master  staff  368 Dec 17 07:28
> Downloads/latest/piwik/plugins/UserCountry/images/flags/do.png
>
>  com.apple.quarantine 67
>
> So some "trouble" files have ext attr or resource forks.
>
> But not the file you mentioned:
>
> ls -l@ /Volumes/Macintosh\
> HD/Users/simon/projects/textmate/Frameworks/layout/tests/t_basic_tree_numeric.cc
>
>
> shows
>
>
> rw-r--r--  1 master  staff  6413 Sep 25  2012 /Volumes/Macintosh
> HD/Users/simon/projects/textmate/Frameworks/layout/tests/t_basic_tree_numeric.cc
> (no metadata).
>
>
>
>
>
> On Thu, Jun 12, 2014 at 12:29 PM, Jonas Borgström <jonas@borgstrom.se>
> wrote:
>
>> On 12/06/14 12:01, Simon Cahuk wrote:
>> > It is the latest git version on both local and remote.
>> > Python 3.4 on both local and remote.
>> > t_basic_tree_numeric.cc is one of the files I wanted to backup. I get
>> > the error for the whole set of files I want to upload (for every single
>> > file).
>> > Yes, I can reproduce the error with every new attic --create
>> > Yes, I have 1 mbit/s upload, so the upload time is normal.
>> >
>> > To expand the story: I have also a local attic backup (not for the same
>> > set of files) and I couldn't reproduce this error. I backup two of my
>> > folders on my Mac and then upload the two attic repositories to
>> > rsync.net <http://rsync.net> since the latter does not support Attic.
>>
>> Do you get the same error for every file (even the first one), or does
>> it start after a while?
>>
>> I've attached a patch, which hopefully will give you a more detailed
>> error message
>>
>> / Jonas
>>
>>
>> diff --git a/attic/archiver.py b/attic/archiver.py
>> index c17b487..7103746 100644
>> --- a/attic/archiver.py
>> +++ b/attic/archiver.py
>> @@ -148,6 +148,7 @@ Type "Yes I am sure" if you understand this and want
>> to continue.\n""")
>>              st = os.lstat(path)
>>          except OSError as e:
>>              self.print_error('%s: %s', path, e)
>> +            raise
>>              return
>>          if (st.st_ino, st.st_dev) in skip_inodes:
>>              return
>> @@ -163,6 +164,7 @@ Type "Yes I am sure" if you understand this and want
>> to continue.\n""")
>>                  archive.process_file(path, st, cache)
>>              except IOError as e:
>>                  self.print_error('%s: %s', path, e)
>> +                raise
>>          elif stat.S_ISDIR(st.st_mode):
>>              if exclude_caches and is_cachedir(path):
>>                  return
>> @@ -171,6 +173,7 @@ Type "Yes I am sure" if you understand this and want
>> to continue.\n""")
>>                  entries = os.listdir(path)
>>              except OSError as e:
>>                  self.print_error('%s: %s', path, e)
>> +                raise
>>              else:
>>                  for filename in sorted(entries):
>>                      self._process(archive, cache, excludes,
>> exclude_caches, skip_inodes,
>>
>>
>

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Jonas Borgström
Date:
2014-06-12 @ 18:42
could not decode message

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Simon Cahuk
Date:
2014-06-12 @ 19:46
OS X 10.9.2 and Homebrew.

I will try the patch. Thanks.


On Thu, Jun 12, 2014 at 8:42 PM, Jonas Borgström <jonas@borgstrom.se> wrote:

> On 2014-06-12 19:05, Simon Cahuk wrote:
> > Here is a log after applying your patch:
> >
> > attic: projects/atom/spec/fixtures/git/working-dir/git.git/index: [Errno
> > 35] Resource temporarily unavailable
>
> *snip*
>
> >   File
> >
> 
"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/remote.py",
> > line 198, in call_many
> >     self.to_send = self.to_send[os.write(self.stdin_fd, self.to_send):]
> > BlockingIOError: [Errno 35] Resource temporarily unavailable
>
> Strange, it looks like os.write can raise BlockingIOError even though
> select() indicates that the fd is writable.
>
> I've never seen this behavior before, and I have a OS X Mavericks system
> myself, that creates a remote backup every hour. What version of OS X do
> you have?
>
> Anyway, I've attached a patch that hopefully fixes this.
>
> / Jonas
>
>
> diff --git a/attic/remote.py b/attic/remote.py
> index 91de219..e4d75c5 100644
> --- a/attic/remote.py
> +++ b/attic/remote.py
> @@ -1,4 +1,5 @@
>  import fcntl
> +from io import BlockingIOError
>  import msgpack
>  import os
>  import select
> @@ -195,7 +196,12 @@ class RemoteRepository(object):
>                          self.to_send = msgpack.packb((1, self.msgid, cmd,
> args))
>
>                  if self.to_send:
> -                    self.to_send = self.to_send[os.write(self.stdin_fd,
> self.to_send):]
> +                    try:
> +                        self.to_send =
> self.to_send[os.write(self.stdin_fd, self.to_send):]
> +                    except BlockingIOError:
> +                        # io.write might raise BlockingIOError even
> though select indicates
> +                        # that the fd should be writable
> +                        pass
>                  if not self.to_send and not (calls or self.preload_ids):
>                      w_fds = []
>          self.ignore_responses |= set(waiting_for)
>
>

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Simon Cahuk
Date:
2014-06-13 @ 06:19
I confirm that your last patch solved my problem, no errors anymore.

Thanks for Attic. You are doing a fantastic job.


On Thu, Jun 12, 2014 at 9:46 PM, Simon Cahuk <simon.cahuk@gmail.com> wrote:

> OS X 10.9.2 and Homebrew.
>
> I will try the patch. Thanks.
>
>
> On Thu, Jun 12, 2014 at 8:42 PM, Jonas Borgström <jonas@borgstrom.se>
> wrote:
>
>> On 2014-06-12 19:05, Simon Cahuk wrote:
>> > Here is a log after applying your patch:
>> >
>> > attic: projects/atom/spec/fixtures/git/working-dir/git.git/index: [Errno
>> > 35] Resource temporarily unavailable
>>
>> *snip*
>>
>> >   File
>> >
>> 
"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/remote.py",
>> > line 198, in call_many
>> >     self.to_send = self.to_send[os.write(self.stdin_fd, self.to_send):]
>> > BlockingIOError: [Errno 35] Resource temporarily unavailable
>>
>> Strange, it looks like os.write can raise BlockingIOError even though
>> select() indicates that the fd is writable.
>>
>> I've never seen this behavior before, and I have a OS X Mavericks system
>> myself, that creates a remote backup every hour. What version of OS X do
>> you have?
>>
>> Anyway, I've attached a patch that hopefully fixes this.
>>
>> / Jonas
>>
>>
>> diff --git a/attic/remote.py b/attic/remote.py
>> index 91de219..e4d75c5 100644
>> --- a/attic/remote.py
>> +++ b/attic/remote.py
>> @@ -1,4 +1,5 @@
>>  import fcntl
>> +from io import BlockingIOError
>>  import msgpack
>>  import os
>>  import select
>> @@ -195,7 +196,12 @@ class RemoteRepository(object):
>>                          self.to_send = msgpack.packb((1, self.msgid,
>> cmd, args))
>>
>>                  if self.to_send:
>> -                    self.to_send = self.to_send[os.write(self.stdin_fd,
>> self.to_send):]
>> +                    try:
>> +                        self.to_send =
>> self.to_send[os.write(self.stdin_fd, self.to_send):]
>> +                    except BlockingIOError:
>> +                        # io.write might raise BlockingIOError even
>> though select indicates
>> +                        # that the fd should be writable
>> +                        pass
>>                  if not self.to_send and not (calls or self.preload_ids):
>>                      w_fds = []
>>          self.ignore_responses |= set(waiting_for)
>>
>>
>

Re: [attic] [Errno 35] Resource temporarily unavailable

From:
Jonas Borgström
Date:
2014-06-13 @ 18:11
On 2014-06-13 08:19, Simon Cahuk wrote:
> I confirm that your last patch solved my problem, no errors anymore.
> 
> Thanks for Attic. You are doing a fantastic job.

Thanks!

I've just pushed a slightly modified patch since the original one did
not work correctly with Python 3.2.

/ Jonas

> 
> 
> On Thu, Jun 12, 2014 at 9:46 PM, Simon Cahuk <simon.cahuk@gmail.com
> <mailto:simon.cahuk@gmail.com>> wrote:
> 
>     OS X 10.9.2 and Homebrew.
> 
>     I will try the patch. Thanks.
> 
> 
>     On Thu, Jun 12, 2014 at 8:42 PM, Jonas Borgström <jonas@borgstrom.se
>     <mailto:jonas@borgstrom.se>> wrote:
> 
>         On 2014-06-12 19:05, Simon Cahuk wrote:
>         > Here is a log after applying your patch:
>         >
>         > attic:
>         projects/atom/spec/fixtures/git/working-dir/git.git/index: [Errno
>         > 35] Resource temporarily unavailable
> 
>         *snip*
> 
>         >   File
>         >
>         
"/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Attic-0.12_35_g95162ce_dirty-py3.4-macosx-10.9-x86_64.egg/attic/remote.py",
>         > line 198, in call_many
>         >     self.to_send = self.to_send[os.write(self.stdin_fd,
>         self.to_send):]
>         > BlockingIOError: [Errno 35] Resource temporarily unavailable
> 
>         Strange, it looks like os.write can raise BlockingIOError even
>         though
>         select() indicates that the fd is writable.
> 
>         I've never seen this behavior before, and I have a OS X
>         Mavericks system
>         myself, that creates a remote backup every hour. What version of
>         OS X do
>         you have?
> 
>         Anyway, I've attached a patch that hopefully fixes this.
> 
>         / Jonas
> 
> 
>         diff --git a/attic/remote.py b/attic/remote.py
>         index 91de219..e4d75c5 100644
>         --- a/attic/remote.py
>         +++ b/attic/remote.py
>         @@ -1,4 +1,5 @@
>          import fcntl
>         +from io import BlockingIOError
>          import msgpack
>          import os
>          import select
>         @@ -195,7 +196,12 @@ class RemoteRepository(object):
>                                  self.to_send = msgpack.packb((1,
>         self.msgid, cmd, args))
> 
>                          if self.to_send:
>         -                    self.to_send =
>         self.to_send[os.write(self.stdin_fd, self.to_send):]
>         +                    try:
>         +                        self.to_send =
>         self.to_send[os.write(self.stdin_fd, self.to_send):]
>         +                    except BlockingIOError:
>         +                        # io.write might raise BlockingIOError
>         even though select indicates
>         +                        # that the fd should be writable
>         +                        pass
>                          if not self.to_send and not (calls or
>         self.preload_ids):
>                              w_fds = []
>                  self.ignore_responses |= set(waiting_for)
> 
> 
>