librelist archives

« back to archive

Install script that takes care of dependencies and environment set-up on OSX (w/or w/o Homebrew), or a detailed tutorial

Install script that takes care of dependencies and environment set-up on OSX (w/or w/o Homebrew), or a detailed tutorial

From:
Date:
2015-07-28 @ 04:37
Since it's a backup program and a quite famous one it would be really 
helpful if there was a script or package for various platform so that at 
least installation would be hassle free and even those people could 
benefit from it who are not that familiar with command line or find 
dealing with dependencies and install exception hard to deal with, 
especially in a programming language they are not familiar with.

I have struggled with installation for an hour before postponing it to 
some time later. Sometimes it was the import error, sometimes just a 
crash, then I could make it work with Homebrew's Python.

I think a script - that takes care of installation (including 
dependencies, python version mismatch, package requirements etc), or a 
full fledged package (for various platforms - I am on OSX Yosemite), or 
a tutorial that gives some steps to set-up Attic and have it working 
will really be helpful.

Are there such resources? Or can we put together one? Even though I have 
not set up mine yet (I have mentioned my errors at the bottom of this 
mail, not sure if it belonged to github issues), I am ready to work on 
this.

My set-up:

- OSX Yosemite
- Command Line Tools (no Xcode)
- Homebrew Python 3.x (thorws error while I tried to setup a virtualenv 
or even if I try to install Attic with pip3)
- System Python 2.x
- Couldn't install Attic (many errors; I couldn't make sense of most of 
them except that there some import error and maybe I need to have some 
packages that attic needs, but then I couldn't set-up either).


Yes, I am willing clean everything up and start afresh in a clean way. I 
have no Python app running from my side which depends either on Sytem or 
Homebrew python versions so I guess I can change/remove them if there is 
a way I get a fresh/cleaner Python/Attic set-up (other than having to 
reinstall the OS).




-------------------------
Here are the errors: (right after installing python3 using Homebrew)
-------------------------

amar@mac:.virtualenvs$ sudo pip3 install --upgrade pip setuptools
Traceback (most recent call last):
   File "/usr/local/bin/pip3", line 9, in <module>
     load_entry_point('pip==7.1.0', 'console_scripts', 'pip3')()
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
558, in load_entry_point
     return get_distribution(dist).load_entry_point(group, name)
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
2682, in load_entry_point
     return ep.load()
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
2355, in load
     return self.resolve()
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
2361, in resolve
     module = __import__(self.module_name, fromlist=['__name__'], 
level=0)
   File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 
15, in <module>
     from pip.vcs import git, mercurial, subversion, bazaar  # noqa
   File "/usr/local/lib/python3.4/site-packages/pip/vcs/subversion.py", 
line 9, in <module>
     from pip.index import Link
   File "/usr/local/lib/python3.4/site-packages/pip/index.py", line 30, 
in <module>
     from pip.wheel import Wheel, wheel_ext
   File "/usr/local/lib/python3.4/site-packages/pip/wheel.py", line 35, 
in <module>
     from pip._vendor.distlib.scripts import ScriptMaker
   File 
"/usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/scripts.py", 
line 14, in <module>
     from .compat import sysconfig, detect_encoding, ZipFile
   File 
"/usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/compat.py", 
line 66, in <module>
     from urllib.request import (urlopen, urlretrieve, Request, 
url2pathname,
ImportError: cannot import name 'HTTPSHandler'


amar@mac:.virtualenvs$ mkvirtualenv --python=/usr/local/bin/python3 
atticenv
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix 
'/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4'
New python executable in atticenv/bin/python3.4
Also creating executable in atticenv/bin/python
Installing setuptools, pip, wheel...
   Complete output from command 
/Users/amar/.virtual...ticenv/bin/python3.4 -c "import sys, pip; 
sys...d\"] + sys.argv[1:]))" setuptools pip wheel:
   Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File 

"/Library/Python/2.7/site-packages/virtualenv_support/pip-7.1.0-py2.py3-none-any.whl/pip/__init__.py",

line 15, in <module>
   File 

"/Library/Python/2.7/site-packages/virtualenv_support/pip-7.1.0-py2.py3-none-any.whl/pip/vcs/subversion.py",

line 9, in <module>
   File 

"/Library/Python/2.7/site-packages/virtualenv_support/pip-7.1.0-py2.py3-none-any.whl/pip/index.py",

line 30, in <module>
   File 

"/Library/Python/2.7/site-packages/virtualenv_support/pip-7.1.0-py2.py3-none-any.whl/pip/wheel.py",

line 35, in <module>
   File 

"/Library/Python/2.7/site-packages/virtualenv_support/pip-7.1.0-py2.py3-none-any.whl/pip/_vendor/distlib/scripts.py",

line 14, in <module>
   File 

"/Library/Python/2.7/site-packages/virtualenv_support/pip-7.1.0-py2.py3-none-any.whl/pip/_vendor/distlib/compat.py",

line 66, in <module>
ImportError: cannot import name 'HTTPSHandler'
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
   File "/Library/Python/2.7/site-packages/virtualenv.py", line 2363, in 
<module>
     main()
   File "/Library/Python/2.7/site-packages/virtualenv.py", line 832, in 
main
     symlink=options.symlink)
   File "/Library/Python/2.7/site-packages/virtualenv.py", line 1004, in 
create_environment
     install_wheel(to_install, py_executable, search_dirs)
   File "/Library/Python/2.7/site-packages/virtualenv.py", line 969, in 
install_wheel
     'PIP_NO_INDEX': '1'
   File "/Library/Python/2.7/site-packages/virtualenv.py", line 910, in 
call_subprocess
     % (cmd_desc, proc.returncode))
OSError: Command /Users/amar/.virtual...ticenv/bin/python3.4 -c "import 
sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with 
error code 1


amar@mac:.virtualenvs$ cd atticenv/


amar@mac:atticenv$ pip3 install Attic
Traceback (most recent call last):
   File "/usr/local/bin/pip3", line 9, in <module>
     load_entry_point('pip==7.1.0', 'console_scripts', 'pip3')()
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
558, in load_entry_point
     return get_distribution(dist).load_entry_point(group, name)
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
2682, in load_entry_point
     return ep.load()
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
2355, in load
     return self.resolve()
   File 
"/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 
2361, in resolve
     module = __import__(self.module_name, fromlist=['__name__'], 
level=0)
   File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 
15, in <module>
     from pip.vcs import git, mercurial, subversion, bazaar  # noqa
   File "/usr/local/lib/python3.4/site-packages/pip/vcs/subversion.py", 
line 9, in <module>
     from pip.index import Link
   File "/usr/local/lib/python3.4/site-packages/pip/index.py", line 30, 
in <module>
     from pip.wheel import Wheel, wheel_ext
   File "/usr/local/lib/python3.4/site-packages/pip/wheel.py", line 35, 
in <module>
     from pip._vendor.distlib.scripts import ScriptMaker
   File 
"/usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/scripts.py", 
line 14, in <module>
     from .compat import sysconfig, detect_encoding, ZipFile
   File 
"/usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/compat.py", 
line 66, in <module>
     from urllib.request import (urlopen, urlretrieve, Request, 
url2pathname,
ImportError: cannot import name 'HTTPSHandler'

amar@mac:atticenv$ pip install Attic
Collecting Attic

/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90:

InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause 
certain SSL connections to fail. For more information, see 
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
   InsecurePlatformWarning
   Downloading Attic-0.16.tar.gz (232kB)
     100% |████████████████████████████████| 233kB 90kB/s
     Complete output from command python setup.py egg_info:
     Attic requires Python 3.2 or later

     ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in 
/private/var/folders/58/rxkccgz17vx2ktfyn0chgrd00000gp/T/pip-build-rQ6MO6/Attic

Re: [attic] Install script that takes care of dependencies and environment set-up on OSX (w/or w/o Homebrew), or a detailed tutorial

From:
Alberto Luaces
Date:
2015-07-28 @ 07:31
I do not know anything about OSX, but if you do not even have a working
python3 environment, obviously it is not going to fly.

I would try first to get that environment.  Googling around I have seen
this

(http://www.marinamele.com/2014/07/install-python3-on-mac-os-x-and-use-virtualenv-and-virtualenvwrapper.html),
that seems to fit your very case.  Disregard the Xcode part.

--
Alberto

Re: [attic] Install script that takes care of dependencies and environment set-up on OSX (w/or w/o Homebrew), or a detailed tutorial

From:
Date:
2015-07-30 @ 01:14
Hi Alberto,

Thank you.

I already had a working set-up of Python3 (as I didn't have to reinstall 
it).

However my attempts to install other apps (e.g. `virtualenv`) were 
failing so I just used `pyvenv` to create a virtualenv and then install 
Attic in it which installed but with a lot of warnings: 
"-Wunused-function" and "-Wunneeded-internal-declaration" warnings.

It ended in:

> Successfully installed Attic-0.16 msgpack-python-0.4.6

Since attic installed in ~/.virtualenvs/attic/bin, I exported the path 
in .bash_profile.


TEST RUN:

> amar@mac:backup$ attic init --encryption=passphrase dropbox.attic
> ...
> Initializing cache...


and then ran an `attic create -e` on my ~/Dropbox folder.

> amar@mac:~$ attic create --stats -e=passphrase 
> backup/dropbox.attic::WeeklyDBox ~/Dropbox/

Entered the same passphrase that I used to create the repo. That's the 
idea, right?

> Warning: The repository at location backup/dropbox.attic was previously 
> located at dropbox.attic
> Do you want to continue? [yN] y

I don't understand this warning. Why was it shown?

> Archive name: WeeklyDBox
> ...
> Duration: 8 minutes 1.47 seconds
> Number of files: 74169
> 
>                       Original size      Compressed size    
> Deduplicated size
> This archive:                6.76 GB              5.28 GB              
> 4.78 GB
> All archives:                6.76 GB              5.28 GB              
> 4.78 GB

I was hoping for better de-duplication but at least backup worked. Or 
maybe I have to pass de-dup and compression parameters (?). 
(https://attic-backup.org/quickstart.html#quickstart page shows 
unbelievable de-duplication). Maybe it's because of the types of files 
I've in Dropbox.

I tried to extract/restore the files but I couldn't find an option in 
Docs which could help me to extract the backup to a location I wanted 
to, so I just renamed my original folder to Dropbox_old and ran:


> amar@mac:~$ attic extract backup/dropbox.attic::WeeklyDBox

Exited after some time. I couldn't get some `--stats` to run with 
`extract` unlike with `create`. It created the entire Folder structure 
inside user home as `/Users/amar/Users/amar/Dropbox`.

How do I modify folder structure for output of `extract` - as in what 
should be extracted to where and under what structure? I guess if I had 
ran this command from root folder `/` it would have been placed in its 
proper location.

Few more basic tests done: attic check, list, info and finally delete.

So can't I just install attic on my client machine and save/send backup 
files/chunks to a remote server which is a dumb storage since I am 
already encrypting/compressing on client side? Or I have to have attic 
installed on both the server and client?

Docs doesn't have usage examples for some commands. Going from the 
discussions in many Github issues, I was wondering whether someone 
inexperienced in Pyhton and Attic could make a pull request.

Also CPU usage on my 3 year old MacBook Air was around 80-90% almost all 
the time.

Anyway I think I will keep playing around with it (and maybe some other 
backup tools) and see where it goes.

Regards.


On 28.07.2015 1:01 pm, Alberto Luaces wrote:
> I do not know anything about OSX, but if you do not even have a working
> python3 environment, obviously it is not going to fly.
> 
> I would try first to get that environment.  Googling around I have seen
> this
> 
(http://www.marinamele.com/2014/07/install-python3-on-mac-os-x-and-use-virtualenv-and-virtualenvwrapper.html),
> that seems to fit your very case.  Disregard the Xcode part.
> 
> --
> Alberto