librelist archives

File Management Query

From:

Ishbir Singh

Date:

2011-08-12 @ 14:26

Hey all,

This is not directly a flask related question, but, nevertheless, I wanted
to get your views about transferring files between a processing server and a
self-owned file storage server. The file storage server has tons of
bandwidth and diskspace. It will be serving images.

What I'm trying to do is, a user upload files to processing server, which
does some manipulations and then puts it on the static file storage server
for retrieval by other users, deleting the local copy in the process. What
would be the best way to do this? What should I use for transferring files?
FTP or what?

-- 
Regards,
Ishbir Singh

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2011-08-12 16:26, Ishbir Singh wrote:

Hi,

> This is not directly a flask related question, but, nevertheless, I 
> wanted to get your views about transferring files between a 
> processing server and a self-owned file storage server. The file 
> storage server has tons of bandwidth and diskspace. It will be 
> serving images.

How fast and reliable is your network connection between the two systems?

> What I'm trying to do is, a user upload files to processing server, 
> which does some manipulations and then puts it on the static file 
> storage server for retrieval by other users, deleting the local copy 
> in the process. What would be the best way to do this? What should I 
> use for transferring files? FTP or what?

I've got a web app that pretty much works that way. Users interact with
a pretty slim web server to upload files, which are stored to an NFSv3
share owned by the processing server, which then does the hard number
crunching on the inputs. This is on a dedicated network connecting these
two boxes, so security is not an issue. I haven't seen any hangs happen
yet, that's a possibility if the processing server dies, though.
Serving out the files again happens via the same NFS share in my case,
the processing server has no direct connection to the internet.

HTH,
Kai

- -- 
Dipl.-Inform. Kai Blin         kai.blin@biotech.uni-tuebingen.de
Institute for Microbiology and Infection Medicine
Division of Microbiology/Biotechnology
Eberhard-Karls-Universität Tübingen
Auf der Morgenstelle 28                 Phone : ++49 7071 29-78841
D-72076 Tübingen                        Fax :   ++49 7071 29-5979
Germany
Homepage: http://www.mikrobio.uni-tuebingen.de/ag_wohlleben
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJORi7FAAoJEKM5lwBiwTTPynYH/1T5HKt8ehzYMubIovyGlcX2
7cvIGalaUwIDrB/TZDPs9cCAmHF0QC/B4z7smnV2bfpHxkZnl8LzJShzhIaM0imr
YqeoyOXvgtxxO4FhVZxukYYC+JO3yW29gYBcUwXupjSpb1rQm3fMnkrLZyKcvEju
l21/975PXAPlOsFtbQQORZqi3trLMlPIa687ZBXPq42QhKlE/mRaV+a5Ws3fUHHW
ZbFrpEgI3xxt1ka1sRW32q2VW9IlI22vWJqEagYGAV0rGzfCSDBOLjGXWqvzlScm
Zl8pVstVxS8pDk8mE6iDcE18uFWjvvfPf9l3XLeN2C8x4ZME6sGqRiDN89ZUhiM=
=Jd2n
-----END PGP SIGNATURE-----

Le 12/08/2011 16:26, Ishbir Singh a écrit :
>
> This is not directly a flask related question, but, nevertheless, I
> wanted to get your views about transferring files between a processing
> server and a self-owned file storage server. The file storage server has
> tons of bandwidth and diskspace. It will be serving images.
>
> What I'm trying to do is, a user upload files to processing server,
> which does some manipulations and then puts it on the static file
> storage server for retrieval by other users, deleting the local copy in
> the process. What would be the best way to do this? What should I use
> for transferring files? FTP or what?

Hi,

If both servers are on the same network and that network is trusted, I’d 
use NFS as suggested. Then your processing server can just write files.

NFS is sensitive to latency and has not so great (IP address-based) 
security by default. NFS4 can have proper security with Kerberos but I 
never could get it to work. (I never tried very hard either.)

Otherwise I’d probably use HTTP to have more control over security and 
network failure. (I’ve seen desktop systems freeze completely when NFS 
on /home fails.) However that is more work on both sides.

Regards,
-- 
Simon Sapin

hi

You can google NFS, all your questions in it.  

-------------------------------
linnchord@gmail.com

已使用 Sparrow (http://www.sparrowmailapp.com/?sig) 发送  

在 2011年8月12日星期五，下午10:26，Ishbir Singh 写道：

> Hey all,
>  
> This is not directly a flask related question, but, nevertheless, I 
wanted to get your views about transferring files between a processing 
server and a self-owned file storage server. The file storage server has 
tons of bandwidth and diskspace. It will be serving images.
>  
> What I'm trying to do is, a user upload files to processing server, 
which does some manipulations and then puts it on the static file storage 
server for retrieval by other users, deleting the local copy in the 
process. What would be the best way to do this? What should I use for 
transferring files? FTP or what?
>  
> --  
> Regards,
> Ishbir Singh