Have you looked into rsync vs. ftp? I've used rsync in many situations to move files and it has many great options and handles interrupted transfers, etc.
You can find it here: http://samba.anu.edu.au/rsync/ Sincerely, Andy KoreLogic Security 603.465.3236 (Office) 603.340.2498 (Mobile) http://www.korelogic.com GnuPG Fingerprint: 688A 79EC B1E5 5748 CE87 1F20 2C45 60E7 0583 23B6 On Thu, Sep 04, 2008 at 06:42:16PM -0400, Flaherty, Patrick wrote: > I've been soliciting solutions from everyone I can think of on moving a > large number of files from inside our lan to a dmz on a regular basis. > > I have a cluster of machine producing 20k small files (30kbytes or so) > inside our lan. After the files are created, they are pushed to a few > web servers in the DMZ using ftp. The push is done by the machine that > created the file. Ideally, the files make it out to the DMZ in less than > 30 seconds but there have been some issues. > > FTP seems to fall down when scaling out to more than a web server or > two, many retries and transfer failures. It also adds to complexity to > the processing. What if one of the web servers is down? How many time do > you retry? Should you notify the other hosts in the cluster? All that > logic needs to be in the pushing script, which becomes a bit ungainly. > There's also the issue with constantly opening up new ftp sessions, > which is a bit expensive. > > So I'm looking for a cleaner architecture. An ideal solution would be an > NFS/CIFS share internal to the lan replicated readonly to an NFS/CIFS > share in the DMZ. The cluster can write to the nfs share, the web > servers can read from the nfs share. Everyone is happy. The big sticking > point is being careful violating the security by multi homing the > storage. Many solutions require an open connection network on many ports > between the two storage boxes, which would be an easy way in to our lan. > > So far I'm poking at (and some downsides): > FUSE + (sshfs/ftpfs): High performance hit (60%ish from what I've read) > ZFS + StorageTek: Great, another operating system train people on. > DRBD: requires full network connection between lan and dmz boxes. > dataplow sfs + das box: sales people will promise you the world. > Software SAN replicators of to many names to mention. > > This is such a common problem, I'm not sure why there isn't a nice > canned solution of two cheap pieces of hardware. Maybe I'm just an idiot > and there is. Oh please please please tell me I'm an idiot. > > Anyone have any brilliant ideas? > > Best, > Patrick > > _______________________________________________ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
pgpBD8POQDhuY.pgp
Description: PGP signature
_______________________________________________ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
