Is there an undocumented redirect to stdout for wget? Or equally good, a
"bash"ful way to redirect (my bass book is at home)?
The 2 GB file size limit has reared its ugly head once more. The kernel
on the VA Linux 6.2.4 distro supports large files, but not the C
libraries. Thus, typical tools (with the exception of dd) still don't
work with large files. So far, I have been able to use ftp to download
smaller reference database files (all shy of 2 GB when compressed) and
using pipes and redirection to get around the size issue, such as:
uncompress -c dbEST.FASTA.dailydump.Z | formatdb -i stdin -p F -o T -n
dbEST
Formatdb results in a compressed format ready for homology searching,
but I can also leave it off and create huge files (useful only in
showing the value of stdout/stdin redirects in this system).
Now the reference databases have to be downloaded in ~ 5 GB file chunks
and I am thinking of a pipe which starts with the download itself, such
as:
wget -q (missing flag) ftp://server/file | gunzip -c stdin |
perlscript.pl
where perlscript.pl will recompress and distribute an equal portion of
the reference db to each node of the cluster and the reformatting can
occur off the master node.
--
Eric Engelhard
