Re max files, here is what I found: It depends on the file system. ext3 suppport ~32000 subdirectories (not files!) in a given directory, with ext4 it's 64000 by default. xfs has no limit to my knowledge. You should consider not putting too many files in a single directory. Most software doesn't handle that well (e.g. mc will be slow, many gui tools will be unusable). It's better to create a hierarchy of nested folders and distribute the files in them using some algorithm (hash of file name or content or any other method which will distribute the files equally). That's what many mature programs (e.g. squid) do.
-----Original Message----- From: Marlon Pierce [mailto:[email protected]] Sent: Thursday, August 28, 2014 7:54 AM To: Airavata Dev Subject: Experiment log messages to different log files I'm wondering if it would be a good idea to create a separate log file for each experiment. We current direct everything to airavata.log and airavata-server.out. As a consequence, log entries for different submissions get interleaved, which makes tracing a particular experiment's life cycle difficult. Note there is a 32,000 file limit per directory in Linux by default (unless my knowledge is obsolete), so we'd need to take this into account. Alternatively, we could do a better job of labeling the entries in airavata.log so that it was clear which experiment is associated with the entry. Other suggestions? Marlon
