On Friday 13 November 2009 15:47, McKown, John wrote:
>This goes back to the person who wanted some way to emulate DD concatenation
> of multiple datasets so that they are read as if they were one. Everybody
> agrees that there isn't an easy way. Now, I don't know filesystem
> internals. But what about a new type of symlink? Normally, a symlink
> contains the real name of the file. Sometimes a symlink will point to
> another symlink, and so on (I don't know how deep). What about a
> multi-symlink. That's where a symlink points to multiple files in a
> specific order. When the symlink is opened and read, each file in the
> symlink is opened and read in order. I know this would require some changes
> to open() as well, in order to make sure that each file in the symlink
> chain is readable by the process.
>
>What think? Or is this just alien to the UNIX mindset?
An interesting idea, and yes it is wierd and rather alien to UNIX minds.
You're implementing something at the filesystem level which is trivially
implemented at the process level. And all to avoid some IPC via pipes? Has
anyone calculated how much overhead there is in using cat to pipe some files
into a process instead of having the process read the files itself?
The more I think about this, the less this seems like a symlink. I thinking
of it as a meta-file: a file of files. This introduces the idea of a new
type of file whose contents are known to and interpreted by the system, in
the way a directory-file's contents are known. Does this really have any
value?
Regardless of its value, in thinking of how to implement this, I see a few
problems:
- What happens if one of the files is missing?
- How do you seek() in such a file?
- Similarlly, how do you implement locks on byte ranges within such a file?
- What happens if another process appends to one of the files while you are
reading a later one in the sequence? Does your read position change?
You can solve those, perhaps, by requiring an open() of a meta-file to open
all of the listed files. If any file open fails, the meta-file open fails
and closes all the others. A meta-file's file descriptor would have to refer
to a new kernel data structure that is a list of the open file descriptors of
the listed files (or rather pointers to the data structures referenced by
those file descriptors). This structure would be used to map an offset
within the meta-file to an offset within one of the list of files, using the
file's lengths. This solves the seek and lock problems. I'm still not sure
about the append problem, though.
Another possible implementation would be entirely within the filesystem, where
the meta-file would have direct access to the data-blocks of the underlying
files. I think that opens up too many cans-o-worms to be a good solution,
though.
Of course, once you have this kind of file, you have meta-files of meta-files
of meta-files of ... Isn't it better to represent such structures in
user-space instead of kernel-space?
>ln -s symlink realfile1 realfile2 /etc/fstab /tmp/somefile
This command-line syntax is already used by ln (the third form listed in the
manpage synopsis) to create several symlinks in a directory, which is the
final argument.
It's an interesting idea, but I'm not convinced of its utility. I'd like to
know what percentage of the I/O time (or CPU cycles) is used by piping files
via cat. Anyone have any measurements?
- MacK.
-----
Edmund R. MacKenty
Software Architect
Rocket Software
275 Grove Street · Newton, MA 02466-2272 · USA
Tel: +1.617.614.4321
Email: m...@rs.com
Web: www.rocketsoftware.com
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
