On 22 Jul 2002, "John E. Malmberg" <[EMAIL PROTECTED]> wrote:
> > Qualities
> >
> > 1. Be reasonably portable: at least in principle, it should be
> > possible to port to Windows, OS X, and various Unixes without major
> > changes.
>
> In general, I would like to see OpenVMS in that list.
Yes, OpenVMS, perhaps also QNX and some other TCP/IP-capable RTOSs.
Having a portable protocol is a bit more important than a portable
implementation. I would hope that with a new system, even if the
implementation was unix-bound, you would at least be able to write a
new client, reusing some of the code, that worked well on ITS.
> A clean design allows optimization to be done by the compiler, and tight
> optimization should be driven by profiling tools.
Right. So, for example, glib has a very smart assembly ntohl() and
LZO is tight code. I would much rather use them than try to reduce
the byte count by a complicated protocol.
> > 4. Keep the socket open until the client gets bored. (Avoids startup
> > time; good for on-line mirroring; good for interactive clients.)
>
> I am afraid I do not quite understand this one. Are you refering to a
> server waiting for a reconnect for a while instead of reconnecting?
What I meant is that I would like to be able to open a connection to a
server, download a file, leave the connection open, decide I need
another file, and then get that one too. You can do this with FTP,
and (kindof) HTTP, but not rsync, which needs to know the command up
front.
Of course the server can drop you too by a timeout or whatever.
> If so, that seems to be a standard behavior for network daemons.
>
> > 5. Similarly, no silly tricks with forking, threads, or nonblocking
> > IO: one process, one IO.
>
> Forking or multiple processes can be high cost on some platforms. I am
> not experienced with Posix threads to judge their portability.
>
> But as long as it is done right, non-blocking I/O is not a problem for me.
>
> If you structure the protocol processing where no subroutine ever posts
> a write and then waits for a read, you can set up a library that can be
> used either blocking or non-blocking.
Yes, that's how librsync is structured.
Is it reasonable to assume that some kind of poll/select arrangement
is available everywhere? In other words, can I check to see if input
is available from a socket without needing to block trying to read
from it?
I would hope that only a relatively small layer needs to know about
how and when IO is scheduled. It will make callbacks (or whatever) to
processes that produce and consume data. That layer can be adapted,
or if necessary, rewritten, to use whatever async IO features are
available on the relevant platform.
> Test programs that internally fork() are very troublesome for me.
> Starting a few hundred individually by a script are not.
If we always use fork/exec (aka spawn()) is that OK? Is it only
processes that fork and that then continue executing the same program
that cause trouble?
> I can only read UNIX shell scripts of minor complexity.
Apparently Python runs on VMS. I'm in favour of using it for the test
suite; it's much more effective than sh.
> > 12. Try to keep the TCP pipe full in both directions at all times.
> > Pursuing this intently has worked well in rsync, but has also led to
> > a complicated design prone to deadlocks.
>
> Deadlocks can be avoided.
Do you mean that in the technical sense of "deadlock avoidance"?
i.e. checking for a cycle of dependencies and failing? That sounds
undesirably complex.
> Make sure if an I/O is initiated, that the
> next step is to return to the protocol dispatching routine.
> > 9 Model files as composed of a stream of bytes, plus an optional
> > table of key-value attributes. Some of these can be distinguished to
> > model ownership, ACLs, resource forks, etc.
>
> Not portable. This will effectively either exclude all non-UNIX or make
> it very difficult to port to them.
"Non-UNIX" is not completely fair; as far as I know MacOS, Amiga,
OS/2, Windows, BeOS, and QNX are {byte stream + attributes + forks}
too.
I realize there are platforms which are record-oriented, but I don't
have much experience on them. How would the rsync algorithm even
operate on such things?
Is it sufficient to model them as ascii+linefeeds internally, and then
do any necessary translation away from that model on IO?
> BINARY files are no real problem. The binary is either meaningful on
> the client or server or it is not. However file attributes may need to
> be maintained. If the file attributes are maintained, it would be
> possible for me to have a OpenVMS indexed file moved up to a UNIX
> server, and then back to another OpenVMS system and be usuable.
Possibly it would be nice to have a way to stash attributes that
cannot be represented on the destination filesystem, but perhaps that
is out of scope.
> I recall seeing a comment somewhere in this thread about timestamps
> being left to 16 bits.
No, 32 bits. 16 bits is obviously silly.
> File timestamps for OpenVMS and for Windows NT are in 64 bits, but use
> different base dates.
I think we should use something like 64-bit microseconds-since-1970,
with a precision indicator.
> File attributes need to be stored somewhere, so a reserved directory or
> filename convention will need to be used.
>
> I assume that there will be provisions for a server to be marked as a
> master reference.
What do you mean "master reference"?
> For flexability, a client may need to provide filename translation, so
> the original filename (that will be used on the wire) should be stored
> as a file attribute. It also follows that it probably is a good idea to
> store the translated filename as an attribute also.
Can you give us an example? Are you talking about things like
managing case-insensitive systems?
--
Martin
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
