Excerpts from William Morgan's message of Thu Oct 01 14:31:20 -0400 2009: > Reformatted excerpts from Ben Gamari's message of 2009-10-01: > > It seems that C would definitely be a good start (or perhaps C++ would > > be a better idea as that is the language in which Xapian is written). > > I was proposing that as a strawman argument. C does not solve my > problem.
Certainly, you will never be able to write a message indexer fast enough to index a source instantly. That is why we have an index to begin with. That being said, I think we can do substantially better than we are currently doing in a lower-level language. With mutt, I can come close to saturating my drive bandwidth while loading a folder. While synchronizing in sup, I am lucky to get a few hundred kilobytes/second. Certainly a large amount of that difference has to do with the amount of processing done by each, but even adjusting for this it seems to me that we should still be I/O bound (or at least close to it). > > > However, I think one of the real issues is the exclusive nature of > > index access. In fact, this is one of my primary gripes with the sup > > workflow. After processing a large number of messages, the write-out > > time can be quite substantial upon killing the buffer. > > It is possible to address this in a variety of ways, many of which have > been discussed over the years, but yes, I agree that a delay is > nonideal. Glad we agree. > > Making message state saving fast, or backgrounded, is a different beast > from scanning over a mailstore. If we are unable to update the index while the client is active, rescanning sources would destroy usability. I would argue that asynchronous writeout is very much a prerequisite for mutable sources. > > I have been working on a Sup server for quite some time that would > address many of these problems, but progress is slow. This was largely what I had in mind. It seems like moving index manipulation out-of-process might be best, ultimately. > > > As an aside, it would be quite nice if one could run multiple > > simultaneous instances of sup. It seems that if one only held write > > access to the index during writes (is this the case presently?), there > > should be nothing preventing this from being possible. > > It might be possible to do this with Xapian, especially if there were no > expectation of updates being transmitted across processes. I think initially no updates between processes would be fine. > With Ferret, > if it is possible, it's only with a tremendous amount of effort. Is ferret even going to be supported after the Xapian backend stabilizes? Thanks for the comments and sup. - Ben _______________________________________________ sup-talk mailing list sup-talk@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-talk
