Jason Tackaberry wrote: > On Tue, 2005-04-19 at 11:27 +0200, Dirk Meyer wrote: >> inotify sounds nice. But it requires a special kernel build. So I >> guess it should be optional in Freevo. But is sounds much better than >> polling directories for changes. > > It's not difficult to build the vfs so that it uses inotify if > available, otherwise fall back to regular polling. I plan to add > dnotify support too, which is much more widely supported in the kernel. > But dnotify has more limitations and its interface is not as nice. So > if inotify wasn't available, I'd use dnotify for watching directories > currently being viewed in the GUI, but not for the repository-wide > monitoring for auto-indexing since that requires monitoring a large > number of directories and dnotify doesn't scale well. (dnotify requires > keeping a file descriptor open for each directory you want to monitor, > whereas with inotify you just talk to /dev/inotify.)
Ok, if we have inotify, we should use it. If not, try dnotify. If
still not, use polling the current directory like we do now.
>> What I found interesting is the sqlite db. I played with
>> sqlite and found using a directory pickle file much faster for normal
>> operations like "go into a directory". But I still need some sort of
>> database or index to do stuff like "show all mp3 files from 2001".
>
> Yes, pickling is going to be faster, especially when you're talking
> about adding a lot of data to the database.
Adding is not a problem. If we have to index a whole directory,
mmpython also takes it time. I need good SELECT speed (see below).
>> I don't like datbases right now, they seem slow. For a huge guide, the
>> epg database is slower than it should be. And for a mediadb, the index
>> can be huge.
>
> I find sqlite pretty reasonable in performance. Yes, it's often not
> going to be as fast as pickling. But it also shouldn't be disgustingly
> slower as long as you're doing it right. For example, are you doing an
> individual SELECT for each file as you iterate over a directory list?
No, I had a field dirname in each record. So I selected by this. Take
a look at WIP/Dischi/mediadb for the code.
> Here are some timing examples. When dealing with 10000 simple records
> in a files table (4 fields: file_id, dir_id, filename, mtime):
>
> 10k INSERTs: 1.48 seconds
> 1 SELECT on whole directory (WHERE dir_id=1): 0.26 seconds
0.26 seconds is a long time for that. The whole process reading the
pickle + checking for new/changed/deleted files for 2k photos takes
0.03 - 0.08 seconds now. Everything except showing the menu is 0.26
seconds.
> So compare the 1 select (which returns 10k records) to the load pickle
> case. It's true that it's faster. It's even much faster. But 0.26
> seconds overhead for loading a directory of 10k entries isn't horrible.
> Of course the actual directory load will take longer (closer to 0.9
> seconds when you do os.listdir(), stat each file, query the database,
> and sort the records), but it's still not unreasonable. And besides,
> who keeps 10k files in one directory? :)
You don't need os.listdir. If the directory mtime is still the same
(you need an extra SELECT to get that information, I don't), you can
skip checking for new/deleted files. You only need to stat the current
once. Using inotify, you could also skip that.
> The main thing is indexing.
Yes, I add a progress box when there are more than x changed/new
items.
> Thumbnailing even a very large image is reasonably quick.
> But even if it takes 0.1 seconds, that's too long. If you do a
> directory of large images and try to scroll the list while it's
> generating thumbnails for it, it will be sluggish.
OK, you won't see all thumbnails at first, but they come later and the
gui still works. And if you pre-cache the thumbnails with a helper,
you don't have that problem. Also think of inserting a new disc and
browse it. I have a DVD with the 2k photos. Indexing creates too much
time, also creating thumbnails and you have to wait will be too much
for the user. But you have not much time between inserting the disc
and showing it. You can't force the user to wait until an extra app
checked the disc.
> My design requirements are very strict in this respect. I demand a
> very smooth experience.
Same here.
> Anything that's sluggish needs to go somewhere else. :) And clearly,
> thumbnailing a video is a worst-case scenario where it may take
> several seconds. So rather than having special cases (thumbnailing
> an image is ok to do in-process, but video we fork or use a thread
> or whatever), just do all indexing the same way: out of the GUI
> process.
Keep in in background if possible, but force it if needed.
> So the reason to put directory monitoring in another process is just to
> keep the design simple. If indexing needs to be done in another process
> and I want to have this auto-indexing feature, then I also need
> directory monitoring logic in that process. Why not put it all in
> there.
The helper sounds nice, but it should be optional.
> Also if there are other processes, say a web server, that wants
> to do directory monitoring, it can just talk to the monitor process via
> IPC, so we don't have multiple processes polling the same directory.
Replace mbus with IPC and I like your idea :)
>> Great. It would help if I have something liek the ImageCancas. I need
>> an imlib2 image, draw on it and put it into evas. Everything else
>> would be easy to rewrite, only the creating of objects is used very
>> often. Is it possible to create an evas object from an imlib2 image?
>
> Taking an Imlib2 image and putting it as an Evas Image object is very
> easy. Both use BGR32 so there's no colorspace conversion that needs to
> be done. But in practice you probably don't want to do this. Evas does
> all imaging internally, and if you try to bolt on mevas's way of doing
> things with Imlib2 images, you're likely to not be happy with
> performance.
We will see what I need and what I don't need.
> As a matter of fact, whereas evas was once tightly coupled
> with Imlib2, now there's nary a reference to be found. They are
> completely divorced, and I think Imlib2 might be on the path to
> deprecation within EFL. (Maybe someone who knows about EFL can
> comment.)
I hope not. And they still need pyimlib2 for epsilon (thumbnails)
Dischi
P.S.: The signature is random, but fits
--
"Everything should be made as simple as possible, but not simpler."
(A.Einstein)
pgp2sZmpH4W9Q.pgp
Description: PGP signature
