Jason Tackaberry wrote: > On Mon, 2005-10-17 at 11:21 +0200, Dirk Meyer wrote: >> What about supporting reading in a thread of the same app? > > I didn't implement it. That might be needed for kaa.vfs, but for epg it > didn't seem necessary. And if we can avoid it, we should, because it > adds a fair amount of complexity to the API. Calls over IPC are pretty > quick. The most common use-case of "fetch all programs between now and > 2 hours from now" takes about 0.04 seconds with a warm cache. (It's > slower when the db indexes aren't yet in memory of course, but that's > not an IPC issue.)
I will play with it. > Fetching 2200 programs over IPC takes about 0.14 seconds. That's time > to query the (warm) db, pickle the data to the client, and convert that > to Program objects. That's fast enough, IMO. And anyway, there's > hardly any situation where you'd need to move 2200 programs at one time. No, we need data from about 6 hours and 10 channels. More doesn't fit on the screen. >> > * Use libxml2 for xml parsing. Updating from a 17MB xmltv file takes >> > 95 seconds instead of 74 minutes. Seriously. >> >> Because something is missing. You just add a program to the db. But >> you need to check if it is already in it. Or if something changed >> (e.g. shows moved around). So the code of _add_program_to_db needs to >> query the db first. > > Yes, that's certainly going to add some more time to that process. But > if you think it's going to inflate the process time to 74 minutes you're > nuts. :) I know, I was saying that it will take some more time. I know the old parser is bad, but it worked ok for me ... but I only have 10 channels in it. >> Please check permissions of that. I don't care if xmltv violates >> copyright somehow, but I don't copyright violations in kaa. > > The software was written by me. I didn't take any code from other > sources. So there is no copyright violation possible. I'm not speaking of the source > zap2it does have license terms on how their data can be used, but that's > the user's responsibility to adhere to them, not ours. And if we assume > the user gives kaa.epg2 their own, legitimate zap2it username/password > to obtain the data, there is no license violation to simply download the > data. If there were, what possible point would zap2it datadirect > service have for existing? I don't know what source you use. Some xmltv grabber parse a website and this is not 100% clear if it is a copyright violation or not. >> > * Updating from backend is asynchronous. Downloading / parsing is done >> > in a thread, and the database is updated in steps within the main loop. >> >> Why do you call parse_channel and parse_programme in the main loop? > > I don't parse the whole xml tree into python objects up front if I don't > have to to keep memory usage low. Walking the tree for each > channel/program is extremely quick. There won't be really any benefit > by doing that beforehand in a thread. Almost all the time in that 0.1 > seconds of _update_process_step() execution is database access. The > rest is epsilon. Doesn't matter. If filling the db takes some seconds longer, no big deal. > What I put into a thread is downloading the guide info (for zap2it) and > calling libxml2.parse since those things aren't easily integrated with > the notifier loop. When that's done, the tree walking/database access > it done in the main loop. Downloading is easy, but calling libxml2 isn't easy. > I could probably reduce the step process time from 0.1 to 0.05 seconds. > It would decrease latency at the expensive of total update time. It's > probably not worth it. > >> > * GuideClient can used in the same process as GuideServer and will avoid >> > the unneeded IPC calls. >> >> So I can use this to read the db from within freevo. Great. And is the >> db itself (sqlite) a thread? It could block. > > It doesn't automatically run in a thread. You could do that easily > enough. But this probably wouldn't work. Your best bet is to stick > with client/server over IPC. I will play around with it. I found one possible problem: you can only use sqlite objects (like the cursor) from the same thread it was created. But I guess I can handle it. Dischi -- Error 13: Illegal brain function. Process terminated.
pgpBtmdklsDA3.pgp
Description: PGP signature
