[vdr] xmltv2vdr speedup and modification
Hi, I recently worked on xmltv2vdr.pl (version 1.0.6) and checked why it was so slow on my mighty Celeron 233. So I modified it a little to avoid reading all the xmltv file for each channel defined in the channels.conf. The result is good : I can process my 5Mo xmltv file in less than 10 minutes whereas it took at least 1 hour with vanilla 1.0.6 release. I also added support for sub-title in the xmltv file (I think someone already posted about that in the list). I only use it for one week so it can still be buggy. I'll be happy to take care of any bug found. Hope this helps those with old hardware like me. Sebastien xmltv2vdrv5.pl Description: Binary data ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] xmltv2vdr speedup and modification
Hi, I'm the original author of xml2vdr (for my sins!), thanks for working on it and improving the performance. If everyone is happy with this new code release and it really improves the speed I'll wrap it up and make a formal release and get Klaus to add it to the VDR FTP site. As a lot of people seem to use xml2vdr, perhaps it would be good to resurrect it and keep it formally updated? Regards, Morfsta On 2/12/07, Sebastien Lucas <[EMAIL PROTECTED]> wrote: Hi, I recently worked on xmltv2vdr.pl (version 1.0.6) and checked why it was so slow on my mighty Celeron 233. So I modified it a little to avoid reading all the xmltv file for each channel defined in the channels.conf. The result is good : I can process my 5Mo xmltv file in less than 10 minutes whereas it took at least 1 hour with vanilla 1.0.6 release. I also added support for sub-title in the xmltv file (I think someone already posted about that in the list). I only use it for one week so it can still be buggy. I'll be happy to take care of any bug found. Hope this helps those with old hardware like me. Sebastien ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
RE: [vdr] xmltv2vdr speedup and modification
> I recently worked on xmltv2vdr.pl (version 1.0.6) and checked
> why it was so slow on my mighty Celeron 233. So I modified it
> a little to avoid reading all the xmltv file for each channel
> defined in the channels.conf. The result is good : I can process
> my 5Mo xmltv file in less than 10 minutes whereas it took at least
> 1 hour with vanilla 1.0.6 release.
Something more what you can do (just by looking source you provided)..
--
Caching of &xmltime2vdr, like
return $timecache{$xmltime}->{$skew} if defined
$timecache{$xmltime}->{$skew}
$secs = &Date::Manip::UnixDate($xmltime, "%s") + $skew*60;
$timecache{$xmltime}->{$skew} = $secs
return secs;
But it depends on how much this function is called.. But hash lookup is probably
faster than running UnixDate from library. So it is a memory tradeoff.
--
I see that there is still some basic Perl based optimizations for this.
For example there is browsing through @xmllines array, and every iteration
you recompile *ALL* regexp's. That is as many times as @xmllines has lines.
And if one recompile takes 1ms -> you waste time @xmllines * 1ms just for
compiling and not doing anything usefull.
Perl switch "o" is recompile once flag, use that everywhere where it is
possible. Variable is not a problem unless variable changes in every iteration.
# New XML Program - doesn't handle split programs yet
if ( ($xmlline =~ /\ if ( $chanCur eq '' )
But this would be very minor effect..
--
Split is heavy operation because of creating arrays, but you can limit it.
( $null, $xmlst, $null, $xmlet, @null ) = split(/\"/, $xmlline);
=> ( $null, $xmlst, $null, $xmlet, $null ) = split(/\"/, $xmlline, 5);
or even using regexp for this. I don't know input line for this, but if it is
foo,"something","something",...
($xmlst,$xmlet) = $xmlline =~ m:\"(.*?)\",\"(.*?)\":o;
or probably combine 2 regexp to a single
($xmlst,$xmlet,$channel) = $xmlline =~
m:\"(.*?)\",\"(.*?)\".*?channel=\"(.*?)\":o;
--
Again something very weird:
if ( ($xmlline =~ /\/, $xmlline);
( $vdrtitle, @null ) = split(/\(.*?)\:o;
Same for XML subtitle
SVDRPsend("T $1") if $xmlline =~ m:\(.*?)\:o;
Generally
if ( ($xmlline =~ /\/, $xmlline);
( $vdrdesc, @null ) = split(/\ if ( ( $chanevent == 0 ) && ($xmlline =~ /\ c
< 250 foo
< 250-foo
> .
< 354-not ok
It could still succeed if from socket buffer "250-" is read. Also the 2 substr
calls
in SVDRPreceive is a bit weird, but I am uncertain if regexp would help that.
At least
change "-" to '-'.
--
There is a LOT to improve, but if you do these, you Celeron will fly. I hope
you'll get
to minute scale (or even better). Look with "time xmltv2vdr" to see how much
process
time is used for user code and how much for kernel code. And to see if
optimizations help.
Have fun.. :)
Best regards,
Jori
Ps. complete untested code, there might be some problems as I didn't run this
code even once.
#!/usr/bin/perl
# xmltv2vdr.pl
#
# Converts data from an xmltv output file to VDR - tested with 1.2.6
#
# The TCP SVDRSend and Receive functions have been used from the getskyepg.pl
# Plugin for VDR.
#
# This script requires: -
#
# The PERL module date::manip (required for xmltv anyway)
#
# You will also need xmltv installed to get the channel information:
# http://sourceforge.net/projects/xmltv
#
# This software is released under the GNU GPL
#
# See the README file for copyright information and how to reach the author.
# 20070210 / SL / Little optimization to make it work on old hardware,
# Added support for sub title.
# 20070212 / JH / Perl and logistic optimization
# $Id: xmltv2vdr.pl 1.0.6 2004/04/19 20:01:04 psr Exp $
use Getopt::Std;
use Time::Local;
use Date::Manip; # Instead of using this, for less dependency could use posix
mktime
# Convert XMLTV time format (MMDDmmss ZZZ) into VDR (secs since epoch)
sub xmltime2vdr
{
my $xmltime=shift;
my $skew=shift;
if (defined $timecache{$xmltime}->{$skew}) {
return $timecache{$xmltime}->{$skew};
} else {
$secs = &Date::Manip::UnixDate($xmltime, "%s") + $skew*60;
$timecache{$xmltime}->{$skew} = $secs;
return $secs;
}
}
# Send info over SVDRP (thanks to Sky plugin)
sub SVDRPsend
{
my $s = shift;
if ($sim == 0)
{
print SOCK "$s\r\n";
}
else
{
print "$s\r\n";
}
}
# Recv info over SVDRP (thanks to Sky plugin)
sub SVDRPreceive
{
my $expect = shift | 0;
if ($sim == 1)
{ return 0; }
my @a = ();
while () {
s/\s*$//; # 'chomp' wouldn't work with "\r\n"
push(@a, $_);
if (substr($_, 3, 1) ne "-") {
my $code = substr($_, 0, 3);
die("expected SVDRP code $expect, but received $code") if ($code !=
$expect);
last;
}
}
return @a;
}
# Process info from XMLTV file / channels.conf and send via SVDRP to VDR
sub ReadChannels {
Re: [vdr] xmltv2vdr speedup and modification
Sebastien Lucas wrote: Hi, I recently worked on xmltv2vdr.pl (version 1.0.6) and checked why it was so slow on my mighty Celeron 233. Yes! Thanks, it includes the Subtitle Patch ;-) Regards, Thomas ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] xmltv2vdr speedup and modification
On 2/12/07, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> I recently worked on xmltv2vdr.pl (version 1.0.6) and checked
> why it was so slow on my mighty Celeron 233. So I modified it
> a little to avoid reading all the xmltv file for each channel
> defined in the channels.conf. The result is good : I can process
> my 5Mo xmltv file in less than 10 minutes whereas it took at least
> 1 hour with vanilla 1.0.6 release.
Something more what you can do (just by looking source you provided)..
--
Caching of &xmltime2vdr, like
return $timecache{$xmltime}->{$skew} if defined
$timecache{$xmltime}->{$skew}
$secs = &Date::Manip::UnixDate($xmltime, "%s") + $skew*60;
$timecache{$xmltime}->{$skew} = $secs
return secs;
But it depends on how much this function is called.. But hash lookup is probably
faster than running UnixDate from library. So it is a memory tradeoff.
I still haven't tested it but I doubt it'll help . I'll check later.
I see that there is still some basic Perl based optimizations for this.
For example there is browsing through @xmllines array, and every iteration
you recompile *ALL* regexp's. That is as many times as @xmllines has lines.
And if one recompile takes 1ms -> you waste time @xmllines * 1ms just for
compiling and not doing anything usefull.
Perl switch "o" is recompile once flag, use that everywhere where it is
possible. Variable is not a problem unless variable changes in every iteration.
[]
I didn't know that (I'm not really a perl guru ... far from it). I'll
update my version. But it didn't help at all with my benchmark.
--
As there is many times $xmlline is matched with regexps etc. You should
experiment
with "study $xmlline;" after chomp $xmlline. Study makes internal search tables
for string matches. So see which way the code is faster, with study or without
study. Use Unix shell's time-command for this. For extra boost with study you
probably would need to take away subroutine "xmltvtranslate" as for it $xmlline
is copied to subroutine's parameter space, and what is matched. And study would
not affect it. So instead of calling "$xmlline=xmltvtranslate($xmlline);"
cut&paste
subroutines code here, and use $xmlline instead of $line.
foreach $xmlline (@xmllines)
{
chomp $xmlline;
study $xmlline;
$xmlline=~s/und uuml;/ü/go;
$xmlline...
This isn't pretty but could probably help a bit. You save time for @xmllines
times calling
subroutine, and study would help you a lot as you use the same string all the
time.
I'll check that later.
For constant string you could use ' ' instead of " ". " causes string to be
evaluated for variables
if ( $chanCur eq "" ) --> if ( $chanCur eq '' )
But this would be very minor effect..
I'll surely be too lazy to test that. sorry.
Split is heavy operation because of creating arrays, but you can limit it.
( $null, $xmlst, $null, $xmlet, @null ) = split(/\"/, $xmlline);
=> ( $null, $xmlst, $null, $xmlet, $null ) = split(/\"/, $xmlline, 5);
or even using regexp for this. I don't know input line for this, but if it is
foo,"something","something",...
($xmlst,$xmlet) = $xmlline =~ m:\"(.*?)\",\"(.*?)\":o;
or probably combine 2 regexp to a single
($xmlst,$xmlet,$channel) = $xmlline =~
m:\"(.*?)\",\"(.*?)\".*?channel=\"(.*?)\":o;
--
Again something very weird:
if ( ($xmlline =~ /\/, $xmlline);
( $vdrtitle, @null ) = split(/\(.*?)\:o;
Same for XML subtitle
SVDRPsend("T $1") if $xmlline =~ m:\(.*?)\:o;
Yes I'll also prefer shorter code. I'll check further if something
like is also allowed to adapt the regex. For
information that change has no impact on my bench.
Generally
if ( ($xmlline =~ /\/, $xmlline);
( $vdrdesc, @null ) = split(/\
Agree. I'll try to modify it.
Some logical errors
if ( ($xmlline =~ /\ if ( ( $chanevent == 0 ) && ($xmlline =~ /\
In fact the check $chanevent == 0 is only usefull if the xml is not
well formed so it doesn't change anything.
Then
elsif ( $chanCur ne $chan )
{
SVDRPsend("c");
SVDRPsend(".");
SVDRPreceive(250);
I think programmer wanted outout of "." -command, and see if it's status is 250?
But now I think code is checking status of "c" -command? As socket is not read
between calls, and there should be data in buffer for c-command. But I cannot be
sure as I don't know SVDRP command that well.
> c
< 250 foo
< 250-foo
> .
< 354-not ok
It could still succeed if from socket buffer "250-" is read. Also the 2 substr
calls
in SVDRPreceive is a bit weird, but I am uncertain if regexp would help that.
At least
change "-" to '-'.
I have no answer about this as I didn't modify that code . it
seems to work so for now I don't touch it.
There is a LOT to improve, but if you do these, you Celeron will fly. I hope
you'll get
to minute scale (or even better). Look with "time xmltv2vdr" to see how
Re: [vdr] xmltv2vdr speedup and modification
On 2/13/07, Morfsta <[EMAIL PROTECTED]> wrote: Hi, I'm the original author of xml2vdr (for my sins!), thanks for working on it and improving the performance. I know and thanks for it. If everyone is happy with this new code release and it really improves the speed I'll wrap it up and make a formal release and get Klaus to add it to the VDR FTP site. I'd be happy it goes that way. As a lot of people seem to use xml2vdr, perhaps it would be good to resurrect it and keep it formally updated? Yes there also was some interesting post in this mailing list about xmltv2vdr. Thanks for your post. Sebastien ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
RE: [vdr] xmltv2vdr speedup and modification
> But it didn't help at all with my benchmark. > ... > For information that change has no impact on my bench. Interesting, what version of Perl are you running if those changes don't do anything? --- Futher improvement is that now it is unnecessary to read whole XML-file into memory, as the file is linearly scanned through. So no need to waste 5MB of memory if you are short of it. -- # Read all the XMLTV stuff into memory - quicker parsing open(XMLTV, "$xmltvfile") || die "cannot open xmltv file"; @xmllines=; close(XMLTV); sub ProcessEpg # Find XML events foreach $xmlline (@xmllines) -- => open(XMLTV, "$xmltvfile") || die "cannot open xmltv file"; sub ProcessEpg while($xmlline = ) ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] xmltv2vdr speedup and modification
On 2/13/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > But it didn't help at all with my benchmark. > ... > For information that change has no impact on my bench. Interesting, what version of Perl are you running if those changes don't do anything? vdr26:~/xmltv# time ./xmltv2vdrv5.pl -s -c channels.conf -x tvguide.xml real3m4.397s user2m50.475s sys 0m6.052s vdr26:~/xmltv# time ./xmltv2vdrv6.pl -s -c channels.conf -x tvguide.xml real3m4.309s user2m48.951s sys 0m7.240s xmltv2vdrv5 = the version I posted xmltv2vdrv6 = the version I posted + the o switch on all regex + title and subtitle now use regex and no more split. about perl (from debian sarge) : vdr26:~/xmltv# perl --version This is perl, v5.8.4 built for i386-linux-thread-multi Futher improvement is that now it is unnecessary to read whole XML-file into memory, as the file is linearly scanned through. So no need to waste 5MB of memory if you are short of it. -- # Read all the XMLTV stuff into memory - quicker parsing open(XMLTV, "$xmltvfile") || die "cannot open xmltv file"; @xmllines=; close(XMLTV); sub ProcessEpg # Find XML events foreach $xmlline (@xmllines) -- => open(XMLTV, "$xmltvfile") || die "cannot open xmltv file"; sub ProcessEpg while($xmlline = ) Good idea, I have not thought about it (my wonderful Celeron 233 has 384Mo of Ram). Thanks for your help. Sébastien ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] xmltv2vdr speedup and modification
On 2/13/07, Morfsta <[EMAIL PROTECTED]> wrote: Hi, I'm the original author of xml2vdr (for my sins!), thanks for working on it and improving the performance. If everyone is happy with this new code release and it really improves the speed I'll wrap it up and make a formal release and get Klaus to add it to the VDR FTP site. As a lot of people seem to use xml2vdr, perhaps it would be good to resurrect it and keep it formally updated? Regards, Morfsta Hi all, new version is attached and this time with a precise list of all changes : - The Xmltv file is now read only one time (big speedup). - You can have many channel xmltv id for each channel in channel.conf, you can have : BBC FOUR (TV):529833:I0C34D0M16B8T2G32Y0:T:27500:201:401,402:0:0:16832:0:0:0:knowledge.bbc.co.uk,test.anytv.com I did that because I use two main xmltv source which of course don't share the same channel id. - sub-title are now processed. - regex are now compiled one (thanks to Jori Hamalainen) - I have replaced all split by regex to simplify (shorten) the code. - Add a little test to drop already finished event (should be an option, but I was too lazy for that for now). - Added some my attributes to make "use strict" happier. Todo : - Handle the case where two channels in channels.conf share the same epg. I worked with vanilla 1.0.6 but not anymore. I was able to get a little more speed of xmltv2vdr with a little help from sed before using xmltv2vdr. Disclaimer : I'm terrible with sed so please keep away the tar and feather ;) # to remove things unused by wmltv2vdr, you may want to adapt this to your source sed -e '//d' -e '/ resultat.xml and then make everything between and be on one single line with script like that : / xmltv2vdr.pl Description: Binary data ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
