Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
Hi Richard,
On Tue, Jun 16, 2009 at 10:05:16PM +0200, Richard van den Berg wrote:
On 6/16/09 12:44 AM, H. Langos wrote:
My first idea now is to
strip the iTunesDB of the stuff that is optional.
Thanks again for the suggestion, that worked really well. I can now see
all my 30415 songs! Whoohoo. :-)
Congratulations! Now please don't go overboard with the patch. :-)
The default behavior for our users should not change. Meaning every possible
mhod should be written. The code and the naming of variables should express
that this is a workaround, not the default. I.e the keepattr would better
be named low_ram_attr or shrink_itunes_db or minimal_attr.
Your mhod skipping code should only be active if that option is set in your
.gnupodrc or given as command line switch.
Did you find out if shrinking the existing attributes (in contrast to
skipping) had any effect? If it did we could keep some counters for mhods
and their cumulative size in mktunes and either print out those stats if we
run in verbose mode (which would have to be added :) ) or print out warnings
if certain limits are reached.
A second patch is needed for tunes2pod
which will read the missing attributes from GNUtunesDB.xml, otherwise
they are gone forever after tunes2pod is run. I'll work on that later.
I wouldn't worry about that, AFAIK tunes2pod is only run once when you start
working with gnupod. Afterwards the flow of information is mostly this:
OTG Playlist
||
vv
GNUtunesDB.xml== iTunesDB
^^
||
Play Counts
You only need to rerun tunes2pod if your GNUtunesDB.xml is lost or if you
played around with iTunes in the meantime ... and we could warn people about
this.
BTW: Does iTunes manage to get your full collection over to your ipod, or
does it also produce an unusable iTunesDB?
Code remarks: I like the passing of optional arguments in a hash. However I
strongly suggest using lower case keys (keep instead of Keep and item instead
of Item.)
- $mktunes-WriteItunesDB;
+ $mktunes-WriteItunesDB(Keep=$opts{'keepattr'});
Shouldn't that be :
$mktunes-WriteItunesDB( {Keep = $opts{'keepattr'}} );
With the additional curly brackets to express that you are defining an
anonymous hash?
Maybe perl quesses right but I am not trusting perl to always do the
right thing. And how would that line look if you add another hashkey?
Would perl put both into one hash or would it pass two separate hashes?
+ my($self, %args) = @_;
+ my $object = $args{Item};
+ my @keep= split(/[ ,]+/, $args{Keep});
my $mhit= ''; # Buffer for the new mhit
my $mhod_chunks = ''; # Buffer for the childs (mhods)
my $mhod_count = 0; # Child counter
@@ -275,6 +277,7 @@
foreach my $key (sort keys(%$object)) {
my $value = $object-{$key};
next unless $value; # Do not write empty values
+ next unless ($#keep0 || grep(/^$key$/,@keep)); # Only
keep specific mhods
my $new_mhod = GNUpod::iTunesDB::mk_mhod({stype=$key,
string=$value});
next unless $new_mhod; # Something went wrong
$mhod_chunks .= $new_mhod;
Instead of @keep, I'd use %keep. and instead of creating it here for
every mhit, I'd create it only once, and pass it along as a hashref.
cheers
-henrik
PS: I've see the following line in the patch. So it seems my empty desc
attributes wouldn't have made it into the iTunesDB anyway:
next unless $value; # Do not write empty values
___
Bug-gnupod mailing list
Bug-gnupod@nongnu.org
http://lists.nongnu.org/mailman/listinfo/bug-gnupod
Re: [Bug-gnupod] Do not stretch artwork
On Tue, Jun 16, 2009 at 10:08:00PM +0200, Richard van den Berg wrote: On 6/15/09 6:01 PM, Christophe Fergeau wrote: The background color is specified in the SysInfoExtended file if you are using/parsing it. Thanks for the suggestion, but gnupod currently doen't parse this file. I think the best thing to do is have an option to select the background color. I think SysInfoExtended is written by gtkpod. I think it was ment as a more readable version of apple's SysInfo file. But newer iPods don't have that SysInfo file anymore. My 3g nano (late 2007) doesn't. See http://www.thismuchiknow.co.uk/?p=15 However, SysInfo is now obsoleted by iTunes 7 / latest firmware update (no device information is actually stored on the mass storage partition anymore). We might add support for it, but currently gnupod is configured via ~/.gnupodrc cheers -henrik ___ Bug-gnupod mailing list Bug-gnupod@nongnu.org http://lists.nongnu.org/mailman/listinfo/bug-gnupod
Re: [Bug-gnupod] SoundCheck/ReplayGain
Hi chris, On Tue, Jun 16, 2009 at 11:26:12PM +0200, chris.com wrote: H. Langos wrote: Hi chris, I borrowed an ipod shuffle 3gen (what apple calls iPod shuffle 2nd generation Early 2008) from a collegue but that one seems to be up to date too. All the volume adjustment fields are 00 00 00 Does the revision as shown by udev represent the firmware version? ID_VENDOR=Apple ID_MODEL=iPod ID_REVISION=2.70 If iTunes shows the firmware version somewhere in device information you could compare it to the output of udevinfo --path /sys/block/sda --query env or /sbin/udevadm info --path /sys/block/sda --query env Actually I have a Firmware-131.1.0.3 loaded and I get: ID_VENDOR=Apple ID_MODEL=iPod ID_REVISION=2.70 Ok, so no luck there. Finding itunes 7.4 was really easy but where might I well find an old firmware for a 2nd generation shuffle? Anyone have an idea? Google? Just look for ipod and firmware and the first hit was this: http://www.felixbruns.de/iPod/firmware/ Well, I tried that. I downloaded the iPod_130.1.0.3.ipsw and iPod_131.1.0.3.ipsw firmware from appldnld.apple.com.edgesuite.net. The Firmware part is identical and when connected to iTunes the firmware version shown is 1.0.4 for both of 'm. felixburns has a 133 version, I guess it is more recent. I haven't tried it. Do you want me to load a 130 firmware and send you the iTunesSD? That would be great. Could you find out when that firmware was released? In general I'd like to get the oldest firmware on it that you possibly can. Did any software come with the shuffle? Maybe a it contains a reset utility that contains old firmware? I've found some comment in a forum in late 2006: http://forums.ilounge.com/showthread.php?t=181616 When I updated my 1G shuffle, the computer said the firmware was for some bug fixes and volume control, though I've noticed no difference in the shuffle's operation. If that firmware FIXED the shuffle's behavior to what we see today, and it didn't change from one vaild behavior (the one from 1g shuffle) to another (todays), then we might simply solve the issue by adding new model strings to gnupod and change the behavior of mktunes accordingly. Does apple publish something like a changelog for their firmware updates? cheers -henrik ___ Bug-gnupod mailing list Bug-gnupod@nongnu.org http://lists.nongnu.org/mailman/listinfo/bug-gnupod
Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
On Wed, June 17, 2009 10:41, H. Langos wrote:
The code and the naming of variables should express
that this is a workaround, not the default. I.e the keepattr would
better be named low_ram_attr or shrink_itunes_db or minimal_attr.
I'll fix that. I was already thinkign you wouldn't like the keepattr name.
:-)
Your mhod skipping code should only be active if that option is set in
your .gnupodrc or given as command line switch.
That is the case now. If keppattr is not set, @keep is empty and $#keep =
-1 and nothing will be filtered. I wouldn't have written it any other way.
Did you find out if shrinking the existing attributes (in contrast to
skipping) had any effect?
It has effect on the iTunesDB size for sure, but not as drastically as
removing mhods. I can do more testing to see if it has an effect on the
iPod accepting the iTunesDB, but I am rather fed up with binary searches
at the moment. It takes a lot of time. I'd have to find the critical point
in my current configuration and then use the size of the mhods to see if I
can push it over or under this point.
You only need to rerun tunes2pod if your GNUtunesDB.xml is lost or if you
played around with iTunes in the meantime ... and we could warn people
about this.
I don't plan on using iTunes, but I could see myself using Floola in
combination with gnupod.
BTW: Does iTunes manage to get your full collection over to your ipod, or
does it also produce an unusable iTunesDB?
I don't have my collection loaded in iTunes. I was saving that as a last
resort. Perhaps I'll give it a try over the weekend, after making sure it
can only access my collection read-only.
Code remarks: I like the passing of optional arguments in a hash. However
I strongly suggest using lower case keys (keep instead of Keep and item
instead of Item.)
I though I was using the gnupod coding convention there. I don't like it
either, so I'll fix the other capitalized keys as well.
Shouldn't that be :
$mktunes-WriteItunesDB( {Keep = $opts{'keepattr'}} );
Absolutely. I already noticed that, and was quite surprised it worked
without the {}.
Instead of @keep, I'd use %keep. and instead of creating it here for
every mhit, I'd create it only once, and pass it along as a hashref.
Will do.
PS: I've see the following line in the patch. So it seems my empty desc
attributes wouldn't have made it into the iTunesDB anyway:
next unless $value; # Do not write empty values
Does $value evaluate to false when it is an empty string? I am planning to
check which mhods I am now actually filtering. I was quite surprised by
the size decrease my patch caused. Perhaps there are some mhods that
should be filtered by default.
Cheers,
Richard
___
Bug-gnupod mailing list
Bug-gnupod@nongnu.org
http://lists.nongnu.org/mailman/listinfo/bug-gnupod
Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
On Wed, Jun 17, 2009 at 12:11:06PM +0200, Richard van den Berg wrote: On Wed, June 17, 2009 10:41, H. Langos wrote: Your mhod skipping code should only be active if that option is set in your .gnupodrc or given as command line switch. That is the case now. If keppattr is not set, @keep is empty and $#keep = -1 and nothing will be filtered. I wouldn't have written it any other way. Ahh, I see ... my brain isn't used to unless statements... I need to translate it to next unless ($#keep0 || grep(/^$key$/,@keep)); #Only keep specific mhods to next if ($#keep=0 !grep(/^$key$/,@keep)); #Only keep specific mhods to understand it right :-) Did you find out if shrinking the existing attributes (in contrast to skipping) had any effect? It has effect on the iTunesDB size for sure, but not as drastically as removing mhods. I can do more testing to see if it has an effect on the iPod accepting the iTunesDB, but I am rather fed up with binary searches at the moment. It takes a lot of time. I'd have to find the critical point in my current configuration and then use the size of the mhods to see if I can push it over or under this point. I don't think we need tu hurry that. It just would be nice to have some numbers when warning users about possible memory issues. You only need to rerun tunes2pod if your GNUtunesDB.xml is lost or if you played around with iTunes in the meantime ... and we could warn people about this. I don't plan on using iTunes, but I could see myself using Floola in combination with gnupod. Wow, Floola looks realy nice. I understand your concern now. This would also be important for people who use gnupod and gtkpod or any other libgpod based applications. So tunes2pod would have to be improved to incorporate new files and playlists and accept changes to existing attributes but to keep those attributes that it already finds in GNUtunesDB.xml. BTW: Does iTunes manage to get your full collection over to your ipod, or does it also produce an unusable iTunesDB? I don't have my collection loaded in iTunes. I was saving that as a last resort. Perhaps I'll give it a try over the weekend, after making sure it can only access my collection read-only. That would be fun and interesting to see which strategy iTunes chooses to save space. Code remarks: I like the passing of optional arguments in a hash. However I strongly suggest using lower case keys (keep instead of Keep and item instead of Item.) I though I was using the gnupod coding convention there. I don't like it either, so I'll fix the other capitalized keys as well. Yes but please keep those changes capitalization changes limited to your new code. I'd liek to put of big cleanup actions until we get the git repository up and runing. I've got things like whitespace cleanup and use warnings; for the old code on my TODO list. Does $value evaluate to false when it is an empty string? I am planning to check which mhods I am now actually filtering. I was quite surprised by the size decrease my patch caused. Perhaps there are some mhods that should be filtered by default. from man perlsyn: Truth and Falsehood The number 0, the strings '0' and '', the empty list (), and undef are all false in a boolean context. All other values are true. Negation of a true value by ! or not returns a special false value. When evaluated as a string it is treated as '', but as a number, it is treated as 0. Cheers -henrik ___ Bug-gnupod mailing list Bug-gnupod@nongnu.org http://lists.nongnu.org/mailman/listinfo/bug-gnupod
Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
On 6/17/09 10:41 AM, H. Langos wrote:
- $mktunes-WriteItunesDB;
+ $mktunes-WriteItunesDB(Keep=$opts{'keepattr'});
Shouldn't that be :
$mktunes-WriteItunesDB( {Keep = $opts{'keepattr'}} );
It turns out it depends on your usage. The form I chose you can used as:
($self,%args) = @_;
@keep=split(/[ ,]+/, $args{Keep});
While the form with the {} can be used as:
($self,$args) = @_;
@keep=split(/[ ,]+/, $args-{Keep});
Both forms are in use in the gnupod code at the moment, I just picked
the first..
Here is a new patch with your suggestions and the tunes2pod code. The
diff for XMLhelper.pm looks a bit weird, but all I did was move the
reset code to a new resetxml() sub. The reset is needed because the
playlists are stored in XDAT by doxml() and won't be overwritten by the
playlists from the iTunesDB otherwise.
Cheers,
Richard
? .gnupod_version
? Makefile
? autom4te.cache
? config.log
? config.status
? configure
Index: doc/gnupodrc.example
===
RCS file: /sources/gnupod/gnupod/doc/gnupodrc.example,v
retrieving revision 1.7
diff -u -r1.7 gnupodrc.example
--- doc/gnupodrc.example5 Jun 2009 12:55:56 - 1.7
+++ doc/gnupodrc.example17 Jun 2009 19:56:32 -
@@ -51,6 +51,14 @@
# mktunes.volume = +10
## Enforce iPod serial number:
# mktunes.fwguid = 000ba3100310abcf
+## Only keep some attributes to make the iTunesDB fit inside small RAM
+## The minimum attributes needed by the iPod are path and title
+## Valid attributes are:
+## title path album artist genre fdesc eq comment category composer group
+## desc podcastguid podcastrss chapterdata subtitle tvshow tvepisode
+## tvnetwork albumartist artistthe keywords sorttitle sortalbum
+## sortalbumartist sortcomposer sorttvshow
+# low_ram_attr = path title artist album
# *** gnupod_search.pl ***
Index: src/mktunes.pl
===
RCS file: /sources/gnupod/gnupod/src/mktunes.pl,v
retrieving revision 1.86
diff -u -r1.86 mktunes.pl
--- src/mktunes.pl 8 Dec 2007 10:26:08 - 1.86
+++ src/mktunes.pl 17 Jun 2009 19:56:32 -
@@ -41,7 +41,7 @@
$opts{mount} = $ENV{IPOD_MOUNTPOINT};
GetOptions(\%opts, version, help|h, ipod-name|n=s, mount|m=s,
volume|v=i, energy|e, fwguid|g=s);
-GNUpod::FooBar::GetConfig(\%opts, {'ipod-name'='s', mount='s', volume='i',
energy='b', fwguid='s', model='s'}, mktunes);
+GNUpod::FooBar::GetConfig(\%opts, {'ipod-name'='s', mount='s', volume='i',
energy='b', fwguid='s', model='s', low_ram_attr='s'}, mktunes);
$opts{'ipod-name'} ||= GNUpod ###__VERSION__###;
@@ -69,7 +69,12 @@
GNUpod::XMLhelper::doxml($con-{xml}) or usage(Could not read
$con-{xml}, did you run gnupod_INIT.pl ?);
print \r .$mktunes-GetFileCount. files parsed, assembling
iTunesDB...\n;
- $mktunes-WriteItunesDB;
+
+ my $keep = {};
+ foreach(split(/[ ,]+/,$opts{'low_ram_attr'})) {
+ $keep-{$_}++;
+ }
+ $mktunes-WriteItunesDB(keep=$keep);
if($fwguid) {
my $k = GNUpod::Hash58::HashItunesDB(FirewireId=$fwguid,
iTunesDB=$con-{itunesdb});
Index: src/tunes2pod.pl
===
RCS file: /sources/gnupod/gnupod/src/tunes2pod.pl,v
retrieving revision 1.49
diff -u -r1.49 tunes2pod.pl
--- src/tunes2pod.pl2 Feb 2008 11:42:58 - 1.49
+++ src/tunes2pod.pl17 Jun 2009 19:56:32 -
@@ -35,6 +35,8 @@
use vars qw(%opts);
$| = 1;
+my $xml_files_parsed=0;
+my $gtdb = {};
print tunes2pod.pl Version ###__VERSION__### (C) Adrian Ulrich\n;
@@ -65,6 +67,11 @@
exit(1);
}
+ print Parsing XML document...\n;
+ GNUpod::XMLhelper::doxml($con-{xml}) or usage(Could not read
$con-{xml}, did you run gnupod_INIT.pl ?);
+ GNUpod::XMLhelper::resetxml;
+ print \r .$xml_files_parsed. files parsed, converting
iTunesDB...\n;
+
open(ITUNES, $con-{itunesdb}) or usage(Could not open
$con-{itunesdb});
while(ITUNES) {}; sysseek(ITUNES,0,0); # the iPod is a slw
mass-storage device, slurp it into the fs-cache
@@ -197,7 +204,7 @@
sub MhitEnd {
my($self, %args) = @_;
if($self-{mode} == MODE_SONGS) {
- GNUpod::XMLhelper::mkfile({file=$self-{ctx}});
# Add file element to xml
+ GNUpod::XMLhelper::mkfile({file=MergeGtdbCtx($self-{ctx})});
# Add file element to xml
$self-{ctx} = ();
# And drop this buffer
my $i = ++$self-{count_songs_done};
if($i % 32 == 0) {
@@ -344,11 +351,35 @@
$self-ResetPlaylists; # Resets podcast and normal playlist data
}
-
-
-
-
-
+#
+# Merge GNUtunesDB with ctx
+sub MergeGtdbCtx {
+
Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?
Patch 2 bound the iTunesDB and GNUtunes.xml on id. Since gnupod
regenerates the id field on every mktunes run, other software might do
the same. So I figured binding them on the path attribute would be a
better idea.
Cheers,
Richard
? .gnupod_version
? Makefile
? autom4te.cache
? config.log
? config.status
? configure
Index: doc/gnupodrc.example
===
RCS file: /sources/gnupod/gnupod/doc/gnupodrc.example,v
retrieving revision 1.7
diff -u -r1.7 gnupodrc.example
--- doc/gnupodrc.example5 Jun 2009 12:55:56 - 1.7
+++ doc/gnupodrc.example18 Jun 2009 04:44:09 -
@@ -51,6 +51,14 @@
# mktunes.volume = +10
## Enforce iPod serial number:
# mktunes.fwguid = 000ba3100310abcf
+## Only keep some attributes to make the iTunesDB fit inside small RAM
+## The minimum attributes needed by the iPod are path and title
+## Valid attributes are:
+## title path album artist genre fdesc eq comment category composer group
+## desc podcastguid podcastrss chapterdata subtitle tvshow tvepisode
+## tvnetwork albumartist artistthe keywords sorttitle sortalbum
+## sortalbumartist sortcomposer sorttvshow
+# low_ram_attr = path title artist album
# *** gnupod_search.pl ***
Index: src/mktunes.pl
===
RCS file: /sources/gnupod/gnupod/src/mktunes.pl,v
retrieving revision 1.86
diff -u -r1.86 mktunes.pl
--- src/mktunes.pl 8 Dec 2007 10:26:08 - 1.86
+++ src/mktunes.pl 18 Jun 2009 04:44:09 -
@@ -41,7 +41,7 @@
$opts{mount} = $ENV{IPOD_MOUNTPOINT};
GetOptions(\%opts, version, help|h, ipod-name|n=s, mount|m=s,
volume|v=i, energy|e, fwguid|g=s);
-GNUpod::FooBar::GetConfig(\%opts, {'ipod-name'='s', mount='s', volume='i',
energy='b', fwguid='s', model='s'}, mktunes);
+GNUpod::FooBar::GetConfig(\%opts, {'ipod-name'='s', mount='s', volume='i',
energy='b', fwguid='s', model='s', low_ram_attr='s'}, mktunes);
$opts{'ipod-name'} ||= GNUpod ###__VERSION__###;
@@ -69,7 +69,12 @@
GNUpod::XMLhelper::doxml($con-{xml}) or usage(Could not read
$con-{xml}, did you run gnupod_INIT.pl ?);
print \r .$mktunes-GetFileCount. files parsed, assembling
iTunesDB...\n;
- $mktunes-WriteItunesDB;
+
+ my $keep = {};
+ foreach(split(/[ ,]+/,$opts{'low_ram_attr'})) {
+ $keep-{$_}++;
+ }
+ $mktunes-WriteItunesDB(keep=$keep);
if($fwguid) {
my $k = GNUpod::Hash58::HashItunesDB(FirewireId=$fwguid,
iTunesDB=$con-{itunesdb});
Index: src/tunes2pod.pl
===
RCS file: /sources/gnupod/gnupod/src/tunes2pod.pl,v
retrieving revision 1.49
diff -u -r1.49 tunes2pod.pl
--- src/tunes2pod.pl2 Feb 2008 11:42:58 - 1.49
+++ src/tunes2pod.pl18 Jun 2009 04:44:09 -
@@ -35,6 +35,8 @@
use vars qw(%opts);
$| = 1;
+my $xml_files_parsed=0;
+my $gtdb = {};
print tunes2pod.pl Version ###__VERSION__### (C) Adrian Ulrich\n;
@@ -65,6 +67,11 @@
exit(1);
}
+ print Parsing XML document...\n;
+ GNUpod::XMLhelper::doxml($con-{xml}) or usage(Could not read
$con-{xml}, did you run gnupod_INIT.pl ?);
+ GNUpod::XMLhelper::resetxml;
+ print \r .$xml_files_parsed. files parsed, converting
iTunesDB...\n;
+
open(ITUNES, $con-{itunesdb}) or usage(Could not open
$con-{itunesdb});
while(ITUNES) {}; sysseek(ITUNES,0,0); # the iPod is a slw
mass-storage device, slurp it into the fs-cache
@@ -197,7 +204,7 @@
sub MhitEnd {
my($self, %args) = @_;
if($self-{mode} == MODE_SONGS) {
- GNUpod::XMLhelper::mkfile({file=$self-{ctx}});
# Add file element to xml
+ GNUpod::XMLhelper::mkfile({file=MergeGtdbCtx($self-{ctx})});
# Add file element to xml
$self-{ctx} = ();
# And drop this buffer
my $i = ++$self-{count_songs_done};
if($i % 32 == 0) {
@@ -344,11 +351,35 @@
$self-ResetPlaylists; # Resets podcast and normal playlist data
}
-
-
-
-
-
+#
+# Merge GNUtunesDB with ctx
+sub MergeGtdbCtx {
+ my($Ctx) = @_;
+ return $Ctx unless $Ctx-{path} $gtdb-{$Ctx-{path}};
+ return {%{$gtdb-{$Ctx-{path}}}, %$Ctx};
+}
+
+#
+# Called by doxml if it finds a new file tag
+sub newfile {
+ my($item) = @_;
+ my $file = $item-{file};
+ my $path = $file-{path};
+
+ $xml_files_parsed++;
+ print \r .$xml_files_parsed. files parsed if $xml_files_parsed %
96 == 0;
+
+ return unless $path;
+ $gtdb-{$path} = {};
+ foreach(keys(%$file)){
+ $gtdb-{$path}-{$_}=$file-{$_};
+ }
+}
+
