Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hey Martin,
> My Java version encountered the same problem, but Java's exception handling
> just complained for that iteration of the loop and moved on. It seems that
> the tile location data is particularly problematic in terms of quality of
> rip, but I've also seen tile paint commands (xor and regular) that don't
> properly index into the palette, so you might want to check for those too. I
> have occaisonally also seen CDG commands that have a sub-command that is
> invalid, but that's fairly rare.
Yep, I've seen those invalid commands as well. Maybe I'll add some kind
of universal sanity-checker on the input data.
> As a separate note, I've been playing with the Java OpenGL bindings
> ("JOGL"), and I was wondering if any of you had considered throwing in a
> PyOpenGl capable renderer? It looks like it works well with PyGame...
Not on the radar at the moment, but if anyone listening is interested in
working on it...
Cheers,
Kelvin.
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Pykaraoke-discuss mailing list
Pykaraoke-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hey all,
My Java version encountered the same problem, but Java's exception handling
just complained for that iteration of the loop and moved on. It seems that
the tile location data is particularly problematic in terms of quality of
rip, but I've also seen tile paint commands (xor and regular) that don't
properly index into the palette, so you might want to check for those too. I
have occaisonally also seen CDG commands that have a sub-command that is
invalid, but that's fairly rare.
As a separate note, I've been playing with the Java OpenGL bindings
("JOGL"), and I was wondering if any of you had considered throwing in a
PyOpenGl capable renderer? It looks like it works well with PyGame...
Best,
Martin
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Kelvin
Lawson
Sent: Thursday, February 01, 2007 7:21 PM
To: William Ferrell
Cc: pykaraoke-discuss@lists.sourceforge.net; Daniel Richard G.
Subject: Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hey Will,
> I suspect this is either a numeric thing or a pygame thing -- aren't
> there some compiled bits in those libraries for speed purposes on some
> systems? I recall having a 64-bit problem myself when I was first
> mucking with PyKaraoke myself ;)
Yep I believe that one was a pygame vs 64-bit bug.
In this case there are no libraries involved, however. It's just our own C
module _pycdgAux.c which is reading the CDG bytes read directly from the CDG
file. The problem is that it's reading an x,y offset from the CDG file which
is outside of the 300x216 screen size, i.e. the data in the file at this
location isn't a perfect rip of the original CDG command. The C code was
then using this x,y offset to index into an array sized at 300x216. This was
what caused the seg fault that Daniel sent the debug info on. It won't
necessarily cause a seg fault on all machines though, as I found on my PC.
It would have just been stomping on some memory somewhere that it probably
shouldn't :-)
It turns out that both of these files fail when using the pure Python
implementation of pycdg as well, for the same reason: array index out of
bounds. I'd better fix that one up too. The lesson here is to sanity-check
the CDG commands as they aren't always perfectly ripped.
Kelvin.
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job
easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Pykaraoke-discuss mailing list
Pykaraoke-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Pykaraoke-discuss mailing list
Pykaraoke-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hey Will, > I suspect this is either a numeric thing or a pygame thing -- aren't > there some compiled bits in those libraries for speed purposes on some > systems? I recall having a 64-bit problem myself when I was first > mucking with PyKaraoke myself ;) Yep I believe that one was a pygame vs 64-bit bug. In this case there are no libraries involved, however. It's just our own C module _pycdgAux.c which is reading the CDG bytes read directly from the CDG file. The problem is that it's reading an x,y offset from the CDG file which is outside of the 300x216 screen size, i.e. the data in the file at this location isn't a perfect rip of the original CDG command. The C code was then using this x,y offset to index into an array sized at 300x216. This was what caused the seg fault that Daniel sent the debug info on. It won't necessarily cause a seg fault on all machines though, as I found on my PC. It would have just been stomping on some memory somewhere that it probably shouldn't :-) It turns out that both of these files fail when using the pure Python implementation of pycdg as well, for the same reason: array index out of bounds. I'd better fix that one up too. The lesson here is to sanity-check the CDG commands as they aren't always perfectly ripped. Kelvin. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Pykaraoke-discuss mailing list Pykaraoke-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
Re: [Pykaraoke-discuss] pycdg segfault on amd64
On 2/1/07, Kelvin Lawson <[EMAIL PROTECTED]> wrote: Hi Daniel, > This problem was indeed happening with multiple .cdg files, though I can't > say if it applied to them all. Because pykaraoke worked with the same files > on x86 without incident, however, I doubt that corrupted/rogue data is the > issue. > > I'll send you a private e-mail re: obtaining the OGG+CDG files in > question for testing purposes. I tried these CDGs on my AMD64 and they worked fine. I guess it's system dependent whether the out of bounds array access will hit somewhere that will cause a seg fault. I suspect this is either a numeric thing or a pygame thing -- aren't there some compiled bits in those libraries for speed purposes on some systems? I recall having a 64-bit problem myself when I was first mucking with PyKaraoke myself ;) - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___ Pykaraoke-discuss mailing list Pykaraoke-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hi Daniel, > I patched the code as you indicated, and the segfaults are gone. Excellent news, the fix is now in CVS. > This problem was indeed happening with multiple .cdg files, though I can't > say if it applied to them all. Because pykaraoke worked with the same files > on x86 without incident, however, I doubt that corrupted/rogue data is the > issue. > > I'll send you a private e-mail re: obtaining the OGG+CDG files in > question for testing purposes. I tried these CDGs on my AMD64 and they worked fine. I guess it's system dependent whether the out of bounds array access will hit somewhere that will cause a seg fault. I did notice that both those files had classic symptoms of errors in rips. Odd bits of graphics the wrong colour etc, which is why I'm pretty sure that these odd pixel values are coming from a very slightly corrupted rip. They both play for a minute or two before one of these out-of-bounds pixel offsets occur. This is pretty common though, so it's a good thing we've nailed it. Thanks for an excellent bug report by the way. There have been mutterings of AMD64 problems since the optimised C version came out, but nothing with all of the detail necessary to fix it, so thanks! Kelvin. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Pykaraoke-discuss mailing list Pykaraoke-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hi Kelvin, I patched the code as you indicated, and the segfaults are gone. This problem was indeed happening with multiple .cdg files, though I can't say if it applied to them all. Because pykaraoke worked with the same files on x86 without incident, however, I doubt that corrupted/rogue data is the issue. I'll send you a private e-mail re: obtaining the OGG+CDG files in question for testing purposes. --Daniel (Please Cc: any replies to me) On Thu, 2007 Feb 01 00:15:25 +, Kelvin Lawson wrote: > Hi Daniel, > > I've been moving house the last couple of weeks so have just got round > to looking into this. > > I originally thought this looked like a sizeof(int) discrepancy > somewhere. I thought perhaps one component (pygame, SDL or _pycdgAux.c) > had been built as 32-bit, or perhaps one of the underlying libraries is > not 64-bit safe in some places. > > Anyway, on looking into it further I see that the row_index value is 378 > (thanks for all the excellent debug info by the way). This is an invalid > index into the array, as the CDG screen width and hence cdgSurfarray[][] > is only max 300. For some reason this line in cdgTileBlockCommon() is > reading an attempt to set pixel offset 378: > > row_index = ((packd->data[3] & 0x3f) * 6); > > CDG rips occasionally contain corrupted bytes, so I could well imagine > that this could happen on the odd CDG file but it sounds like this > probably happens on all files (please correct me if I'm wrong there). > > Whether it's corrupted data or not, we really should be range-checking > the row and column index here anyway. Could you try adding the following > lines in cdgTileBlockCommon() please: > > // 2 lines of current code below > column_index = ((packd->data[2] & 0x1f) * 12); > row_index = ((packd->data[3] & 0x3f) * 6); > > // Enter the following 4 lines > if (column_index > (CDG_FULL_HEIGHT - 12)) > column_index = (CDG_FULL_HEIGHT - 12); > if (row_index > (CDG_FULL_WIDTH - 6)) > row_index = (CDG_FULL_WIDTH - 6); > > If it's rogue data, then hopefully this will fix the seg fault. If it's > not rogue data and somehow we're reading the CDG data out incorrectly, > well we'll find that out. I hope you don't mind trying this out - I do > actually have an AMD64 machine here but unfortunately I can't reproduce > the problem. Perhaps if you could send the offending cdg+ogg files I > could reproduce it. > > Many thanks, > Kelvin. > -- NAME = Daniel Richard G. ## Remember, skunks _\|/_ meef? EMAIL1 = [EMAIL PROTECTED]## don't smell bad---(/o|o\) / EMAIL2 = [EMAIL PROTECTED] ## it's the people who < (^),> WWW= http://www.**.org/ ## annoy them that do!/ \ -- (** = site not yet online) - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Pykaraoke-discuss mailing list Pykaraoke-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
Re: [Pykaraoke-discuss] pycdg segfault on amd64
Hi Daniel, I've been moving house the last couple of weeks so have just got round to looking into this. I originally thought this looked like a sizeof(int) discrepancy somewhere. I thought perhaps one component (pygame, SDL or _pycdgAux.c) had been built as 32-bit, or perhaps one of the underlying libraries is not 64-bit safe in some places. Anyway, on looking into it further I see that the row_index value is 378 (thanks for all the excellent debug info by the way). This is an invalid index into the array, as the CDG screen width and hence cdgSurfarray[][] is only max 300. For some reason this line in cdgTileBlockCommon() is reading an attempt to set pixel offset 378: row_index = ((packd->data[3] & 0x3f) * 6); CDG rips occasionally contain corrupted bytes, so I could well imagine that this could happen on the odd CDG file but it sounds like this probably happens on all files (please correct me if I'm wrong there). Whether it's corrupted data or not, we really should be range-checking the row and column index here anyway. Could you try adding the following lines in cdgTileBlockCommon() please: // 2 lines of current code below column_index = ((packd->data[2] & 0x1f) * 12); row_index = ((packd->data[3] & 0x3f) * 6); // Enter the following 4 lines if (column_index > (CDG_FULL_HEIGHT - 12)) column_index = (CDG_FULL_HEIGHT - 12); if (row_index > (CDG_FULL_WIDTH - 6)) row_index = (CDG_FULL_WIDTH - 6); If it's rogue data, then hopefully this will fix the seg fault. If it's not rogue data and somehow we're reading the CDG data out incorrectly, well we'll find that out. I hope you don't mind trying this out - I do actually have an AMD64 machine here but unfortunately I can't reproduce the problem. Perhaps if you could send the offending cdg+ogg files I could reproduce it. Many thanks, Kelvin. > I've confirmed this with pykaraoke built from current CVS code. If anyone > needs it, I can provide the cdg+ogg files used below. > > BEGIN (slightly edited) GDB SESSION > $ PYTHONPATH=lib/python2.4/site-packages gdb /usr/bin/python2.4 > GNU gdb 6.4.90-debian > Copyright (C) 2006 Free Software Foundation, Inc. > > (gdb) run bin/pycdg 02--Jolly_Holiday--karaoke.cdg > Starting program: /usr/bin/python2.4 bin/pycdg 02--Jolly_Holiday--karaoke.cdg > (no debugging symbols found) > [Thread debugging using libthread_db enabled] > [New Thread 47461752184672 (LWP 10233)] > (no debugging symbols found) > [New Thread 1082132832 (LWP 10236)] > [New Thread 1090525536 (LWP 10237)] > [Thread 1090525536 (zombie) exited] > [New Thread 1090525536 (LWP 10239)] > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 47461752184672 (LWP 10233)] > 0x2b2a90ba6489 in __cdgTileBlockCommon (self=0x2ae18010, > packd=0x7fff1e4879c0, xor=1) at _pycdgAux.c:783 > 783 self->__cdgSurfarray[row_index + j][column_index + i] = > self->__cdgColourTable[new_col]; > (gdb) p row_index > $2 = 378 > (gdb) p j > $3 = 0 > (gdb) p column_index > $4 = 36 > (gdb) p i > $5 = 0 > (gdb) p new_col > $6 = 0 > (gdb) p self->__cdgColourTable[new_col] > $7 = 11149824 > (gdb) p self->__cdgSurfarray[row_index + j][column_index + i] > Cannot access memory at address 0x2ae779fc > (gdb) where > #0 0x2b2a90ba6489 in __cdgTileBlockCommon (self=0x2ae18010, > packd=0x7fff1e4879c0, xor=1) at _pycdgAux.c:783 > #1 0x2b2a90ba583f in __cdgPacketProcess (self=0x2ae18010, > packd=0x7fff1e4879c0) at _pycdgAux.c:464 > #2 0x2b2a90ba53bd in CdgPacketReader_DoPackets (self=0x2ae18010, > args=0x2b2a8c69e050, kwds=0x0) at _pycdgAux.c:316 > #3 0x00474517 in PyEval_EvalFrame () > #4 0x00472c72 in PyEval_EvalFrame () > #5 0x00472c72 in PyEval_EvalFrame () > #6 0x00472c72 in PyEval_EvalFrame () > #7 0x00472c72 in PyEval_EvalFrame () > #8 0x0047505c in PyEval_EvalCodeEx () > #9 0x00475182 in PyEval_EvalCode () > #10 0x00498da2 in PyRun_FileExFlags () > #11 0x00498fa0 in PyRun_SimpleFileExFlags () > #12 0x00410a5f in Py_Main () > #13 0x2b2a8cbf04ca in __libc_start_main () from /lib/libc.so.6 > #14 0x0040ff3a in _start () > END GDB SESSION > > pykaraoke 0.5 handled this same file without a problem on an i386 system, > so some 64-bit snafu is likely at fault. > > (Please Cc: any replies to me, as I am not subscribed to this list) > > > --Daniel > > - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Pykaraoke-discuss mailing list Pykaraoke-discuss@lists.sourceforge.net https://
[Pykaraoke-discuss] pycdg segfault on amd64
I've confirmed this with pykaraoke built from current CVS code. If anyone needs it, I can provide the cdg+ogg files used below. BEGIN (slightly edited) GDB SESSION $ PYTHONPATH=lib/python2.4/site-packages gdb /usr/bin/python2.4 GNU gdb 6.4.90-debian Copyright (C) 2006 Free Software Foundation, Inc. (gdb) run bin/pycdg 02--Jolly_Holiday--karaoke.cdg Starting program: /usr/bin/python2.4 bin/pycdg 02--Jolly_Holiday--karaoke.cdg (no debugging symbols found) [Thread debugging using libthread_db enabled] [New Thread 47461752184672 (LWP 10233)] (no debugging symbols found) [New Thread 1082132832 (LWP 10236)] [New Thread 1090525536 (LWP 10237)] [Thread 1090525536 (zombie) exited] [New Thread 1090525536 (LWP 10239)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 47461752184672 (LWP 10233)] 0x2b2a90ba6489 in __cdgTileBlockCommon (self=0x2ae18010, packd=0x7fff1e4879c0, xor=1) at _pycdgAux.c:783 783 self->__cdgSurfarray[row_index + j][column_index + i] = self->__cdgColourTable[new_col]; (gdb) p row_index $2 = 378 (gdb) p j $3 = 0 (gdb) p column_index $4 = 36 (gdb) p i $5 = 0 (gdb) p new_col $6 = 0 (gdb) p self->__cdgColourTable[new_col] $7 = 11149824 (gdb) p self->__cdgSurfarray[row_index + j][column_index + i] Cannot access memory at address 0x2ae779fc (gdb) where #0 0x2b2a90ba6489 in __cdgTileBlockCommon (self=0x2ae18010, packd=0x7fff1e4879c0, xor=1) at _pycdgAux.c:783 #1 0x2b2a90ba583f in __cdgPacketProcess (self=0x2ae18010, packd=0x7fff1e4879c0) at _pycdgAux.c:464 #2 0x2b2a90ba53bd in CdgPacketReader_DoPackets (self=0x2ae18010, args=0x2b2a8c69e050, kwds=0x0) at _pycdgAux.c:316 #3 0x00474517 in PyEval_EvalFrame () #4 0x00472c72 in PyEval_EvalFrame () #5 0x00472c72 in PyEval_EvalFrame () #6 0x00472c72 in PyEval_EvalFrame () #7 0x00472c72 in PyEval_EvalFrame () #8 0x0047505c in PyEval_EvalCodeEx () #9 0x00475182 in PyEval_EvalCode () #10 0x00498da2 in PyRun_FileExFlags () #11 0x00498fa0 in PyRun_SimpleFileExFlags () #12 0x00410a5f in Py_Main () #13 0x2b2a8cbf04ca in __libc_start_main () from /lib/libc.so.6 #14 0x0040ff3a in _start () END GDB SESSION pykaraoke 0.5 handled this same file without a problem on an i386 system, so some 64-bit snafu is likely at fault. (Please Cc: any replies to me, as I am not subscribed to this list) --Daniel -- NAME = Daniel Richard G. ## Remember, skunks _\|/_ meef? EMAIL1 = [EMAIL PROTECTED]## don't smell bad---(/o|o\) / EMAIL2 = [EMAIL PROTECTED] ## it's the people who < (^),> WWW= http://www.**.org/ ## annoy them that do!/ \ -- (** = site not yet online) - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Pykaraoke-discuss mailing list Pykaraoke-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pykaraoke-discuss
