Re: [Savage40] Re: Savage dri does not resume from disk
Hi Dario, With the help of Sergey Zharkov I narrowed down the problem. I'm CCing this to the dri-devel and savage40 lists as a summary. The problem seems to be in the AGP GART driver which is in the Linux kernel. A new snapshot won't help you, I can't fix the problem there. I haven't had the time to prepare a good bug-report for the kernel developers yet. I would have to upgrade the kernel and get swsusp to work with it first (last time I tried this was several months ago). I think I won't have time for this in the next two weeks. :( See my comment below for a good workaround. Am Montag, den 21.03.2005, 21:11 +0100 schrieb Dario Saccavino: I am experiencing the same problem with swsusp2 (version 2.1.8, kernel 2.6.11): after resuming from suspend to disk, if I launch a 3D application the computer hangs. I can avoid the problem completely by specifying DmaMode none in xorg.conf or by shutting down X before suspending. In order to make swsusp work reliably you should set BusType to PCI. This will disable any use of AGP memory, both for textures and for DMA. With BusType PCI you do not need to set DmaMode to None, but you may get slightly better performance. Your mileage may vary. I'm using a CVS snapshot dating about 2 weeks ago; is this fixed now, or going to be? I can supply logs and testing, if you need some help. Dario Regards, Felix On Wed, 16 Mar 2005 14:47:01 +0300, Sergey Zharkov [EMAIL PROTECTED] wrote: Felix, I've tried some coding to make savage resume looking at radeon resume code but failed completely :(( Unfortunately I'am ABAP/4 developer - not a C guru. The only thing that may help real dri developers to suggest how to resume - when I switched DMA mode from command or vertex to None i've lost about 5 % of speed but the dri system was able to resume from disk if no glx apps were running. And even with glx apps running it was resuming, but the glx apps were resuming with black window. Anyway after resume I was able to start glx apps and they worked, with dma enebled I had to pull a power cord and battery out to restart. And the bug is not in the kernel I guess - restarting X after resume with enabled dma works fine. --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595alloc_id=14396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Am Donnerstag, den 17.03.2005, 08:17 +0300 schrieb Sergey Zharkov: Felix, It works great!!! Thanks for the patch - now my suspend-resume works like piece of cake - without DMA modes yet Didn't you say it worked with PCI DMA? With BusType PCI and DmaMode != None it is using DMA. Though PCI DMA is slower than AGP DMA, in some cases even slower than no DMA at all. :-/ but even without dma the performance is much better than on savage DRI 1.0. If there will be any changes in agp - i am ready to test. The patch from yesterdays mm4 version did not help. Hmm, ok. I'll commit my patch to Xorg CVS and see if I can get the via-agp issue resolved with the kernel people. Regards, Felix On Thu, 2005-03-17 at 00:26 +0100, Felix Kühling wrote: Am Mittwoch, den 16.03.2005, 16:26 +0100 schrieb Felix Kühling: Am Mittwoch, den 16.03.2005, 17:35 +0300 schrieb Sergey Zharkov: [snip] What I'am afraid of is that savage_dri or savage_drv drivers writhe something into the chip hardware that is not loaded there by kernel modules start - so if i do resume from ram the chip registers keep those values and resume even if glx app was running in time of suspend. But if i resume from disk kernel modules are loaded back with initial values - but some commands or chip register values that were initialised by xorg or dri drivers need to be restored on enterVT function. The same way as radeon do. As far as I can tell, the Savage hardware state is almost completely restored in enterVT. EnterVT doesn't care about 3D state though, that's why 3D windows that were running during suspend are black after resume. I think it would be easy to get it working with a running 3D application during suspension. The Xserver would have to invalidate the 3D hardware state in the sarea, so that the 3D driver restores it after resume. I'll send you a patch later tonight for testing this. Please try the attached patch (apply in xc/programs/Xserver/hw/xfree86/drivers/savage). Test with BusType PCI, with a 3D app running while suspending. On resume it should now display correctly. If it works, I will commit this to Xorg CVS. [snip] Thanks, Felix -- | Felix Kühling [EMAIL PROTECTED] http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 | --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95alloc_id396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Am Mittwoch, den 16.03.2005, 14:47 +0300 schrieb Sergey Zharkov: Felix, I've tried some coding to make savage resume looking at radeon resume code but failed completely :(( Unfortunately I'am ABAP/4 developer - not a C guru. The only thing that may help real dri developers to suggest how to resume - when I switched DMA mode from command or vertex to None i've lost about 5 % of speed but the dri system was able to resume from disk if no glx apps were running. And even with glx apps running it was resuming, but the glx apps were resuming with black window. Anyway after resume I was able to start glx apps and they worked, with dma enebled I had to pull a power cord and battery out to restart. And the bug is not in the kernel I guess - restarting X after resume with enabled dma works fine. Could you try if Option BusType PCI makes any difference with DMA enabled? Try it with a DRI app running while suspending and without. This is to confirm if the problem is in the Savage DRM or in the AGP driver (or both :( ). Thanks, Felix -- | Felix Kühling [EMAIL PROTECTED] http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 | --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95alloc_id396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Am Mittwoch, den 16.03.2005, 15:17 +0300 schrieb Sergey Zharkov: Setting BusType to PCI and even BusType AGP with DmaType PCI helps - the thing resumes. So it seems to be a bug in AGP dma resuming. But not in kernel one - there is a resume code in via-agp driver, that worked for DRI 1.0 You mean the old Savage DRM driver 1.0? That one didn't use DMA, which is about the same as setting DmaMode to None now. Another thing you can try is running textured applications with only AGP textures (with BusType AGP and DmaType PCI). Run the application like this: texture_heaps=2 textured DRI application This will disable the local memory texture heap and use only the AGP heap. If our theory is correct, it should lock up after resume. Regards, Felix On Wed, 2005-03-16 at 13:01 +0100, Felix Kühling wrote: Am Mittwoch, den 16.03.2005, 14:47 +0300 schrieb Sergey Zharkov: Felix, I've tried some coding to make savage resume looking at radeon resume code but failed completely :(( Unfortunately I'am ABAP/4 developer - not a C guru. The only thing that may help real dri developers to suggest how to resume - when I switched DMA mode from command or vertex to None i've lost about 5 % of speed but the dri system was able to resume from disk if no glx apps were running. And even with glx apps running it was resuming, but the glx apps were resuming with black window. Anyway after resume I was able to start glx apps and they worked, with dma enebled I had to pull a power cord and battery out to restart. And the bug is not in the kernel I guess - restarting X after resume with enabled dma works fine. Could you try if Option BusType PCI makes any difference with DMA enabled? Try it with a DRI app running while suspending and without. This is to confirm if the problem is in the Savage DRM or in the AGP driver (or both :( ). Thanks, Felix -- | Felix Kühling [EMAIL PROTECTED] http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 | --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95alloc_id396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Felix, I've tried some coding to make savage resume looking at radeon resume code but failed completely :(( Unfortunately I'am ABAP/4 developer - not a C guru. The only thing that may help real dri developers to suggest how to resume - when I switched DMA mode from command or vertex to None i've lost about 5 % of speed but the dri system was able to resume from disk if no glx apps were running. And even with glx apps running it was resuming, but the glx apps were resuming with black window. Anyway after resume I was able to start glx apps and they worked, with dma enebled I had to pull a power cord and battery out to restart. And the bug is not in the kernel I guess - restarting X after resume with enabled dma works fine. On Wed, 2005-03-09 at 00:31 +0100, Felix Kühling wrote: Am Mittwoch, den 09.03.2005, 02:08 +0300 schrieb Sergey Zharkov: Felix, Sorry for reporting bug on this address - just culd not find any place t report for DRI, and you seems to be the only man who actually program the driver. The DRI developers list is [EMAIL PROTECTED] Savage issues can also be discussed at [EMAIL PROTECTED], which is a moderated list. I've recently moved from Savage DRM 1.x to 2.x from cvs with kernel 2.6.11 - I've received 10% increase in speed, quality of rendering looks better as well (i got rid of some ugly artifacts on my favorite SeriousSam game) - the only thing DRI does not resume from disk anymore. It resumes from memory (acpi state S3) fine, but when I resume from disk even having no apps using DRI during suspend-resume, each time I try to start 3D application everything hangs and the only chance I have is to switch power off. Just curious are you aware about this - may be you have any idea how to make resume from disk working again, Not really. Resume from disk was only half working for me with DRM 1.x. It locked up after some time with more demanding applications. I was guessing that it locked up whenever the driver started using AGP textures, but I never looked into it more seriously. With the new driver I have never tried resume from disk, and I don't know what would be needed to make it work. I'm CCing to dri-devel. Can someone give me a brief summary what a driver needs to support resume from disk? I seem to remember that i915 and unichrome saw changes in that area in the last months. The hardware is Twinhead Efio 121A, KN266 based laptop Thank you Sergey -- Sergey Zharkov [EMAIL PROTECTED] --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595alloc_id=14396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Am Mittwoch, den 16.03.2005, 17:35 +0300 schrieb Sergey Zharkov: Felix, Unfortunately I never posted a bug for kernel and have no idea how to do that. I found a related bug: http://bugzilla.kernel.org/show_bug.cgi?id=3864. Though that user was not getting lockups, just some distortions. Probably with different graphics hardware (no details in the report). You can register at bugzilla.kernel.org (click on New Account near the bottom of the page) and make an additional comment. I have tryed all kernel versions starting from 2.6.11 - mm1,2,3 , ac series, nitro patches, builtin swsuspend as well as swsuspend2 2.1.8 That's good. - 2.1.8.2 . Hardware is Twinhead Efio 121A laptop Via KN266 chipset, Athlon XP-M 1700+. lspci output would be nice too (for them). The only thing - I am not shure that the via-agp driver is causing this bug - if I restart X after resuming I get DRM working again, so agp module is there. When X is started it allocates AGP memory, when it quits, the memory is freed again. Between suspend and resume via-agp it has to remember that state and restore the AGP hardware properly on resume. That doesn't seem to work correctly. Restarting X is a reinitialization of the relevant AGP hardware state and restores AGP to working order. (That's my understanding anyway) Actually swsusp2 is auloading all modules on suspend and loading them back - so I have agp functionality restarted properly. When you suspend with DRI enabled, the AGP module is used, so swsusp can't unload it. If it forces unloading, you're in trouble, but I don't think it would do that. What I'am afraid of is that savage_dri or savage_drv drivers writhe something into the chip hardware that is not loaded there by kernel modules start - so if i do resume from ram the chip registers keep those values and resume even if glx app was running in time of suspend. But if i resume from disk kernel modules are loaded back with initial values - but some commands or chip register values that were initialised by xorg or dri drivers need to be restored on enterVT function. The same way as radeon do. As far as I can tell, the Savage hardware state is almost completely restored in enterVT. EnterVT doesn't care about 3D state though, that's why 3D windows that were running during suspend are black after resume. I think it would be easy to get it working with a running 3D application during suspension. The Xserver would have to invalidate the 3D hardware state in the sarea, so that the 3D driver restores it after resume. I'll send you a patch later tonight for testing this. The only thing I could not find is what is actually written to chip - probably it could be easeir for the guys who really create the driver. If you can suggest some places in code (function names etc ) which do modify hardware registers I could try to play with them. Actually what radeon does (if I got it correct ) - is reinitialising DMA each enterVT function, but I could not find function that just reinitialise DMA in savage code. KernelInit or AGPInit in EnterVT restarted the whole X server instead or restoring DMA state. DMA itself seems to be working fine, because it worked with PCI DMA. I'm pretty sure it's AGP that's broken after resume. On Wed, 2005-03-16 at 15:24 +0100, Felix Kühling wrote: Am Mittwoch, den 16.03.2005, 15:36 +0300 schrieb Sergey Zharkov: You right - just running texture_heaps=2 lament killed my system immediately. Ok, that makes it very likely that the via-agp driver is at fault. We should report a bug in the Linux bugzilla. First we should test if the latest kernel version is affected too. Which kernel version have you been using? Since you have done all the testing and I am otherwise busy ATM it would be very helpful if you could take care or reporting the bug in the kernel bugzilla. I suppose you know that you have to provide accurate information about your hardware, what tests you performed and the results ... Send the bug id to dri-devel, so interested DRI experts (me for one ;-) can help resolve this. Thanks, Felix On Wed, 2005-03-16 at 13:30 +0100, Felix Kühling wrote: Am Mittwoch, den 16.03.2005, 15:17 +0300 schrieb Sergey Zharkov: Setting BusType to PCI and even BusType AGP with DmaType PCI helps - the thing resumes. So it seems to be a bug in AGP dma resuming. But not in kernel one - there is a resume code in via-agp driver, that worked for DRI 1.0 You mean the old Savage DRM driver 1.0? That one didn't use DMA, which is about the same as setting DmaMode to None now. Another thing you can try is running textured applications with only AGP textures (with BusType AGP and DmaType PCI). Run the application like this: texture_heaps=2 textured DRI application This will disable the local memory texture heap and use only the AGP heap. If our theory is correct,
Re: Savage dri does not resume from disk
I found a related bug: http://bugzilla.kernel.org/show_bug.cgi?id=3864. Though that user was not getting lockups, just some distortions. Probably with different graphics hardware (no details in the report). - 2.1.8.2 . Hardware is Twinhead Efio 121A laptop Via KN266 chipset, Athlon XP-M 1700+. I haven't been able to make S3 (suspend-to-ram) work in my laptop (Acer Aspire 1314LC). Although I can recover the X, and use the mouse and keyboard, there are problems with waking up the HD. Maybe other users of the same hardware can report about it: 00:00.0 Host bridge: VIA Technologies, Inc. P/KN266 Host Bridge Subsystem: VIA Technologies, Inc. P/KN266 Host Bridge Flags: bus master, 66Mhz, medium devsel, latency 8 Memory at b000 (32-bit, prefetchable) [size=128M] Capabilities: available only to root 00:01.0 PCI bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266 AGP] (prog-if 00 [Normal decode]) Flags: bus master, 66Mhz, medium devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: c000-dfff Memory behind bridge: e000-efff Prefetchable memory behind bridge: a000-afff Capabilities: available only to root Albert. --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595alloc_id=14396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Am Mittwoch, den 16.03.2005, 16:26 +0100 schrieb Felix Kühling:
Am Mittwoch, den 16.03.2005, 17:35 +0300 schrieb Sergey Zharkov:
[snip]
What I'am afraid of is that savage_dri or savage_drv drivers writhe
something into the chip hardware that is not loaded there by kernel
modules start - so if i do resume from ram the chip registers keep those
values and resume even if glx app was running in time of suspend. But if
i resume from disk kernel modules are loaded back with initial values -
but some commands or chip register values that were initialised by xorg
or dri drivers need to be restored on enterVT function. The same way as
radeon do.
As far as I can tell, the Savage hardware state is almost completely
restored in enterVT. EnterVT doesn't care about 3D state though, that's
why 3D windows that were running during suspend are black after resume.
I think it would be easy to get it working with a running 3D application
during suspension. The Xserver would have to invalidate the 3D hardware
state in the sarea, so that the 3D driver restores it after resume. I'll
send you a patch later tonight for testing this.
Please try the attached patch (apply in
xc/programs/Xserver/hw/xfree86/drivers/savage). Test with BusType PCI,
with a 3D app running while suspending. On resume it should now display
correctly. If it works, I will commit this to Xorg CVS.
[snip]
Thanks,
Felix
--
| Felix Kühling [EMAIL PROTECTED] http://fxk.de.vu |
| PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 |
--- ./savage_driver.c.~1.31.~ 2005-03-03 17:33:26.0 +0100
+++ ./savage_driver.c 2005-03-17 00:16:33.933174152 +0100
@@ -31,6 +31,7 @@
#ifdef XF86DRI
#define _XF86DRI_SERVER_
#include savage_dri.h
+#include savage_sarea.h
#endif
@@ -2025,6 +2026,7 @@
#ifdef XF86DRI
SavagePtr psav= SAVPTR(pScrn);
ScreenPtr pScreen;
+SAVAGESAREAPrivPtr pSAREAPriv;
#endif
TRACE((SavageEnterVT(%d)\n, flags));
@@ -2035,6 +2037,10 @@
#ifdef XF86DRI
if (psav-directRenderingEnabled) {
pScreen = screenInfo.screens[scrnIndex];
+ pSAREAPriv = (SAVAGESAREAPrivPtr)DRIGetSAREAPrivate(pScreen);
+ /* Assume that 3D state was clobbered, invalidate it by
+ * changing ctxOwner in the sarea. */
+ pSAREAPriv-ctxOwner = DRIGetContext(pScreen);
DRIUnlock(pScreen);
psav-LockHeld = 0;
}
Re: Savage dri does not resume from disk
Am Mittwoch, den 09.03.2005, 02:08 +0300 schrieb Sergey Zharkov: Felix, Sorry for reporting bug on this address - just culd not find any place t report for DRI, and you seems to be the only man who actually program the driver. The DRI developers list is [EMAIL PROTECTED] Savage issues can also be discussed at [EMAIL PROTECTED], which is a moderated list. I've recently moved from Savage DRM 1.x to 2.x from cvs with kernel 2.6.11 - I've received 10% increase in speed, quality of rendering looks better as well (i got rid of some ugly artifacts on my favorite SeriousSam game) - the only thing DRI does not resume from disk anymore. It resumes from memory (acpi state S3) fine, but when I resume from disk even having no apps using DRI during suspend-resume, each time I try to start 3D application everything hangs and the only chance I have is to switch power off. Just curious are you aware about this - may be you have any idea how to make resume from disk working again, Not really. Resume from disk was only half working for me with DRM 1.x. It locked up after some time with more demanding applications. I was guessing that it locked up whenever the driver started using AGP textures, but I never looked into it more seriously. With the new driver I have never tried resume from disk, and I don't know what would be needed to make it work. I'm CCing to dri-devel. Can someone give me a brief summary what a driver needs to support resume from disk? I seem to remember that i915 and unichrome saw changes in that area in the last months. The hardware is Twinhead Efio 121A, KN266 based laptop Thank you Sergey -- | Felix Kühling [EMAIL PROTECTED] http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 | --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95alloc_id396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
On Wed, 09 Mar 2005 00:31:02 +0100, Felix Kühling [EMAIL PROTECTED] wrote: Am Mittwoch, den 09.03.2005, 02:08 +0300 schrieb Sergey Zharkov: Felix, Sorry for reporting bug on this address - just culd not find any place t report for DRI, and you seems to be the only man who actually program the driver. The DRI developers list is [EMAIL PROTECTED] Savage issues can also be discussed at [EMAIL PROTECTED], which is a moderated list. I've recently moved from Savage DRM 1.x to 2.x from cvs with kernel 2.6.11 - I've received 10% increase in speed, quality of rendering looks better as well (i got rid of some ugly artifacts on my favorite SeriousSam game) - the only thing DRI does not resume from disk anymore. It resumes from memory (acpi state S3) fine, but when I resume from disk even having no apps using DRI during suspend-resume, each time I try to start 3D application everything hangs and the only chance I have is to switch power off. Just curious are you aware about this - may be you have any idea how to make resume from disk working again, Not really. Resume from disk was only half working for me with DRM 1.x. It locked up after some time with more demanding applications. I was guessing that it locked up whenever the driver started using AGP textures, but I never looked into it more seriously. With the new driver I have never tried resume from disk, and I don't know what would be needed to make it work. I'm CCing to dri-devel. Can someone give me a brief summary what a driver needs to support resume from disk? I seem to remember that i915 and unichrome saw changes in that area in the last months. radeon has support for suspect resume as well. you need to add pm hooks into your agp module to re-initialize it on resume (some agp drivers may already have this). you also need to re-init the video hardware properly. the radeon DDX does this in the enter/leaveVT functions. this page may be of help: http://cpbotha.net/dri_resume.html The Xv stuff may need resume support too. Alex The hardware is Twinhead Efio 121A, KN266 based laptop Thank you Sergey -- | Felix Kühling [EMAIL PROTECTED] http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 | --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95alloc_id396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
Re: Savage dri does not resume from disk
Alex Deucher [EMAIL PROTECTED] writes: radeon has support for suspect resume as well. you need to add pm hooks into your agp module to re-initialize it on resume (some agp drivers may already have this). you also need to re-init the video hardware properly. the radeon DDX does this in the enter/leaveVT functions. this page may be of help: http://cpbotha.net/dri_resume.html The Xv stuff may need resume support too. there is a problem with the radeon resume though. on my thinkpad t40, running glxgears the frame rate drops from about 3300 to 3000 and if i run more graphics intensive applications (like flightgear for example) they are noticeably slower and feel choppy. restarting the x server cures the problem, so i think this has something to do with how the hardware is being re-initialized on resume but i don't know how to debug it. --alex-- -- | I believe the moment is at hand when, by a paranoiac and active | | advance of the mind, it will be possible (simultaneously with | | automatism and other passive states) to systematize confusion | | and thus to help to discredit completely the world of reality. | --- SF email is sponsored by - The IT Product Guide Read honest candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595alloc_id=14396op=click -- ___ Dri-devel mailing list Dri-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dri-devel
