On 8/20/06, Zach Deedler <[EMAIL PROTECTED]> wrote:
I wouldn't expect using a networked drive to help any if you are using PIO in either case.
Well, its possible, but I've done experiments with paging on single processor (before the days of hyperthreading0 machines with little penalty. The issue is one of cooperation between threads.
The reason I brought up DMA, is that, if the paging thread is using Programmed I/O to transfer data, then it will be, indeed, hogging up the CPU during file transfers. If you are using DMA, then file transfers should be a tiny hiccup in CPU usage. When vertical sync is enabled, you can be sure that there will be a period of time each frame that the rendering thread will be dormant. It is during this time that you want the pager to do its work. If the pager is "stuck" in programmed I/O, then it will not relinquish the CPU to the rendering thread until its operation is done.
The graph you posted is quite helpful. However, there is still a mystery as to why the database thread is so CPU bound during file I/O. It does seem to indicate some kind of spin lock that is stuck.
Now, if I were handling this problem under some kind of Unix, I would use prof and get a profile of the database thread to find out where it is spending most of its cycles. Do you have that kind of capability with your compilers/tools? If so, that is probably the best place to start at narrowing down where exactly CPU time is being spent.
-don
Hi Don,I did try messing with the DMA settings from another link I had to try and get Ultra DMA Mode 6, but it didn't work. Looks like I'm using Ultra DMA Mode 4. Also, loading files off another computer (ie. drive Z:\ mapped to a different computer), did not help either.
I wouldn't expect using a networked drive to help any if you are using PIO in either case.
Is it possible that the paging thread and main thread are running on the same processor on Windows, but not Linux?
Well, its possible, but I've done experiments with paging on single processor (before the days of hyperthreading0 machines with little penalty. The issue is one of cooperation between threads.
The reason I brought up DMA, is that, if the paging thread is using Programmed I/O to transfer data, then it will be, indeed, hogging up the CPU during file transfers. If you are using DMA, then file transfers should be a tiny hiccup in CPU usage. When vertical sync is enabled, you can be sure that there will be a period of time each frame that the rendering thread will be dormant. It is during this time that you want the pager to do its work. If the pager is "stuck" in programmed I/O, then it will not relinquish the CPU to the rendering thread until its operation is done.
The graph you posted is quite helpful. However, there is still a mystery as to why the database thread is so CPU bound during file I/O. It does seem to indicate some kind of spin lock that is stuck.
Now, if I were handling this problem under some kind of Unix, I would use prof and get a profile of the database thread to find out where it is spending most of its cycles. Do you have that kind of capability with your compilers/tools? If so, that is probably the best place to start at narrowing down where exactly CPU time is being spent.
-don
Zach
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Don BurnsSent: Sunday, August 20, 2006 16:44
To: osg users
Subject: Re: [osg-users] Paging performance on WindowsHi Zach,
Could you try configuring DMA on your system for file I/O and see if it makes a difference? I found this page regarding configuration for DMA:
http://support.microsoft.com/kb/247426/
-don
On 8/20/06, Zach Deedler <[EMAIL PROTECTED]> wrote:Sorry to keep this topic open for so long, but I really want to exhaust any possibility of their being a bug. I have another useful piece of information:I have loaded my paged database onto an unnamed vendor's image generator, IG, and used their software to load it. I do not see the same hitching issues I see with osg. The computer is a Windows XP computer. So I know a Windows computer can handle this database. However, I did notice that their was some real-time software installed on the IG, (Ardence). That software possibly makes a difference in the paging.Zach
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Don BurnsSent: Sunday, August 20, 2006 12:07To: osg users
Subject: Re: [osg-users] Paging performance on Windows
On 8/19/06, Zach Deedler <[EMAIL PROTECTED]> wrote:
QUESTION1) Is there a way to create ive files that don't have the textures embedded in them? (This would be a quick way for me to get file sizes down)
Hi Zach,
Simply hide the textures. That is, use
osgconv file.flt file.ive
but put the textures somewhere where osgDB can't find them.
-don
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Don Burns
Sent: Friday, August 18, 2006 16:57
To: osg usersSubject: Re: [osg-users] Paging performance on WindowsHi Zach,
I'm happy to hear your approach on this problem. The reason I ask why, is that my constant experience with OS's from microsoft is that they consistently perform at sub optimal levels. This is after 16 years of testing. My position is not "religious", it is simply an expectation I have come to accept based on technical experiences.
If you can accept that Windows will not load a file as fast as a UNIX based file system (which is my constant conclusion), but are constrained to continue to build on Windows, then you can start to tune your application, not with the goal of trying to match Linux performance, but with the goal of getting the best performance you can from a Windows system. This will probably mean that you will need to pay more attention to the size of tile blocks that are being paged at any given time. Consider that developers of games that appear to perform well on Windows have probably gone through this painful process already.
The other experience I have under Windows is that of poor thread management performance. UNIX has been an OS that was designed from the ground up to multi-task and has decades of refinements for schedulers. I tend to trust them more in these environments.
One of the behaviors I've come to recognize as a significant difference between Linux/UNIX and Windows based systems is that of I/O. Windows will literally freeze when heavy I/O is occuring, until it is done. This is not so (as much, at least) with Linux/UNIX systems. Even in a threaded environment, when you should have cooperation between separate threads, if I/O decides to take over the system, including the blocking of thread and process schedulers, then there is literally nothing you can do. You have to force the cooperation by limiting your I/O operations to minimize the time spent during data transfers.
I think that if you take a stance which does not accept the limitations of Windows as a real-time system you will limit your options. If you accept it then you can work around it.
-don
On 8/18/06, Zach Deedler <[EMAIL PROTECTED]> wrote:Hi Don,
Why am I unwilling to accept that it is a limitation in the OS?
1) The paging thread is virtually taking over the processor for periods of
multiple seconds.
2) I know games can page data on a Windows OS with out this problem.
Just from my investigations in graphing processor time for the threads. I
can see the paging thread virtually taking over the processor for multiple
seconds. From the graphs, I know that the main thread still gets some time
during those seconds. I know that if the paging thread was told to sleep
for awhile then the hitching problem would not look so bad. When no
hitching is occuring, the paging thread takes no more than 30% of the proc.
Somehow the paging thread is given almost full control of the processor for
a period of multiple seconds. Even if an atomic operation on Windows took 1
full second, I should be able to code around this to get better performance
(sleep on the thread that needs one second). Given the fact that I'm using
a dual-core processor should also help.
If reading one file is an atomic action in Windows, then I am willing to
accept this issue is an OS issue. Otherwise, someone will have to prove it
to me. I'm not saying that Linux won't outperform Windows with disk access,
and I am by no means promoting Windows over Linux. I'm just trying to get
better performance out of my paged database on a Windows system. I am
confident this can be done if I can narrow down the operations that are
taking soo long without the main thread noticing and accounting for it.
>From: "Don Burns" < [EMAIL PROTECTED]>
>Reply-To: osg users <[email protected]>
>To: "osg users" < [email protected]>
>Subject: Re: [osg-users] Paging performance on Windows
>Date: Thu, 17 Aug 2006 10:33:17 -0700
>
>On 8/16/06, Zach Deedler < [EMAIL PROTECTED]> wrote:
>>
>>
>>5) Given TRIAL4, seems like a #ifdef WIN32 problem, or limitation
>>in the
>>OS
>>(I'm still unwilling to accept this)
>
>
>
>why ?
>_______________________________________________
>osg-users mailing list
>[email protected]
> http://openscenegraph.net/mailman/listinfo/osg-users
>http://www.openscenegraph.org/
_______________________________________________
osg-users mailing list
[email protected]
http://openscenegraph.net/mailman/listinfo/osg-users
http://www.openscenegraph.org/
_______________________________________________
osg-users mailing list
[email protected]
http://openscenegraph.net/mailman/listinfo/osg-users
http://www.openscenegraph.org/
_______________________________________________
osg-users mailing list
[email protected]
http://openscenegraph.net/mailman/listinfo/osg-users
http://www.openscenegraph.org/
_______________________________________________
osg-users mailing list
[email protected]
http://openscenegraph.net/mailman/listinfo/osg-users
http://www.openscenegraph.org/
_______________________________________________ osg-users mailing list [email protected] http://openscenegraph.net/mailman/listinfo/osg-users http://www.openscenegraph.org/
