Re: [Plplot-devel] The wxwidgets device driver speed/idle time

[Phil Rosenberg]

Hi Alan
It's on my list - I will do it as soon as I can :-). I actually think
I had a similar system like this set up - although I think I used
named semaphores. I would have to go back to the code to check.

Phil

On 19 November 2016 at 07:39, Alan W. Irwin  wrote:
> Hi Phil:
>
> I got tired of speculating about more efficient Linux IPC methods so I
> implemented (as of commit e166866) the "Unnamed semaphores example"
> paradigm outlined starting on page 73 of
> . (I
> discussed this possibility in earlier versions of this thread.)
> This test project is described in cmake/test_linux_ipc/README.  Follow
> the directions there to test it on your own Linux systems.
>
> My tests show this method is quite efficient for transferring data
> between processes on POSIX-compliant systems.  The transfer process is
> under complete control of unnamed semaphores that reside in the shared
> memory so coordinating required data transfer between two applications
> is completely straightforward, and there is very little idle time
> required during these transfers.
>
> The IPC-relevant routines that are called in this test project are
> shm_open, ftruncate, mmap, sem_init, sem_wait, sem_post, and
> shm_unlink, and according to the Linux man pages, all of these
> functions are provided by platforms that are compliant with POSIX.1-2001.
>
> So on the present evidence from this test project I think this method
> looks extremely promising for efficiently transferring data both ways
> between -dev wxwidgets and wxPLViewer for POSIX-compliant systems.
>
> If after trying the test of the method documented in
> cmake/test_linux_ipc/README you agree with this assessment, would you
> be willing to have a go at the required wxwidgets and wxPLViewer
> source code changes to see if that essentially eliminates the
> large idle times we have now on Linux?
>
>
> Alan
> __
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __
>
> Linux-powered Science
> __

--
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] The wxwidgets device driver speed/idle time

[Alan W. Irwin]

Hi Phil:

I got tired of speculating about more efficient Linux IPC methods so I
implemented (as of commit e166866) the "Unnamed semaphores example"
paradigm outlined starting on page 73 of
. (I
discussed this possibility in earlier versions of this thread.)
This test project is described in cmake/test_linux_ipc/README.  Follow
the directions there to test it on your own Linux systems.

My tests show this method is quite efficient for transferring data
between processes on POSIX-compliant systems.  The transfer process is
under complete control of unnamed semaphores that reside in the shared
memory so coordinating required data transfer between two applications
is completely straightforward, and there is very little idle time
required during these transfers.

The IPC-relevant routines that are called in this test project are
shm_open, ftruncate, mmap, sem_init, sem_wait, sem_post, and
shm_unlink, and according to the Linux man pages, all of these
functions are provided by platforms that are compliant with POSIX.1-2001.

So on the present evidence from this test project I think this method
looks extremely promising for efficiently transferring data both ways
between -dev wxwidgets and wxPLViewer for POSIX-compliant systems.

If after trying the test of the method documented in 
cmake/test_linux_ipc/README you agree with this assessment, would you
be willing to have a go at the required wxwidgets and wxPLViewer
source code changes to see if that essentially eliminates the
large idle times we have now on Linux?

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] The wxwidgets device driver speed/idle time

[Alan W. Irwin]

On 2016-11-15 22:52- p.d.rosenb...@gmail.com wrote:

> Hi Alan, Laurent
> I can't say I have much to add. It wasn't obvious to me where the lag was 
> last time i looked, but I agree it is an issue
>

> I haven't done much plplot work for some time, but when I can i will
look at this. If anyone has any suggestions I would be keen to hear.

Hi Phil:

>From what you say above it sounds like any serious time you can spend
on this would be pretty limited and not immediate.  Given those
conditions, I think that likely the most valuable thing you can do for
us immediately is to write up a paragraph or so describing how you are
currently using IPC to communicate between -dev wxwidgets and
wxPLViewer including a summary of where waits might be occurring. I
suggest you commit that overview in the new file
drivers/README.wxwidgets_IPC (or I will do that if you just want to
send your overview by e-mail). That IPC overview would be a big help
in getting up to speed for anyone else wanting to figure out what is
going on with these weird idle times that we are encountering on the
Linux platform.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] The wxwidgets device driver speed/idle time

[p.d.rosenberg]

Hi Alan, Laurent
I can't say I have much to add. It wasn't obvious to me where the lag was last 
time i looked, but I agree it is an issue

I haven't done much plplot work for some time, but when I can i will look at 
this. If anyone has any suggestions I would be keen to hear.

Phil 

Sent from my Windows 10 phone

From: Alan W. Irwin
Sent: 15 November 2016 21:22
To: Laurent Berger
Cc: PLplot development list
Subject: Re: [Plplot-devel] The wxwidgets device driver speed/idle time

On 2016-11-15 14:59+0100 Laurent Berger wrote:

> Hi Alan,
>
> I haven't got qt installed.
> My results looks like yours results :
>
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev wxwidgets -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m0.407s
> user0m0.108s
> sys0m0.016s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev wxwidgets -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m10.997s
> user0m0.064s
> sys0m0.040s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xwin -np
> PLplot library version: 5.11.1
>
> real0m0.122s
> user0m0.056s
> sys0m0.020s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xwin -np
> PLplot library version: 5.11.1
>
> real0m0.122s
> user0m0.060s
> sys0m0.016s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xcairo -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m0.125s
> user0m0.088s
> sys0m0.016s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xcairo -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m0.128s
> user0m0.084s
> sys0m0.020s
> lb@mp12-lb:~/plplot.git/examples/c++$

Hi Laurent:

Thanks very much for the above report.

Your timing results verify mine and show that -dev wxwidgets sometimes
has very poor "real" timing results even though the actual cpu times
indicated by the sum of "user" + "sys" are quite fast and consistent.

For this case

> real0m10.997s
> user0m0.064s
> sys0m0.040s

the sum of user + sys is 0.104 seconds so that means for this test the
cpu was idle for 10.893 seconds out of the 10.997 seconds or 99.05 per
cent of the time which is equivalent to two (!) orders of magnitude
slowdown.  Since your results and mine confirm the problem for a wide
range of Linux kernel versions, my working hypothesis is these
excessive idle times/slowdowns are due to the inefficient way that
Linux kernel IPC is being used to implement the necessary two-way
communication between -dev wxwidgets and the wxPLviewer application on
Linux.

Gaining two orders of magnitude in speed is certainly an attractive
goal for the wxwidgets device/wxPLviewer combination so I am tempted
to immediately start reviewing the Linux kernel IPC use between -dev
wxwidgets and wxPLviewer.  However, my current knowledge of Linux
kernel IPC is small, and I 

Re: [Plplot-devel] The wxwidgets device driver speed/idle time

[Alan W. Irwin]

On 2016-11-15 14:59+0100 Laurent Berger wrote:

> Hi Alan,
>
> I haven't got qt installed.
> My results looks like yours results :
>
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev wxwidgets -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m0.407s
> user0m0.108s
> sys0m0.016s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev wxwidgets -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m10.997s
> user0m0.064s
> sys0m0.040s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xwin -np
> PLplot library version: 5.11.1
>
> real0m0.122s
> user0m0.056s
> sys0m0.020s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xwin -np
> PLplot library version: 5.11.1
>
> real0m0.122s
> user0m0.060s
> sys0m0.016s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xcairo -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m0.125s
> user0m0.088s
> sys0m0.016s
> lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xcairo -np
> PLplot library version: 5.11.1
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 103: 
> Having multiple values in  isn't supported and may not work as expected
> Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 138: 
> Having multiple values in  isn't supported and may not work as expected
>
> real0m0.128s
> user0m0.084s
> sys0m0.020s
> lb@mp12-lb:~/plplot.git/examples/c++$

Hi Laurent:

Thanks very much for the above report.

Your timing results verify mine and show that -dev wxwidgets sometimes
has very poor "real" timing results even though the actual cpu times
indicated by the sum of "user" + "sys" are quite fast and consistent.

For this case

> real0m10.997s
> user0m0.064s
> sys0m0.040s

the sum of user + sys is 0.104 seconds so that means for this test the
cpu was idle for 10.893 seconds out of the 10.997 seconds or 99.05 per
cent of the time which is equivalent to two (!) orders of magnitude
slowdown.  Since your results and mine confirm the problem for a wide
range of Linux kernel versions, my working hypothesis is these
excessive idle times/slowdowns are due to the inefficient way that
Linux kernel IPC is being used to implement the necessary two-way
communication between -dev wxwidgets and the wxPLviewer application on
Linux.

Gaining two orders of magnitude in speed is certainly an attractive
goal for the wxwidgets device/wxPLviewer combination so I am tempted
to immediately start reviewing the Linux kernel IPC use between -dev
wxwidgets and wxPLviewer.  However, my current knowledge of Linux
kernel IPC is small, and I have many other PLplot issues on my plate
at the moment so I will have to delay this for quite a while.  Thus,
there is an opportunity for someone else to contribute substantially
here if they have some knowledge about the most efficient way to use
Linux kernel IPC or want to learn more about that interesting topic.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project 

Re: [Plplot-devel] The wxwidgets device driver speed/idle time

[Laurent Berger]

Hi Alan,

I haven't got qt installed.
My results looks like yours results :

lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev wxwidgets -np
PLplot library version: 5.11.1
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
103: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
138: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
103: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
138: Having multiple values in  isn't supported and may not work 
as expected

real0m0.407s
user0m0.108s
sys0m0.016s
lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev wxwidgets -np
PLplot library version: 5.11.1
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
103: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
138: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
103: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
138: Having multiple values in  isn't supported and may not work 
as expected

real0m10.997s
user0m0.064s
sys0m0.040s
lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xwin -np
PLplot library version: 5.11.1

real0m0.122s
user0m0.056s
sys0m0.020s
lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xwin -np
PLplot library version: 5.11.1

real0m0.122s
user0m0.060s
sys0m0.016s
lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xcairo -np
PLplot library version: 5.11.1
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
103: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
138: Having multiple values in  isn't supported and may not work 
as expected

real0m0.125s
user0m0.088s
sys0m0.016s
lb@mp12-lb:~/plplot.git/examples/c++$ time ./x01 -dev xcairo -np
PLplot library version: 5.11.1
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
103: Having multiple values in  isn't supported and may not work 
as expected
Fontconfig warning: "/etc/fonts/conf.d/65-droid-sans-fonts.conf", line 
138: Having multiple values in  isn't supported and may not work 
as expected

real0m0.128s
user0m0.084s
sys0m0.020s
lb@mp12-lb:~/plplot.git/examples/c++$






Le 14/11/2016 à 22:19, Alan W. Irwin a écrit :
> Hi Laurent:
>
> I have put my reply to you on list for obvious reasons.
>
> On 2016-11-14 14:26+0100 Laurent Berger wrote:
>
>> Hi Alan,
>>
>> I would try to help you
>
> Thanks in advance for that.
>
>> but I'm not familiar on linux (see attached file for version).
>
> From what you included, your Linux platform has access to wxwidgets
> version 3.0 (as do I).  But the rest of your platform (old Linux Mint
> distribution using a relatively old kernel version 3.2.0-2-amd64
> compared to my much more modern Debian Jessie using a fairly
> up-to-date kernel version 3.16.0-4-amd64) provides an extremely useful
> testing contrast in case the issue is some lameness in the IPC
> implementation for my particular Linux kernel.  (IPC, i.e.,
> interprocess communication is used to communicate between -dev
> wxwidgets and the wxPLViewer application that displays the results.)
>
>> I think it is this message 
>> (https://sourceforge.net/p/plplot/mailman/message/34744869/)
>
> Yes, that message and the following one that provided the screenshots 
> referred to
> in that message are the relevant ones.
>
>>
>> I have build plplot on linux using cmake 3.0.2 using "Unix Makefiles"
>> Then sudo make install. I have attached git log result and cmakecache
>> Then  I run in folder examples/c++
>> ./x01c -dev wxdrivers
>>
>> I can see a window with 4 curves. Windows title is wxPLviewer. No 
>> problem
>> Now what do you want to test ?
>
> Speed.
>
> You measure speed on linux using, e.g.,
>
> software@raven> time examples/c/x01c -dev wxwidgets -np
> PLplot library version: 5.11.1
>
> real0m0.652s
> user0m0.064s
> sys 0m0.016s
> software@raven> software@raven> time examples/c/x01c -dev wxwidgets -np
> PLplot library version: 5.11.1
>
> real0m9.327s
> user0m0.072s
> sys 0m0.008s
>
> That -np option stands for "no pause" and allows you to measure time
> without waiting for the human reaction of clicking on the GUI to
> finish the display.
>
> Those two measurements obviously have a huge difference in the real
> time used. The first timing is not too bad, but (by chance) the second
> timed run shows the