On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote: > > > Le 14/10/2014 14:15, David T. Lewis a ?crit : > >On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote: > >>Hi Dave, > >> > >>took me a while to do some testing. It seems there is an issue, but I > >>couldn't find out where. First, the data: > >> > >>Loading a package from a local gitfiletree repository (get from the > >>command a zipped git archive containing the package) : > >> > >>4.5.11, 4.6.8: 2132 msec. > >>4.5.12, 4.6.9: 50847 msec. > >> > >>I tried to rewrite the main command, remove the closePipes, etc... No > >>change in the runtime, so your right. > >> > >>Profiling doesn't help much: execution time is mostly idle time (but > >>with a significant increase on PipeableOSProcess commands: goes from > >>invisible to 2700 ms. A user break in the middle shows it delay waiting > >>on output of the external process). > >> > >>Only hint I could see is that it seems to be linked with the size of the > >>output of the command. > > > >Thierry, > > > >Aha! I am quite sure that you have located the problem. There must be > >something preventing the OS process from running to completion with > >a larger amount of output data. It definitely sounds like a bug in my > >latest update to Pipeable OSProcess. > > The strange thing is that it runs to completion (i.e. I got no errors; > if any data was missing or corrupted on output, my load would fail); but > a lot slower than it should be. > > >Thank you very much for finding this. I will get it fixed as quickly > >as I can. > > Thanks. I'm unable to help you much, I'm afraid :(
Thierry, I do not yet have a solution to this, but I have been able to confirm that the performance difference that you see is primarily associated with the AioPlugin not being present in your VM. When I changed PipeableOSProcess to handle pipe handle closing properly, I used a buffered output stream that is driven by aio events. When data is available to be read from the external OS process, an aio notification pulls the available data into the buffered stream, and when no more data is available, the pipe (file) handles are closed. If the plugin is not present in the VM, I provided a polling mechanism to read data from the external OS process. Apparently this is very inefficient, and I am sure it could be greatly improved. That is the cause of the extremely slow performance that you are seeing. So in summary: The enhancement to PipeableOSProcess relies on an aio event driven reader, and if the AIO plugin is missing, the fallback code relies on polling, and is horribly inefficient. More to follow, and thank you again for identifying the problem. If you can convice your VM vendor to include AioPlugin in your VM, that would be helpful also ;-) Dave