Re: Slow to upload thousands of tiny files over cpiod

2017-11-06 Thread miha . plesko . xlab
Yes, I'd kindly ask for a patch that introduces the `--verbose` for cpiod, whole Capstan would benefit from it. The problem when using --redirect option is that ALL the output will then be redirected, not only cpiod's - which is not always desired. We usually want to see at least some logs from

Re: Slow to upload thousands of tiny files over cpiod

2017-11-05 Thread Waldek Kozaczuk
Nadav, Thanks for the explanation. Meanwhile I think it would be worth to modify cpiod to support new '--verbose' option that would make it print files/directories/symlink information only when passed in. Unless you think it a bad idea I will try to provide a patch to address this. Also I no

Re: Slow to upload thousands of tiny files over cpiod

2017-11-05 Thread Nadav Har'El
On Sat, Nov 4, 2017 at 5:14 AM, Waldek Kozaczuk wrote: > Miha, > > It looks like you are right. When I disabled printing of files names in > cpiod it made some difference when using QEMU and huge difference when > using VirtualBox on OSX. With QEMU on average I was able to cut down the > time fro

Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread Waldek Kozaczuk
On Linux I see twofold improvement from 13 seconds to 5 seconds against QEMU and 5-fold improvement from 31 seconds to 5 seconds. QEMU seems to be way more faster on Linux than OSX (probably because of aio=threads because native not supported on OSX). On Fri, Nov 3, 2017 at 11:41 PM, Dor Laor wro

Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread Dor Laor
On Fri, Nov 3, 2017 at 8:14 PM, Waldek Kozaczuk wrote: > Miha, > > It looks like you are right. When I disabled printing of files names in > cpiod it made some difference when using QEMU and huge difference when > using VirtualBox on OSX. With QEMU on average I was able to cut down the > time fro

Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread Waldek Kozaczuk
Miha, It looks like you are right. When I disabled printing of files names in cpiod it made some difference when using QEMU and huge difference when using VirtualBox on OSX. With QEMU on average I was able to cut down the time from ~ 3 minutes to ~2 minutes and from ~10 minutes to ~ 10 SECONDS

Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread miha . plesko . xlab
Hi Waldek, I think the slowness comes because cpiod reports each file on the stdout. Could you try to comment-out the logging in cpiod and try if it works faster? So if you open two terminals and in the first one run: ``` $ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f 1000

Re: Slow to upload thousands of tiny files over cpiod

2017-11-02 Thread Waldek Kozaczuk
Adding couple of capstan contributors to the thread. Sent from my iPhone > On Nov 2, 2017, at 16:16, Waldek Kozaczuk wrote: > > I think this might be a well known and undestood issue by capstan team but I > wanted to uderstand why it happens for my own sake. > > In my scenario I am usually

Slow to upload thousands of tiny files over cpiod

2017-11-02 Thread Waldek Kozaczuk
I think this might be a well known and undestood issue by capstan team but I wanted to uderstand why it happens for my own sake. In my scenario I am usually able to upload 300 of files totaling 60MB in 10-12 seconds (Java app) using capstan which I believe employs same mechanism as OSv build s