With the problem size bigger I see OSv consistently beating Linux host (at
least on my laptop, Ubuntu 19.10).
Linux:
./radix -p 1 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
1 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 620025 127815 492206
TIMING INFORMATION
Start time : 1582659397056280
Initialization finish time : 1582659400086786
Overall finish time : 1582659400706811
Total time with initialization : 3650531
Total time without initialization : 620025
./radix -p 2 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
2 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 333298 74005 258944
TIMING INFORMATION
Start time : 1582659471193401
Initialization finish time : 1582659472761435
Overall finish time : 1582659473094733
Total time with initialization : 1901332
Total time without initialization : 333298
./radix -p 4 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
4 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 176586 38013 137823
TIMING INFORMATION
Start time : 1582659586192838
Initialization finish time : 1582659586997985
Overall finish time : 1582659587174571
Total time with initialization : 981733
Total time without initialization : 176586
OSv:
./scripts/run.py -p qemu_microvm --qemu-path
/home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
--nics 0 --nogdb -m 1G -c 1 --block-device-cache writeback,aio=threads -e
'/radix -p 1 -r4096 -n16777216'
OSv v0.54.0-119-g4ee4b788
Booted up in 4.15 ms
Cmdline: /radix -p 1 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
1 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 535304 130142 405145
TIMING INFORMATION
Start time : 1582659265555955
Initialization finish time : 1582659268568276
Overall finish time : 1582659269103580
Total time with initialization : 3547625
Total time without initialization : 535304
./scripts/run.py -p qemu_microvm --qemu-path
/home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
--nics 0 --nogdb -m 1G -c 2 --block-device-cache writeback,aio=threads -e
'/radix -p 2 -r4096 -n16777216'
OSv v0.54.0-119-g4ee4b788
Booted up in 5.39 ms
Cmdline: /radix -p 2 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
2 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 293180 78253 211421
TIMING INFORMATION
Start time : 1582659500041834
Initialization finish time : 1582659501640537
Overall finish time : 1582659501933717
Total time with initialization : 1891883
Total time without initialization : 293180
./scripts/run.py -p qemu_microvm --qemu-path
/home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
--nics 0 --nogdb -m 1G -c 4 --block-device-cache writeback,aio=threads -e
'/radix -p 4 -r4096 -n16777216'
OSv v0.54.0-119-g4ee4b788
Booted up in 5.03 ms
Cmdline: /radix -p 4 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
4 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 163844 47966 114362
TIMING INFORMATION
Start time : 1582659574048566
Initialization finish time : 1582659575031430
Overall finish time : 1582659575195274
Total time with initialization : 1146708
Total time without initialization : 163844
So maybe (at least in this case) OSv scales pretty well with the number of
CPUs:
Most likely because in this use case is very computation-heavy and OSv does
not make many exits to the host (I wonder if OSv tracing mechanism has a
way to show count of all exists).
On Tuesday, February 25, 2020 at 2:12:10 PM UTC-5, twee...@comcast.net
wrote:
> Wow, awesome results! I will try to reproduce shortly from here! If you
> want to increase the problem size, add -n 16777216 (which is 64x bigger
> than the current problem size of 262144, something will take take a couple
> of seconds to run).
>
> I am taking note of your commands to run the build and will be doing the
> same on my machine to see what I produce. I will post screenshots :)
>
>
> On Tuesday, February 25, 2020 at 1:09:08 PM UTC-5, Waldek Kozaczuk wrote:
>>
>> So I did try to build and run the radix test (please note my Ubuntu
>> laptop has only 4 cores and hyper-threading disabled). BTW it seems that
>> particular benchmark does not need read-write FS so I used ROFS):
>>
>> ./scripts/manifest_from_host.sh -w ../splash2-posix/kernels/radix/radix
>> && ./scripts/*build* fs=rofs --append-manifest -j4
>>
>> Linux host 1 cpu:
>>
>> ./radix -p 1 -r4096
>>
>>
>> Integer Radix Sort
>>
>> 262144 Keys
>>
>> 1 Processors
>>
>> Radix = 4096
>>
>> Max key = 524288
>>
>>
>>
>> PROCESS STATISTICS
>>
>> Total Rank Sort
>>
>> Proc Time Time Time
>>
>> 0 7335 2568 4765
>>
>>
>> TIMING INFORMATION
>>
>> Start time : 1582652832386234
>>
>> Initialization finish time : 1582652832444092
>>
>> Overall finish time : 1582652832451427
>>
>> Total time with initialization : 65193
>>
>> Total time without initialization : 7335
>>
>>
>> Linux host 2 cpus:
>> ./radix -p 2 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 2 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 4325 1571 2704
>>
>> TIMING INFORMATION
>> Start time : 1582652821496771
>> Initialization finish time : 1582652821531279
>> Overall finish time : 1582652821535604
>> Total time with initialization : 38833
>> Total time without initialization : 4325
>>
>> host 4 cpus:
>> ./radix -p 4 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 4 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 2599 1077 1470
>>
>> TIMING INFORMATION
>> Start time : 1582653906150199
>> Initialization finish time : 1582653906171932
>> Overall finish time : 1582653906174531
>> Total time with initialization : 24332
>> Total time without initialization : 2599
>>
>>
>> OSv 1 CPU
>> ./scripts/run.py -p qemu_microvm --qemu-path
>> /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
>>
>> --nics 0 --nogdb -m 64M -c 1 --block-device-cache writeback,aio=threads -e
>> '/radix -p 1 -r4096'
>> OSv v0.54.0-119-g4ee4b788
>> Booted up in 3.75 ms
>> Cmdline: /radix -p 1 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 1 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 6060 2002 4049
>>
>> TIMING INFORMATION
>> Start time : 1582652845450708
>> Initialization finish time : 1582652845500348
>> Overall finish time : 1582652845506408
>> Total time with initialization : 55700
>> Total time without initialization : 6060
>>
>> OSv 2 CPUs:
>> ./scripts/run.py -p qemu_microvm --qemu-path
>> /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
>>
>> --nics 0 --nogdb -m 64M -c 2 --block-device-cache writeback,aio=threads -e
>> '/radix -p 2 -r4096'
>> OSv v0.54.0-119-g4ee4b788
>> Booted up in 4.81 ms
>> Cmdline: /radix -p 2 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 2 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 5797 1702 4089
>>
>> TIMING INFORMATION
>> Start time : 1582653305076852
>> Initialization finish time : 1582653305129792
>> Overall finish time : 1582653305135589
>> Total time with initialization : 58737
>> Total time without initialization : 5797
>>
>> OSv 4 cpus
>> ./scripts/run.py -p qemu_microvm --qemu-path
>> /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
>>
>> --nics 0 --nogdb -m 64M -c 4 --block-device-cache writeback,aio=threads -e
>> '/radix -p 4 -r4096'
>> OSv v0.54.0-119-g4ee4b788
>> Booted up in 5.26 ms
>> Cmdline: /radix -p 4 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 4 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 6498 2393 4099
>>
>> TIMING INFORMATION
>> Start time : 1582653946823458
>> Initialization finish time : 1582653946875522
>> Overall finish time : 1582653946882020
>> Total time with initialization : 58562
>> Total time without initialization : 6498
>>
>>
>> As you can see with single CPU the benchmark seems to be 10-15 % faster.
>> But with two and four CPUs OSv barely sees any improvements, whereas on
>> host the app runs 40% faster. So OSv does not seem to scale at all
>> (somebody mentioned it used to) so it would be nice to understand why. OSv
>> has many sophisticated tracing tools that can help here -
>> https://github.com/cloudius-systems/osv/wiki/Trace-analysis-using-trace.py
>>
>> Waldek
>>
>> BTW1. I tried to bump size of the matrix to something higher but with
>> -r8192 the app crashes on both Linux and OSv.
>> BTW2. It would be interestingly to compare OSv with Linux guest (vs host).
>>
>> On Tuesday, February 25, 2020 at 10:05:08 AM UTC-5, twee...@comcast.net
>> wrote:
>>>
>>> Thanks for the response! I will get this information to you after work
>>> with the few modifications you recommended! The application is essentially
>>> just testing CPU performance using multiprocessing. Nothing too fancy about
>>> it! The code I am using can be found at:
>>>
>>> https://www.github.com/ProfessorWest/splash2-posix
>>>
>>> In side of the kernels folder located at radix.c and I change the
>>> problem size to 16,777,206.
>>>
>>> If you happen to examine the code, do ignore the lacking cleanness of
>>> the code...we just smashed everything into one file for simplicity on our
>>> end. (Running the same code across all platforms being benchmarked).
>>>
>>> On Tuesday, February 25, 2020 at 8:52:48 AM UTC-5, Waldek Kozaczuk wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am quite late to the party :-) Could you run OSv on single CPU with
>>>> verbose on (add -V to run.py) and send us the output so we can see a
>>>> little
>>>> more what is happening. To disable networking you need to add '--nics=0'
>>>> (for all 50 options run.py supports run it with '--help'). I am not
>>>> familiar with that benchmark but I wonder if it needs read-write FS (ZFS
>>>> in
>>>> OSv case), if not that you can build OSv images with read-only FS
>>>> (./scripts/build fs=rofs). Lastly, you can improve boot time by running
>>>> OSv
>>>> on firecracker (
>>>> https://github.com/cloudius-systems/osv/wiki/Running-OSv-on-Firecracker)
>>>> or on QEMU microvm (-p qemu_imcrovm - requires QEMU >= 4.1), with
>>>> read-only
>>>> FS on both OSv should boot within 5ms, ZFS within 40ms). Last thing -
>>>> writing to console on OSv can be quite slow, I wonder how much this
>>>> benchmark does it.
>>>>
>>>> While I definitely agree with my colleague Nadav, where he essentially
>>>> says do not use OSv if the raw performance matters (database for example)
>>>> and Linux will beat it no matter what, OSv may have advantages in use
>>>> cases
>>>> where pure performance does not matter (it still needs to be reasonable).
>>>> I
>>>> think the best use cases for OSv are serverless or stateless apps
>>>> (microservices or web assembly) running on single CPU where all state
>>>> management is delegated to a remote persistent store (most custom-built
>>>> business apps are like that) and where high isolation matters.
>>>>
>>>> Relatedly, I think it might be more useful to think of OSv (and other
>>>> unikernels) as highly isolated processes. To that end, we still need to
>>>> optimize memory overhead (stacks for example) and improve virtio-fs
>>>> support
>>>> (in this case to run the app on OSv you do not need full image, just
>>>> kernel
>>>> to run a Linux app).
>>>>
>>>> Also, I think the lack of good tooling in unikernel space affects their
>>>> adoption. Compare it with docker - build, push, pull, run. OSv has its
>>>> equivalent - capstan - but at this point, we do not really have a registry
>>>> where one can pull the latest OSv kernel or push, pull images. Trying to
>>>> run an app on OSv is still quite painful to a business app developer - it
>>>> probably takes at least 30 minutes or so.
>>>>
>>>> Lastly, I think one of the main reasons for Docker adoption, was
>>>> repeatability (besides its fantastic ease of use) where one can create an
>>>> image and expect it to run almost the same way in production. Imagine you
>>>> can achieve that with OSv.
>>>>
>>>> Waldek
>>>>
>>>> On Tuesday, February 25, 2020 at 7:00:16 AM UTC-5, twee...@comcast.net
>>>> wrote:
>>>>>
>>>>> Very well explained. Thank you for that. That does make perfect sense
>>>>> as well.
>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/4f08a5b2-59ef-453c-aa3c-0f5bcebcfe28%40googlegroups.com.
