On Wed, Aug 7, 2019 at 8:30 PM Waldek Kozaczuk <jwkozac...@gmail.com> wrote:
> Hi, > > On Tuesday, August 6, 2019 at 12:45:09 PM UTC-4, Joe Duarte wrote: >> >> Hi all, >> >> I think there might be an opportunity for good speedups in OSv if modern >> compiler optimization options are used. >> > Before you start discussing "speedups in OSv", you need to think what kind of application you're trying to optimize, and how much of its slowness is actually due to the performance of OSv functions. Also, while better optimization never (or rarely) hurts, many times there are much bigger issues involved when we see performance problems: E.g., our algorithm may be taking locks too many times (e.g., at some point we noticed that sending a packet requires taking and releasing locks several dozens times, and worked to reduce that number), using too many atomic operations or memory barriers, the CPU is kept idle too much because of less-than-ideal load balancing, and many other issues which often hurt much more than the compiler optimization. But that being said it, it never hurts to try. -flto (gcc's Link Time Optimization, or what Visual Studio calls Whole >> Program Optimization) is a big booster in my experience, but I don't know >> if OSv uses dynamic linking at runtime, >> > OSv uses dynamic linking when running the user application, but OSv itself, the kernel, is statically linked, and could benefit internally from inter-procedural optimization. > or what happens with specific applications that are built with Capstan. >> -flto doesn't appear anywhere in the OSv repo. >> > > I have very little experience with '-flto'. I may have even tried it on > one of my branches. But I think the biggest bang for the buck (speed up due > to compiler/linker optimizations of the code, smaller dynamic linker lookup > symbol table so faster lookups, memory utilization, etc) we would get from > hiding/eliminating most symbols that right now get exported whether we need > it or not - please see this issue and my latest comments - > https://github.com/cloudius-systems/osv/issues/97#issuecomment-513809341. > I think its a worthwhile direction to work on, and will result in a smaller kernel and other benfits - but it will not result in any speed improvements. Symbol lookup is O(1), so while having many symbols may slow boot a bit, it will not slow down symbol lookup. And even if it did, applications only look up each symbol once - not every time they call a function, so the lookup time has negligable cost for applications. > > Causally I counted ~* 30 thousands symbols* exported in the release > version of loader.elf after stripping. In reality OSv needs to export only > probably no more than 2K of symbols given it implements only subset of > Linux API (one can easily count exported symbols in Linux version of the > libraries that OSv implicitly supplies (see supplied_modules in > core/elf.cc) assuming we hide boost and C++ standard library as well). > Right now many of the libraries are also linked as 'whole-archive' (see > this part of the Makefile - > https://github.com/cloudius-systems/osv/blob/master/Makefile#L1885-L1894) > which makes the kernel quite big (around 6.7Mib) uncompressed (it used to > be almost 10MiB which I lowered by removing on of the libraries from that > whole-archive list - see > https://github.com/cloudius-systems/osv/commit/ea69b89d83d980ec4bb5f40f3dc5eefff8978517). > I believe once we hide most of the symbols none of these libraries will > have to be linked as whole-archive. > > We have another issue bout hiding boost libraries which is kind of prelude > to issue #97 - https://github.com/cloudius-systems/osv/issues/1040 - > where I have dumped a lot of my findings. > > So all in all flto would be a 'plum on a cake' if I am paraphrasing it > correctly :-) > >> >> -O2 flags are rare in the repo, not used for most of OSv as far as I can >> tell. I see a lot of -Og, which is a debug flag and will produce slow code. >> > This is not true, we do compile with -O2. We have a "mode" parameter during build - "make mode=release" is the default, and as you can see the code does "include conf/$(mode).mk", i.e., includes conf/release.mk - which adds -O2. >> I would try a bunch of flags myself, but the root makefile is epic, over >> 2,000 lines, >> > The Makefile is big, but most of these lines are just lists of source files, and the Makefile is heavily commented to explain what every trick does, so should be very readable. > and I don't know where to edit it. It might be worth trying flags like >> -O2, -flto, and a CPU floor like -march=westmere. That's the CPU generation >> before Sandy Bridge so it's pretty conservative for any cloud contexts. >> Westmere would include the AES-NI instructions, carryless multiplication, >> and all the SSE 4.2 and previous instructions. Vanilla Linux runs faster >> with a modern CPU floor, and there's a proposal to set Haswell as the floor >> for Fedora 31, which is too aggressive and will probably be rejected, but >> Westmere is a big bump from generic x86_64 and conservative. >> >> Regarding the size of Makefile some of the complexity it the makefile is > due to the "host vs external" thing which we are close to fixing - see > https://github.com/cloudius-systems/osv/issues/743. Once we eliminate > using external the Makefile should be simpler and smaller. There is also > some ARM related stuff. Maybe one way to make this Makefile more manageable > would be to use include and move some stuff to smaller *.gmk files > I think the opposite - the Makefile is now big, but readable precisely because it's just one big linear file. If it became several dozen small files including each other, newbies won't know where to find anything. This is exactly what happend to Joe with the single parameter (-O2) we include from another makefile :-) > (especially these lists of objects - > https://github.com/cloudius-systems/osv/blob/master/Makefile#L798-L1809). > Sure, these can be moved out, but can also easily be skipped when reading the Makefile. > I think that paper that Waldek posted might have have confound in that >> their Ubuntu 1804 install was likely built with better compiler >> optimizations than the Xen+OSv install. >> >> Cheers, >> >> Joe >> > If you want to help us with any of these, it would be very much > appreciated. We are definitely looking for volunteers. > Indeed! Good luck, and tell us if you have any more questions about the Makefile or anything else. Cheers, > Waldek > > -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to osv-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/osv-dev/90ac0f25-4f09-491c-aa4d-b1517254e883%40googlegroups.com > <https://groups.google.com/d/msgid/osv-dev/90ac0f25-4f09-491c-aa4d-b1517254e883%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CANEVyjv-_o27op0wzXPd1NOfEaf93Q-fZP2RpdqjE-4sAQiLrA%40mail.gmail.com.
