Re: Memory limit?
> Out of memory: could not reclaim any further. Current memory: 5122256 Kb > > This suggests there was 5GB free while the allocation failed. > This *can* be a fragmentation issue (e.g., you asked for a 1 GB allocation, > but we couldn't free a 1GB consecutive area), but can also be a malloc() of a > ridiculous amount. Since commit 7ea953ca7d6533c025e535be49ee5bd2567fc8f3 a > malloc() of over the amount of memory we have prints a different error > message, but perhaps you still have some very large (but less than 10GB) > single allocation? > > The sad thing is that since we fail in the memory reclaimer, not in the > malloc(), you know which malloc() failed. This is > https://github.com/cloudius-systems/osv/issues/585. > One ad-hoc thing you can try is to connect with gdb, and see which OSv thread > is waiting in malloc - and see what malloc() it is trying to do. In an attempt to work around this, I've been trying to get the BEAM vm to pre-allocate memory, which it does via mmap. However, this didn't help initially as the memory wasn't being populated. I altered the mmap calls to include MAP_POPULATE to get them filled at startup, and now I get this crash. The debug output is from the erlang runtime system's os_mmap function. It seems to turn from the first call to mmap for a 2GB chunk, but asserts shortly after that (and the following is all I get): Attempting to mmap 2147483648 bytes to 0 mmaped 2147483648 bytes to address 2040 Assertion failed: !large() (arch/x64/arch-mmu.hh: next_pt_addr: 82) [backtrace] 0x002281da <__assert_fail+26> 0x00331a35 0x0033da0c , 1>::operator()(mmu::hw_ptep<1>, unsigned long)+76> 0x0033dc4a , 2>::operator()(mmu::hw_ptep<2>, unsigned long)+314> 0x0033debc , 3>::operator()(mmu::hw_ptep<3>, unsigned long)+284> 0x0033e11d >(unsigned long, unsigned long, unsigned long, mmu::populate<(mmu::account_opt)1>&, unsigned long)+413> 0x0033ee0f (mmu::vma*, void*, unsigned long, bool)+1231> 0x00337521 0x00459345 Any clues? Cheers, Rick -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Slow to upload thousands of tiny files over cpiod
Yes, I'd kindly ask for a patch that introduces the `--verbose` for cpiod, whole Capstan would benefit from it. The problem when using --redirect option is that ALL the output will then be redirected, not only cpiod's - which is not always desired. We usually want to see at least some logs from application to determine if everything works as expected. Having --redirect employed, you can't see a thing. And if OSv crashes for some reason, you're then not able to access the log file via http server because the OSv is not running anymore... So fixing the issue only for cpiod at the moment would be fantastic! Dne ponedeljek, 06. november 2017 03.59.35 UTC+1 je oseba Waldek Kozaczuk napisala: > > Nadav, > > Thanks for the explanation. > > Meanwhile I think it would be worth to modify cpiod to support new > '--verbose' option that would make it print files/directories/symlink > information only when passed in. Unless you think it a bad idea I will try > to provide a patch to address this. > > Also I noticed OSv supports '--redirect' option that allows redirection of > STDOUT and STDERR to some arbitrary file. This would also help with the > slowness issue. One more unrelated question about redirect option - would > it also print to that file in case of OSv crashing (registers, stack trace, > etc)? > > Waldek > > On Sunday, November 5, 2017 at 10:22:08 AM UTC-5, Nadav Har'El wrote: >> >> On Sat, Nov 4, 2017 at 5:14 AM, Waldek Kozaczuk >> wrote: >> >>> Miha, >>> >>> It looks like you are right. When I disabled printing of files names in >>> cpiod it made some difference when using QEMU and huge difference when >>> using VirtualBox on OSX. With QEMU on average I was able to cut down the >>> time from ~ 3 minutes to ~2 minutes and from ~10 minutes to ~ 10 SECONDS >>> with VirtualBox when uploading 13,000 tiny files. Please note i have not >>> run any tests on Linux yet which I am planning to do next. >>> >>> Does this mean that in general printing to standard out (over serial >>> console?) is slow and should be avoided? >>> >> >> Yes, it is very slow. As Dor explained, it needs to do a lot of work on >> every individual character. >> >> It is completely synchronous: for every character it needs to write to an >> IO register (incurring the usual hypervisor exit cost), but then, things >> deteriorate: Before we output the next character we need to loops and check >> when the hypervisor consumed the previous character. If this doesn't happen >> immediately and for some reason the guest and hypervisor are on the same >> CPU, we may need to wait until a context switch which can take ages. >> >> I don't know why VirtualBox is particularly slow in this. Maybe it >> answers the serial port write less immediately or maybe threads are pinned >> to CPUs differently. >> >> Maybe isa_serial_console::putchar() can be done differently to make it >> more virtualbox friendly, worth checking. >> >> Two things thing that are definitely worth checking are: >> >> 1. Currently we assume that the UART has a buffer of only *one* >> character. Could it be possible that VirtualBox and even QEMU actually >> support larger UART buffers, which can be used to significantly reduce the >> silly exits and busy loops? >> For example Linux's tty/serial/8250/8250_core.c suggests that 16550A may >> have a 16-byte output FIFO. Some sources I read suggest that qemu also has >> a 16-byte TX FIFO. >> >> 2. Currently we busy-wait until we can output another character. It might >> make sense to use TX interrupts instead, so that we can immediately let the >> host threads run (and handle the characters in the fifo). >> >> I'll create an issue with these ideas, in case someone wants to work on >> it. >> >> >> >>> And it is extra slow on some hypervisors/emulators like VirtualBox? Does >>> this have anything with this recent patch - >>> https://github.com/cloudius-systems/osv/commit/a543b9e6dd8776c43e070ec1b677f2fe1dcbb72e >>> ? >>> >> >> No, this patch was about cases where there was no serial port at all, and >> input would hang. I don't think it's related to your case. >> >> >> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Remotely set cmdline
Hi Waldek,
so what you'd like to achieve is to run cpiod only on first run, then run
the actual bootcmd on all subsequent runs.
I find such behaviour useful and would be very happy if we could do it in a
way that would work for cloud-init
as well.
I was thinking, could we perhaps modify cpiod so that it would exit
immediately if there was some specific file present on
the unikernel's filesystem (e.g. /cpiod-skip)? In other words, the command
would then look like this:
- '/tools/cpiod.so --prefix / --conditional; runscript/the_config'
When the "--conditional" flag is used and "/cpiod-skip" is present, then
the first part ("'/tools/cpiod.so --prefix / --conditional") would return
immediately. And the bootcmd would effectively become only
"runscript/the_config".
All we need to do then is to make sure the file "/cpiod-skip" is uploaded
during the "capstan package compose-remote".
Such approach would also work with cloud-init even if all commands there
are always run.
What do you think?
Dne ponedeljek, 06. november 2017 06.23.30 UTC+1 je oseba Waldek Kozaczuk
napisala:
>
> In last couple of weeks I have been working with MikelAngelo capstan team
> to add a feature 'compose-remote' that allows composing and uploading files
> to a remote OSv instance with cpiod running.
>
> This feature is aimed to help speed-up creation of OSv AWS immutable amis
> which right now is extremely slow if one wants to import pre-baked OSv raw
> image as a snapshot and create an AMI of it. The new idea is that every
> time user wanted to create final application AMI he/she would create a
> bootstrap or 'base" AMI (cmdline=''/tools/cpiod.so --prefix /") in a
> traditional way, start EC2 instance with running cpiod, upload extra files
> using capstan and take a snapshot and create new AMI which I tested and is
> much faster. This could be applied to all other cases where uploading to a
> remote instance makes more sense.
>
> The only remaining problem how one set new cmdline. There is a least one
> way of achieving it. One would bake in cloud-init module in a base AMI and
> have it set cmdline using httpserver and proper command in user data ec2
> start. There are many downsides - it would not work in non cloud-init
> friendly environments, cloud init is pretty heavy and complicated tool to
> achieve something so simple and depends on httpserver. Please it does not
> let me create immutable AMIs with preset cmdline.
>
> Here is my idea: create simple new tool - setcmdline.so that would behave
> like so:
>
>- set new cmdline if it is passed non-empty explicit cmdline
>- set new cmdline from a content of file /cmdline if one present and
>delete it
>- do not change cmdline otherwise
>
> Now user would use it in one of 2 ways - set cmdline of base AMI like so:
>
>- '/tools/cpiod.so --prefix /; /tools/setcmdline.so' and have capstan
>upload a file /cmdline with appropriate content when uploading all other
>files
>- '*/tools/cpiod.so --prefix /; /tools/setcmdline.so "**runscript
>/run/the_config**"'*
>
> Another alternative I toyed with would be to enhance cpiod to handle new
> 'setcmdline' command but it seems pretty hackish and also requires
> modification of capstan.
>
> What do you think?
> Waldek
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
