Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-11-02 Thread Waldek Kozaczuk
Hi, I have just applied this patch to master. It seems to be working based on number of workloads I have tested with new automated scripts I have just added as well. On Mon, Oct 7, 2019 at 22:26 Waldek Kozaczuk wrote: > I have just sent a proper patch addressing this issue. Feel free to apply >

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-07 Thread Waldek Kozaczuk
I have just sent a proper patch addressing this issue. Feel free to apply it and test it. On Sunday, October 6, 2019 at 2:30:11 PM UTC-4, Henrique Fingler wrote: > > Not yet, I'll probably do it today and post the results on this thread. > > On Sunday, October 6, 2019 at 1:16:01 PM UTC-5, Waldek

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-06 Thread Henrique Fingler
Not yet, I'll probably do it today and post the results on this thread. On Sunday, October 6, 2019 at 1:16:01 PM UTC-5, Waldek Kozaczuk wrote: > > Did you try my latest patch from Friday that also frees memory? > > On Sun, Oct 6, 2019 at 14:14 Henrique Fingler > wrote: > >> That makes sense. I

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-06 Thread Waldek Kozaczuk
Did you try my latest patch from Friday that also frees memory? On Sun, Oct 6, 2019 at 14:14 Henrique Fingler wrote: > That makes sense. I did try using the hugepage alloc that I had before > but it still crashed. Thank you for the response! > I'll try it out with *alloc_phys_contiguous_aligne

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-06 Thread Henrique Fingler
That makes sense. I did try using the hugepage alloc that I had before but it still crashed. Thank you for the response! I'll try it out with *alloc_phys_contiguous_aligned.* On Friday, October 4, 2019 at 2:28:01 PM UTC-5, Waldek Kozaczuk wrote: > > Hi, > > So there were a couple of issues with

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-06 Thread Nadav Har'El
On Fri, Oct 4, 2019 at 10:28 PM Waldek Kozaczuk wrote: > Hi, > > So there were a couple of issues with my and your patch: > 1) We should NOT be using straight *malloc*() to allocate 17 pages of > memory. It needs to be page-aligned, so *aligned_alloc()* is the correct > choice. > Even better, pl

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-05 Thread Waldek Kozaczuk
I have opened a new issue on OSv side - https://github.com/cloudius-systems/osv/issues/1058 and one on the firecracker side - https://github.com/firecracker-microvm/firecracker/issues/1314. On Friday, October 4, 2019 at 3:28:01 PM UTC-4, Waldek Kozaczuk wrote: > > Hi, > > So there were a couple

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-10-04 Thread Waldek Kozaczuk
Hi, So there were a couple of issues with my and your patch: 1) We should NOT be using straight *malloc*() to allocate 17 pages of memory. It needs to be page-aligned, so *aligned_alloc()* is the correct choice. 2) I missed recognizing that we also need to *free()* instead of *free_page()* in t

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-26 Thread Henrique Fingler
Waldek, I'm getting a general protection fault when doing some HTTP requests from OSv, do you think it might be related to the hack to make it work on Firecracker? Here's the MWE, a few requests go through, then it faults. import urllib.request for i in range (10): response = urllib.request.u

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread Waldek Kozaczuk
Yes quite substantial. On firecracker ZFS needs at least 50-60 ms to initialize on my machine. Whereas RoFS images takes 1 millisecond - the smallest native example takes 5-6 ms to boot including RoFS mount and ~10ms in total to execute (10 ms includes that 5-6 ms of boot time). Sent from my i

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread zhiting zhu
Is there any difference on boot time between zfs and rofs? On Fri, Sep 20, 2019 at 2:45 PM Henrique Fingler wrote: > I'll check that out. > > Instead of detecting what hypervisor we are dealing with, we should simply >> act accordingly based on what features have been negotiated and agreed > >

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread Henrique Fingler
I'll check that out. Instead of detecting what hypervisor we are dealing with, we should simply > act accordingly based on what features have been negotiated and agreed Yep, you're right. Five minutes after I hit Post I remembered what "negotiate" means. Whoops. Also, I have noticed with my

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread Waldek Kozaczuk
On Friday, September 20, 2019 at 8:56:35 AM UTC-4, Waldek Kozaczuk wrote: > > See my answers below. > > On Thursday, September 19, 2019 at 11:34:56 PM UTC-4, Henrique Fingler > wrote: >> >> I agree that this is mostly a thing that should be done on Firecracker. >> For now, if there's a way to

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread Waldek Kozaczuk
On Friday, September 20, 2019 at 9:04:38 AM UTC-4, Waldek Kozaczuk wrote: > > So instead of what we do here for "chained buffer" - > https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-net.cc#L593-L603 > - > if enabled we would do something like this (pseudo-code and mist likely

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread Waldek Kozaczuk
So instead of what we do here for "chained buffer" - https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-net.cc#L593-L603 - if enabled we would do something like this (pseudo-code and mist likely buggy): while (vq->avail_ring_not_empty()) { vq->init_sg(); for (int i

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-20 Thread Waldek Kozaczuk
See my answers below. On Thursday, September 19, 2019 at 11:34:56 PM UTC-4, Henrique Fingler wrote: > > I agree that this is mostly a thing that should be done on Firecracker. > For now, if there's a way to detect the hypervisor we can switch that. > Personally I'm only using Firecracker so I'

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-19 Thread Henrique Fingler
I agree that this is mostly a thing that should be done on Firecracker. For now, if there's a way to detect the hypervisor we can switch that. Personally I'm only using Firecracker so I'll leave this in. I wrote pretty much the same code but instead of *malloc *I used *memory::alloc_hugepage(

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-19 Thread Waldek Kozaczuk
This patch seems to do the job: diff --git a/drivers/virtio-net.cc b/drivers/virtio-net.cc index e78fb3af..fe5f1ae0 100644 --- a/drivers/virtio-net.cc +++ b/drivers/virtio-net.cc @@ -375,6 +375,8 @@ void net::read_config() net_i("Features: %s=%d,%s=%d", "Host TSO ECN", _host_tso_ecn, "CSUM",

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-19 Thread Henrique Fingler
I'm trying to check if it works on qemu, but scripts/run and capstan run set the network differently than Firecracker's script. With the regular user networking (no "-n") it works. When I try running it with with "-n -b br0" or just "-n" the execution hangs after printing OSv version. I'm tr

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-19 Thread Waldek Kozaczuk
Most likely it is a bug on OSv side. It could be in the virtio-net features negotiation logic - https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-net.cc#L351-L378 or https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-net.cc#L283-L297 . I also saw this comment

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-18 Thread Henrique Fingler
How do I go about disabling GSO? I think I found how to disable TSO (diff below), but I can't find where to disable GSO. Disabling just TSO didn't fix it. The loop where Firecracker gets stuck (fn rx_single_frame) tries to write an entire frame (7318 bytes) and it notices it doesn't fit into

Re: [osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-18 Thread Asias He
On Thu, Sep 19, 2019 at 7:06 AM Henrique Fingler wrote: > First of all, thank you for being active and helping out users! > > Here's my setup: I'm building a python3 image, with a script that does > > * response = urllib.request.urlopen("http://")* > > The execution just hangs for a few second

[osv-dev] Can't download "large" files using python3 on top of Firecracker: ...virtio/net.rs:257] Receiving buffer is too small to hold frame of current size

2019-09-18 Thread Henrique Fingler
First of all, thank you for being active and helping out users! Here's my setup: I'm building a python3 image, with a script that does * response = urllib.request.urlopen("http://")* The execution just hangs for a few seconds, then a storm of warnings from Firecracker show up: 2019-09-18T1