Re: reduce networking latency
On Wed, Oct 22, 2014 at 04:35:07PM -0400, David Xu wrote: > 2014-10-15 14:30 GMT-04:00 David Xu : > > 2014-09-29 5:04 GMT-04:00 Michael S. Tsirkin : > >> On Wed, Sep 24, 2014 at 02:40:53PM -0400, David Xu wrote: > >>> Hi Michael, > >>> > >>> I found this interesting project from KVM TODO website: > >>> > >>> allow handling short packets from softirq or VCPU context > >>> Plan: > >>>We are going through the scheduler 3 times > >>>(could be up to 5 if softirqd is involved) > >>>Consider RX: host irq -> io thread -> VCPU thread -> > >>>guest irq -> guest thread. > >>>This adds a lot of latency. > >>>We can cut it by some 1.5x if we do a bit of work > >>>either in the VCPU or softirq context. > >>> Testing: netperf TCP RR - should be improved drastically > >>> netperf TCP STREAM guest to host - no regression > >>> > >>> Would you mind saying more about the work either in the vCPU or > >>> softirq context? > >> > >> For TX, we might be able to execute it directly from VCPU context. > >> For RX, from softirq context. > >> > > Do you mean for RX, we directly put data to a shared buffer accessed > by guest VM bypassing the IO thread? For TX, in vCPU context network > data is added to the shared buffer and kick host IRQ to send them? Yes, that's the idea. > >>> Why it is only for short packets handling? > >> > >> That's just one idea to avoid doing too much work like this. > >> > >> Doing too much work in VCPU context would break pipelining, > >> likely degrading stream performance. > >> Work in softirq context is not accounted against the correct > >> cgroups, doing a lot of work there will mean guest can steal > >> CPU from other guests. > >> > >>> Thanks a > >>> lot! > >>> > >>> > >>> Regards, > >>> > >>> Cong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: reduce networking latency
2014-10-15 14:30 GMT-04:00 David Xu : > 2014-09-29 5:04 GMT-04:00 Michael S. Tsirkin : >> On Wed, Sep 24, 2014 at 02:40:53PM -0400, David Xu wrote: >>> Hi Michael, >>> >>> I found this interesting project from KVM TODO website: >>> >>> allow handling short packets from softirq or VCPU context >>> Plan: >>>We are going through the scheduler 3 times >>>(could be up to 5 if softirqd is involved) >>>Consider RX: host irq -> io thread -> VCPU thread -> >>>guest irq -> guest thread. >>>This adds a lot of latency. >>>We can cut it by some 1.5x if we do a bit of work >>>either in the VCPU or softirq context. >>> Testing: netperf TCP RR - should be improved drastically >>> netperf TCP STREAM guest to host - no regression >>> >>> Would you mind saying more about the work either in the vCPU or >>> softirq context? >> >> For TX, we might be able to execute it directly from VCPU context. >> For RX, from softirq context. >> Do you mean for RX, we directly put data to a shared buffer accessed by guest VM bypassing the IO thread? For TX, in vCPU context network data is added to the shared buffer and kick host IRQ to send them? >>> Why it is only for short packets handling? >> >> That's just one idea to avoid doing too much work like this. >> >> Doing too much work in VCPU context would break pipelining, >> likely degrading stream performance. >> Work in softirq context is not accounted against the correct >> cgroups, doing a lot of work there will mean guest can steal >> CPU from other guests. >> >>> Thanks a >>> lot! >>> >>> >>> Regards, >>> >>> Cong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: reduce networking latency
2014-09-29 5:04 GMT-04:00 Michael S. Tsirkin : > On Wed, Sep 24, 2014 at 02:40:53PM -0400, David Xu wrote: >> Hi Michael, >> >> I found this interesting project from KVM TODO website: >> >> allow handling short packets from softirq or VCPU context >> Plan: >>We are going through the scheduler 3 times >>(could be up to 5 if softirqd is involved) >>Consider RX: host irq -> io thread -> VCPU thread -> >>guest irq -> guest thread. >>This adds a lot of latency. >>We can cut it by some 1.5x if we do a bit of work >>either in the VCPU or softirq context. >> Testing: netperf TCP RR - should be improved drastically >> netperf TCP STREAM guest to host - no regression >> >> Would you mind saying more about the work either in the vCPU or >> softirq context? > > For TX, we might be able to execute it directly from VCPU context. > For RX, from softirq context. > Which step is removed for TX and RX compared with vanilla? Or what's the new path? TX: guest thread-> host irq? TX: host irq-> ? Thanks. >> Why it is only for short packets handling? > > That's just one idea to avoid doing too much work like this. > > Doing too much work in VCPU context would break pipelining, > likely degrading stream performance. > Work in softirq context is not accounted against the correct > cgroups, doing a lot of work there will mean guest can steal > CPU from other guests. > >> Thanks a >> lot! >> >> >> Regards, >> >> Cong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: reduce networking latency
On Wed, Sep 24, 2014 at 02:40:53PM -0400, David Xu wrote: > Hi Michael, > > I found this interesting project from KVM TODO website: > > allow handling short packets from softirq or VCPU context > Plan: >We are going through the scheduler 3 times >(could be up to 5 if softirqd is involved) >Consider RX: host irq -> io thread -> VCPU thread -> >guest irq -> guest thread. >This adds a lot of latency. >We can cut it by some 1.5x if we do a bit of work >either in the VCPU or softirq context. > Testing: netperf TCP RR - should be improved drastically > netperf TCP STREAM guest to host - no regression > > Would you mind saying more about the work either in the vCPU or > softirq context? For TX, we might be able to execute it directly from VCPU context. For RX, from softirq context. > Why it is only for short packets handling? That's just one idea to avoid doing too much work like this. Doing too much work in VCPU context would break pipelining, likely degrading stream performance. Work in softirq context is not accounted against the correct cgroups, doing a lot of work there will mean guest can steal CPU from other guests. > Thanks a > lot! > > > Regards, > > Cong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
reduce networking latency
Hi Michael, I found this interesting project from KVM TODO website: allow handling short packets from softirq or VCPU context Plan: We are going through the scheduler 3 times (could be up to 5 if softirqd is involved) Consider RX: host irq -> io thread -> VCPU thread -> guest irq -> guest thread. This adds a lot of latency. We can cut it by some 1.5x if we do a bit of work either in the VCPU or softirq context. Testing: netperf TCP RR - should be improved drastically netperf TCP STREAM guest to host - no regression Would you mind saying more about the work either in the vCPU or softirq context? Why it is only for short packets handling? Thanks a lot! Regards, Cong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
reduce networking latency
Hi All, I found this interesting project from KVM TODO website: allow handling short packets from softirq or VCPU context Plan: We are going through the scheduler 3 times (could be up to 5 if softirqd is involved) Consider RX: host irq -> io thread -> VCPU thread -> guest irq -> guest thread. This adds a lot of latency. We can cut it by some 1.5x if we do a bit of work either in the VCPU or softirq context. Testing: netperf TCP RR - should be improved drastically netperf TCP STREAM guest to host - no regression Developer: MST I am also tuning the vCPU scheduling of KVM. If someone would like to say some details about the work either in the vCPU or softirq context, I will be very appreciated. BTW, how to get the evaluation results that the shortcut can improve the performance by up to 1.5X? Thanks a lot! Regards, Cong -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
