Andrew Gallatin wrote: >Rao Shoaib writes: > > >If there is a reader/writer lock, what are the implications of waiting > > >a "long time" for the network send to complete if zero-copy is done > > >by NFS, rather than nearly instantly if the data is bcopied? > > > > > > > > This is one of the reasons NFS does not use this approach. Infact > >I worked with a grad student at Duke on doing zero-copy like this >years back in "slice" (an NFS-like cluster filesystem) using FreeBSD. >I'm glad I still remember the issues :) > > > desballoca was implemented specifcially for NFS so that the buffer is > > freed right away as opposed to esballoca which delays the freeing of the > > buffer. However we ran into a file corruption issue. It turns out some > > NIC drivers (do not remember which) wrote on the mblks. > >Those drivers should be taken out and shot. These days, there is a >zerocopy attribute that you should be able to check for, similar to >how the sendfile code works. > > I agree. However I dont think any modern drivers do that or else sendfile would not work.
> > BTW I am not convinced that zero copy is always necessary. When I was > > doing NFS performance in Solaris 8/9 time frame (pre fire engine) I > > coded an nfs server where there was no copy i.e the read and write on > > the server were no-ops. I did not see any performance gain as the > > network and cpu were the bottleneck. That could have changed now with > > faster networking and CPU's but than the copying is also much faster today. > >It has been a while since I've had a zero-copy FS to play with. I can >tell you that with the 10GbE nic I do drivers for, the CPU overhead >drops by 50-60% when you skip the copy on socket writes from userspace >(sendfile vs write) on both opteron and T2000. This translates to >more than doubling the single stream bandwidth on anemic CPUs (like a >single core on a T2000). > >I think you really should look at zero-copy again. It would be like >doubling the CPU on a busy server... > > Sure I am ccing NFS discuss so that they can look into this again if they are not already. It will be interesting to know if there is extra over head in doing user to kernel copy than just kernel to kernel copy (which will be the FS case). BTW as you may know sendfile does not always uses zero copy. I am assuming in your measurements it was using zero copy. A question for the VM team. In my experiments I found that on the client the biggest overhead was getting a page for the file and that is why I coded nfs direct io option to by pass the VM. Will the new VM make this cheaper ? Rao. >Drew > >
