On May 31, 2022, at 9:14 AM, ron minnich <rminn...@gmail.com> wrote: > > On Mon, May 30, 2022 at 12:21 AM Bakul Shah <ba...@iitbombay.org> wrote: >> 9p itself is low performance but that is a separate issue. > > Bakul, what are the units? It might be helpful to quantify this > statement. Are you possibly conflating Plan 9 file systems being slow > and 9p being slow?
I did a quick test: >From a 9front VM to another machine I get about 11.7 MBps caching. The first time around it was close to 7.3 MBps). >From an Ubuntu VM to another machine I get about 111 MBps (cached. The first time around it was close to 62 MBps). Both VMs run on the same host. Test copies to the same target machine. I used 9p read for 9front, scp for Linux, copy to /dev/null. The target machine is freebsd. The VMs talk to the target over a 1Gbps ethernet (so 111 MBps is the wirespeed limit). 9front uses hjfs. Ubuntu uses ext4. On the host I give a file as the guest "disk", using 'nvme' type device on bhyve to each VM. Both 9front and ubuntu are 64 bit kernels. This is a very rough measurement as there are many differences between the systems. The filesystem overhead is clearly an issue but 10 times worse? ----- Looking at the protocol: For read/write 9p uses 4 byte for size so in theory you can send very large packets but then you have to buffer up a lot of data. Ideally you want streaming (some sort of sliding window). May be you can use the tag field to do something more intelligent. Not sure any implementations do so. You also have head of line blocking if you can have only one TCP connection to a server. > As Rob pointed out in 2013, "If go install is slow on Plan 9, it's > because Plan 9's file system is > slow (which it is and always has been)", so slowness in Plan 9 file > systems is to be expected. > > 9p itself does have its limits, which is why Bell Labs Antwerp started > an effort in 2011 to replace it, but the new work never went very far. > > I also know of a number of efforts in the virtualization world where > 9p was discarded for performance reasons. It's hard to argue with the > 100x performance improvement that comes with virtiofs, for example. Why is virtiofs 100x faster? Just lot of hardwork and tuning? May be that is good place to look to learn what needs to change (in case someone wants to replace 9p with something else)? > Gvisor is replacing 9p: https://github.com/google/gvisor/milestone/6. > Although, in the latter case, I would argue the problem is more with > Linux limitations than 9p limitations -- linux can't seem to walk more > than one pathname component at a time, for example, since it has the > old school namei loop. > > But I'm wondering if you have a measurement with numbers. > > For rough order of magnitude, HPC file systems can deliver 10 Gbytes/ > second for file reads nowadays, but getting there took 20 years of > work. When we ran Plan 9 on Blue Gene, with the 6 Gbyte/second > toroidal mesh connect for each node, we never came remotely close to > that figure. Given that experience, why do you need "numbers"? :-) Running 10Gbps links even @ home is quite doable now. With TCP you can achieve decent performance if not quite wirespeed. NVMe "disks" are pretty damn fast - you can easily get 2-4 GBps. But I think at remote filesystem protocol level you'd have to optimize multiple things in order to get close to wirespeed performance. Minimize copying, increase concurrency, reduce overhead in frequently used common path code, reduce user/kernel crossings etc. I think rdma and mmap will probably get used a lot too (obviously on non-plan9 OSes!). May be if you pushed 9p knowledge down to a smart NIC, it can map a tag value to compute location where the data needs to go. But all this is just handwaving. Without a real project and funding it is hard to get sufficiently motivated to do more. ------------------------------------------ 9fans: 9fans Permalink: https://9fans.topicbox.com/groups/9fans/T769854fafd2b7d35-Mf37a0689afc5c54c9aba65d7 Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
