Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
>> How many clients are you using? Each client op can only (currently) be >> handled in a single thread, and client's won't send more ops until the >> current one is ack'd, so Ganesha can basically only parallelize on a >> per-client basis at the moment. >> >>Actually, 2.6 should handle as many concurrent client requests as you like. (Up to 250 of them.) That's one of its features. < The client is not sending concurrent requests. There is only one client with 2 worker threads each worker thread generating 64 outstanding requests. Configured both servers (knfs & ganesha) with 128 worker threads. I think client is fine and is sending the request concurrently. For knfs and nfsganesha client is exactly the same, the only change is server exporting the directory. >> I'm sure there are locking issues; so far we've mostly worked on correctness >> rather than performance. 2.6 has changed the threading model a fair amount, >> and 2.7 will have more improvements, but it's a slow process. >> But the planned 2.7 improvements are mostly throughput related, not IOPS. > On 02/13/2018 06:38 PM, Deepak Jagtap wrote: >> Yeah user-kernel context switching is definitely adding up latency, but I >> wonder ifrpc or some locking overhead is also in the picture. >> < ifrpc? Sorry typo: ifrpc => 'if rpc' >> With 70% read 30% random workload nfs ganesha CPU usage was close to 170% >> while remaining 2 cores were pretty much unused (~18K IOPS, latency ~8ms) >> >> With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS, latency >> ~2ms). >> < Those latency numbers seem suspect to me. The dominant latency should be < the file system. The system calls shouldn't add more than microseconds. < If Ganesha is adding 6 ms to every read operation, we have a serious < problem, and need to profile immediately! Yeah that's right ideally in the end only physical disk IO latency should dominate in this complete stack. From: William Allen Simpson Sent: Friday, February 16, 2018 8:23:16 AM To: d...@redhat.com; Deepak Jagtap; nfs-ganesha-devel@lists.sourceforge.net Subject: Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance On 2/14/18 8:32 AM, Daniel Gryniewicz wrote: > How many clients are you using? Each client op can only (currently) be > handled in a single thread, and client's won't send more ops until the > current one is ack'd, so Ganesha can basically only parallelize on a > per-client basis at the moment. > Actually, 2.6 should handle as many concurrent client requests as you like. (Up to 250 of them.) That's one of its features. The client is not sending concurrent requests. > I'm sure there are locking issues; so far we've mostly worked on correctness > rather than performance. 2.6 has changed the threading model a fair amount, > and 2.7 will have more improvements, but it's a slow process. > But the planned 2.7 improvements are mostly throughput related, not IOPS. > On 02/13/2018 06:38 PM, Deepak Jagtap wrote: >> Yeah user-kernel context switching is definitely adding up latency, but I >> wonder ifrpc or some locking overhead is also in the picture. >> ifrpc? >> With 70% read 30% random workload nfs ganesha CPU usage was close to 170% >> while remaining 2 cores were pretty much unused (~18K IOPS, latency ~8ms) >> >> With 100% read 30% random nfs ganesha CPU usage ~250% ( ~50K IOPS, latency >> ~2ms). >> Those latency numbers seem suspect to me. The dominant latency should be the file system. The system calls shouldn't add more than microseconds. If Ganesha is adding 6 ms to every read operation, we have a serious problem, and need to profile immediately! -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hi Frank, I tried both 2.6 and 2.5-stable. 2.6 showed marginal improvements
at-least for my workload (improvement of ~1K for 70% read, 30% random workload).
-Deepak
From: Frank Filz
Sent: Tuesday, February 13, 2018 7:26:50 AM
To: Deepak Jagtap; nfs-ganesha-devel@lists.sourceforge.net
Subject: RE: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
What version are you running? Significant improvements have been made in V2.6
(which is close to being tagged V2.6.0).
Frank
From: Deepak Jagtap [mailto:deepak.jag...@maxta.com]
Sent: Monday, February 12, 2018 4:01 PM
To: nfs-ganesha-devel@lists.sourceforge.net
Subject: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hey Guys,
I ran few performance tests to compare nfs gansha and nfs kernel server and
noticed significant difference.
Please find my test result:
SSD formated with EXT3 exported using nfs ganesha : ~18K IOPSAvg latency:
~8ms Throughput: ~60MBPS
same directory exported using nfs kernel server: ~75K IOPSAvg
latency: ~0.8msThroughput: ~300MBPS
nfs kernel and nfs ganesha both of them are configured with 128 worker threads.
nfs ganesha is configured with VFS FSAL.
Am I missing something major in nfs ganesha config or this is expected behavior.
Appreciate any inputs as how the performance can be improved for nfs ganesha.
Please find following ganesha config file that I am using:
NFS_Core_Param
{
Nb_Worker = 128 ;
}
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 77;
# Exported path (mandatory)
Path = /host/test;
Protocols = 3;
# Pseudo Path (required for NFS v4)
Pseudo = /host/test;
# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;
# Exporting FSAL
FSAL {
Name = VFS;
}
CLIENT
{
Clients = *;
Squash = None;
Access_Type = RW;
}
}
Thanks & Regards,
Deepak
[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon>
Virus-free.
www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
> What is the actual test?
- nfs server: RHEL based host exporting ext3 formated SSD.
- nfs client: RHEL based host running windows vm IOmeter with 70% read, 30%
random workload, 2 worker threads doing IOs with 64 outstanding IO queue depth
each.
> What are the export options in the knfsd case?
- I am using this for exporting using knfsd.
exportfs -oinsecure,rw,sync,no_root_squash,fsid=77 *:/host/test
> Is there any reason you're testing ext3 instead of ext4 or xfs?
- Not in particular. I could have used ext4. I was evaluating nfs server
impact in both scenarios.
>What's the client? What are the kernel versions on server and client?
- Both of them are at 3.10.0-693.17.1.el7.x86_64 kernel version
Regards,
Deepak
From: J. Bruce Fields
Sent: Tuesday, February 13, 2018 11:18:51 AM
To: Deepak Jagtap
Cc: nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
On Tue, Feb 13, 2018 at 12:01:04AM +, Deepak Jagtap wrote:
> I ran few performance tests to compare nfs gansha and nfs kernel server and
> noticed significant difference.
What is the actual test?
What are the export options in the knfsd case?
Is there any reason you're testing ext3 instead of ext4 or xfs?
What's the client? What are the kernel versions on server and client?
--b.
> Please find my test result:
>
>
> SSD formated with EXT3 exported using nfs ganesha : ~18K IOPSAvg
> latency: ~8ms Throughput: ~60MBPS
>
> same directory exported using nfs kernel server: ~75K IOPSAvg
> latency: ~0.8msThroughput: ~300MBPS
>
>
> nfs kernel and nfs ganesha both of them are configured with 128 worker
> threads. nfs ganesha is configured with VFS FSAL.
>
>
> Am I missing something major in nfs ganesha config or this is expected
> behavior.
>
> Appreciate any inputs as how the performance can be improved for nfs ganesha.
>
>
>
> Please find following ganesha config file that I am using:
>
>
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
>
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
>
>
>
>
> Thanks & Regards,
>
> Deepak
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hi Matt,
I am using v2.5-stable branch for this evaluation.
Regards,
Deepak
From: Matt Benjamin
Sent: Monday, February 12, 2018 8:02:12 PM
To: Deepak Jagtap
Cc: nfs-ganesha-devel@lists.sourceforge.net
Subject: Re: [Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hi Deepak,
There's known knowns and unknown knowns related to nfs-ganesha
performance, including lots of ongoing work.
One of the first things I think we'd like to know is...what version of
the software you're testing.
Matt
On Mon, Feb 12, 2018 at 7:01 PM, Deepak Jagtap wrote:
> Hey Guys,
>
>
> I ran few performance tests to compare nfs gansha and nfs kernel server and
> noticed significant difference.
>
>
> Please find my test result:
>
>
> SSD formated with EXT3 exported using nfs ganesha : ~18K IOPSAvg
> latency: ~8ms Throughput: ~60MBPS
>
> same directory exported using nfs kernel server: ~75K IOPS
> Avg latency: ~0.8msThroughput: ~300MBPS
>
>
> nfs kernel and nfs ganesha both of them are configured with 128 worker
> threads. nfs ganesha is configured with VFS FSAL.
>
>
> Am I missing something major in nfs ganesha config or this is expected
> behavior.
>
> Appreciate any inputs as how the performance can be improved for nfs
> ganesha.
>
>
>
> Please find following ganesha config file that I am using:
>
>
> NFS_Core_Param
> {
> Nb_Worker = 128 ;
> }
>
> EXPORT
> {
> # Export Id (mandatory, each EXPORT must have a unique Export_Id)
>Export_Id = 77;
># Exported path (mandatory)
>Path = /host/test;
>Protocols = 3;
># Pseudo Path (required for NFS v4)
>Pseudo = /host/test;
># Required for access (default is None)
># Could use CLIENT blocks instead
>Access_Type = RW;
># Exporting FSAL
>FSAL {
> Name = VFS;
>}
>CLIENT
>{
> Clients = *;
> Squash = None;
> Access_Type = RW;
>}
> }
>
>
>
> Thanks & Regards,
>
> Deepak
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] nfs ganesha vs nfs kernel performance
Hey Guys,
I ran few performance tests to compare nfs gansha and nfs kernel server and
noticed significant difference.
Please find my test result:
SSD formated with EXT3 exported using nfs ganesha : ~18K IOPSAvg latency:
~8ms Throughput: ~60MBPS
same directory exported using nfs kernel server: ~75K IOPSAvg
latency: ~0.8msThroughput: ~300MBPS
nfs kernel and nfs ganesha both of them are configured with 128 worker threads.
nfs ganesha is configured with VFS FSAL.
Am I missing something major in nfs ganesha config or this is expected behavior.
Appreciate any inputs as how the performance can be improved for nfs ganesha.
Please find following ganesha config file that I am using:
NFS_Core_Param
{
Nb_Worker = 128 ;
}
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 77;
# Exported path (mandatory)
Path = /host/test;
Protocols = 3;
# Pseudo Path (required for NFS v4)
Pseudo = /host/test;
# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;
# Exporting FSAL
FSAL {
Name = VFS;
}
CLIENT
{
Clients = *;
Squash = None;
Access_Type = RW;
}
}
Thanks & Regards,
Deepak
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
