Re: [Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-12 Thread Serkan Çoban 
Sorry, I already destroyed 3.12.0 and servers are in production with 3.10.5.

On Tue, Sep 12, 2017 at 12:46 PM, Pranith Kumar Karampuri
 wrote:
> Serkan,
> Will it be possible to provide gluster volume profile  info
> output with 3.10.5 vs 3.12.0? That should give us clues about what could be
> happening.
>
> On Tue, Sep 12, 2017 at 1:51 PM, Serkan Çoban  wrote:
>>
>> Hi,
>> Servers are in production with 3.10.5, so I cannot provide 3.12
>> related information anymore.
>> Thanks for help, sorry for inconvenience.
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-12 Thread Pranith Kumar Karampuri 
Serkan,
Will it be possible to provide gluster volume profile 
info output with 3.10.5 vs 3.12.0? That should give us clues about what
could be happening.

On Tue, Sep 12, 2017 at 1:51 PM, Serkan Çoban  wrote:

> Hi,
> Servers are in production with 3.10.5, so I cannot provide 3.12
> related information anymore.
> Thanks for help, sorry for inconvenience.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-12 Thread Serkan Çoban 
Hi,
Servers are in production with 3.10.5, so I cannot provide 3.12
related information anymore.
Thanks for help, sorry for inconvenience.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-11 Thread Shyam Ranganathan 

Here are my results:

Summary: I am not able to reproduce the problem, IOW I get relatively 
equivalent numbers for sequential IO when going against 3.10.5 or 3.12.0


Next steps:
- Could you pass along your volfile (both for a brick and also the 
client vol file (from 
/var/lib/glusterd/vols//patchy.tcp-fuse.vol and a brick vol 
file from the same place)
  - I want to check what options are in use in your setup as compared 
to mine and see if that makes a difference


- Is it possible for you to run the IOZone test, as below (it needs more 
clarification in case you have not used IOZone before, so reach out in 
that case) in your setup and report the results?


- Details:

Test: IOZone iozone -+m / -+h  -C -w -c -e -i 
0 -+n -r 256k -s 2g -t 4/8/16/32
  - Sequential IOZone write tests, with -t number of threads (files) 
per client, across 4 clients, and using a 256k record size

  - Essentially 8/16/32 threads per client, which are 4 in total

Volume: 6x(4+2) disperse, on 36 disks, each disk is a SAS 10k JBOD, 
configured with the defaults, when creating the volume using 3.10.5 as a 
start point.


Server network saturation expectation: The brick nodes will get 1.5 
times the data that the client generated (4+2). As a result, aggregate 
IOZone results should be seen as, 1.5XAggregate/4 per server network 
utilization.


Results:
Threads/client  3.10.5  3.12.0
 (bytes/sec aggregate)
8   1938564 1922249
16  2044677 2082817
32  2465990 2435669

The brick nodes (which are separate from the client nodes) have a 
(greater than) 10G interface.


At best (32 threads/client case), I see the server link getting utilized as,
3.10.5: (2465990*1.5)/(4*1024) = 903MB/sec
3.12.0: (2465990*1.5)/(4*1024) = 892MB/sec

Shyam
On 09/07/2017 12:07 AM, Serkan Çoban wrote:

It is sequential write with file size 2GB. Same behavior observed with
3.11.3 too.

On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan  wrote:

On 09/06/2017 05:48 AM, Serkan Çoban wrote:


Hi,

Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
100 clients are writing each has 5 threads total 500 threads.
With 3.10.5 each server has 800MB/s network traffic, cluster total is
32GB/s
With 3.12.0 each server has 200MB/s network traffic, cluster total is
8GB/s
I did not change any volume options in both configs.



I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume
and compared this against 3.10.5 and 3.12.0. The tests are no where near
your capacity, but I do not see anything alarming in the results. (4 server,
4 clients, 4 worker thread per client)

I do notice a 6% drop in Sequential and random write performance, and gains
in the sequential and random reads.

I need to improve the test to do larger files and for a longer duration,
hence not reporting any numbers as yet.

Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and
remounting on the clients (after the versions were upgraded there as well).

I guess your test can be characterized as a sequential write workload
(ingestion of data). What is the average file size being ingested? I can
mimic something equivalent to that to look at this further.

I would like to ensure there are no evident performance regressions as you
report.

Shyam

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-06 Thread Serkan Çoban 
It is sequential write with file size 2GB. Same behavior observed with
3.11.3 too.

On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan  wrote:
> On 09/06/2017 05:48 AM, Serkan Çoban wrote:
>>
>> Hi,
>>
>> Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
>> 100 clients are writing each has 5 threads total 500 threads.
>> With 3.10.5 each server has 800MB/s network traffic, cluster total is
>> 32GB/s
>> With 3.12.0 each server has 200MB/s network traffic, cluster total is
>> 8GB/s
>> I did not change any volume options in both configs.
>
>
> I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume
> and compared this against 3.10.5 and 3.12.0. The tests are no where near
> your capacity, but I do not see anything alarming in the results. (4 server,
> 4 clients, 4 worker thread per client)
>
> I do notice a 6% drop in Sequential and random write performance, and gains
> in the sequential and random reads.
>
> I need to improve the test to do larger files and for a longer duration,
> hence not reporting any numbers as yet.
>
> Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and
> remounting on the clients (after the versions were upgraded there as well).
>
> I guess your test can be characterized as a sequential write workload
> (ingestion of data). What is the average file size being ingested? I can
> mimic something equivalent to that to look at this further.
>
> I would like to ensure there are no evident performance regressions as you
> report.
>
> Shyam
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-06 Thread Shyam Ranganathan 

On 09/06/2017 05:48 AM, Serkan Çoban wrote:

Hi,

Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
100 clients are writing each has 5 threads total 500 threads.
With 3.10.5 each server has 800MB/s network traffic, cluster total is 32GB/s
With 3.12.0 each server has 200MB/s network traffic, cluster total is 8GB/s
I did not change any volume options in both configs.


I just performed some *basic* IOZone tests on a 6 x (4+2) disperse 
volume and compared this against 3.10.5 and 3.12.0. The tests are no 
where near your capacity, but I do not see anything alarming in the 
results. (4 server, 4 clients, 4 worker thread per client)


I do notice a 6% drop in Sequential and random write performance, and 
gains in the sequential and random reads.


I need to improve the test to do larger files and for a longer duration, 
hence not reporting any numbers as yet.


Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and 
remounting on the clients (after the versions were upgraded there as well).


I guess your test can be characterized as a sequential write workload 
(ingestion of data). What is the average file size being ingested? I can 
mimic something equivalent to that to look at this further.


I would like to ensure there are no evident performance regressions as 
you report.


Shyam
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

2017-09-06 Thread Serkan Çoban 
Hi,

Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
100 clients are writing each has 5 threads total 500 threads.
With 3.10.5 each server has 800MB/s network traffic, cluster total is 32GB/s
With 3.12.0 each server has 200MB/s network traffic, cluster total is 8GB/s
I did not change any volume options in both configs.

Any thoughts?
Serkan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users