[Gluster-devel] Fwd: [Gluster-users] gluster and LIO, fairly basic setup, having major issues

[Paul Cuzner]

-- Forwarded message --
From: Michael Ciccarelli 
Date: Thu, Oct 6, 2016 at 1:56 PM
Subject: [Gluster-users] gluster and LIO, fairly basic setup, having major
issues
To: gluster-us...@gluster.org


So I have a fairly basic setup using glusterfs between 2 nodes. The nodes
have 10 gig connections and the bricks reside on SSD LVM LUNs:

Brick1: media1-be:/gluster/brick1/gluster_volume_0
Brick2: media2-be:/gluster/brick1/gluster_volume_0


On this volume I have a LIO iscsi target with 1 fileio backstore that's
being shared out to vmware ESXi hosts. The volume is around 900 gig and the
fileio store is around 850g:

-rw-r--r-- 1 root root 912680550400 Oct  5 20:47 iscsi.disk.3

I set the WWN to be the same so the ESXi hosts see the nodes as 2 paths to
the same target. I believe this is what I want. The issues I'm seeing is
that while the IO wait is low I'm seeing high CPU usage with only 3 VMs
running on only 1 of the ESX servers:

this is media2-be:
  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND

 1474 root  20   0 1396620  37912   5980 S 135.0  0.1 157:01.84
glusterfsd
 1469 root  20   0  747996  13724   5424 S   2.0  0.0   1:10.59
glusterfs

And this morning it seemed like I had to restart the LIO service on
media1-be as the VMware was seeing time-out issues. I'm seeing issues like
this on the VMware ESX servers:

2016-10-06T00:51:41.100Z cpu0:32785)WARNING: ScsiDeviceIO: 1223: Device naa.
600140501ce79002e724ebdb66a6756d performance has deteriorated. I/O latency
increased from average value of 33420 microseconds to 732696 microseconds.

Are there any special settings I need to have gluster+LIO+vmware to work?
Has anyone gotten this to work fairly well that it is stable? What am I
missing?

thanks,
Mike



___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] 'Reviewd-by' tag for commits

[Michael Adam]

On 2016-10-05 at 09:45 -0400, Ira Cooper wrote:
> "Feedback-given-by: "

I like that one - thanks! :-)

Michael

> - Original Message -
> > On 2016-09-30 at 17:52 +0200, Niels de Vos wrote:
> > > On Fri, Sep 30, 2016 at 08:50:12PM +0530, Ravishankar N wrote:
> > > > On 09/30/2016 06:38 PM, Niels de Vos wrote:
> > > > > On Fri, Sep 30, 2016 at 07:11:51AM +0530, Pranith Kumar Karampuri
> > > > > wrote:
> > > ...
> > > > > Maybe we can add an additional tag that mentions all the people that
> > > > > did do reviews of older versions of the patch. Not sure what the tag
> > > > > would be, maybe just CC?
> > > > It depends on what tags would be processed to obtain statistics on 
> > > > review
> > > > contributions.
> > > 
> > > Real statistics would come from Gerrit, not from the 'git log' output.
> > > We do have a ./extras/who-wrote-glusterfs/ in the sources, but that is
> > > only to get an idea about the changes that were made and should not be
> > > used for serious statistics.
> > > 
> > > It is possible to feed the Gerrit comment-stream into things like
> > > Elasticsearch and get an accurate impression how many reviews people do
> > > (and much more). I hope we can get some contribution diagrams from
> > > someting like this at one point.
> > > 
> > > Would some kind of Gave-feedback tag for people that left a comment on
> > > earlier versions of the patch be appreciated by others? It will show in
> > > the 'git log' who was involved in some way or form.
> > 
> > I think this would be fair.
> > 
> > Reviewed-by tags should imho be reserved for the final
> > incarnation of the patch. Those mean that the person named
> > in the tag has aproved this version of the patch for getting
> > into the official tree. A previous version of the patch can
> > have been entirely different, so a reviewed-by for that
> > previous version may not actually apply to the new version at all
> > and hence create a false impression!
> > 
> > It is also difficult to track all activities by tags,
> > and anyone who wants to measure performance and contributions
> > only by looking at git commit tags will not be doing several
> > people justice. We could add 'discussed-with' or 'designed-by'
> > tags, etc ... ;-)
> > 
> > On a serious note, in Samba we use 'Pair-programmed-with' tags,
> > because we do pair-programming a lot, but only one person can
> > be an author of a git commit ...
> > 
> > The 'Gave-feedback' tag I do like. even though it does
> > not quite match with the foobar-by pattern of other tags.
> > 
> > Michael
> > 
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel


signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] quota-rename.t core in netbsd

[Vijay Bellur]

Hi All,

I observed a few crashes due to quota-rename.t in netbsd regression
runs [1] [2].  Raghavendra - can you please take a look when you get a
chance?

The core files and logs cannot be downloaded from the URLs in jenkins
job console history for NetBSD. I have logged a bug [3] on the
infrastructure for that.

Thanks,
Vijay

[1] https://build.gluster.org/job/netbsd7-regression/942/consoleFull

[2]  https://build.gluster.org/job/netbsd7-regression/945/console

[3] https://bugzilla.redhat.com/show_bug.cgi?id=1382097
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Regression caused to gfapi applications with enabling client-io-threads by default

[Pranith Kumar Karampuri]

On Wed, Oct 5, 2016 at 2:00 PM, Soumya Koduri  wrote:

> Hi,
>
> With http://review.gluster.org/#/c/15051/, performace/client-io-threads
> is enabled by default. But with that we see regression caused to
> nfs-ganesha application trying to un/re-export any glusterfs volume. This
> shall be the same case with any gfapi application using glfs_fini().
>
> More details and the RCA can be found at [1].
>
> In short, iot-worker threads spawned  (when the above option is enabled)
> are not cleaned up as part of io-threads-xlator->fini() and those threads
> could end up accessing invalid/freed memory post glfs_fini().
>
> The actual fix is to address io-threads-xlator->fini() to cleanup those
> threads before exiting. But since those threads' IDs are currently not
> stored, the fix could be very intricate and take a while. So till then to
> avoid all existing applications crash, I suggest to keep this option
> disabled by default and update this known_issue with enabling this option
> in the release-notes.
>
> I sent a patch to revert the commit - http://review.gluster.org/#/c/15616/
> [2]
>

Good catch! I think the correct fix would be to make sure all threads die
as part of PARENT_DOWN then?


> Comments/Suggestions are welcome.
>
> Thanks,
> Soumya
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1380619#c11
> [2] http://review.gluster.org/#/c/15616/
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] 'Reviewd-by' tag for commits

[Ira Cooper]

"Feedback-given-by: "

Cheers,

-Ira

- Original Message -
> On 2016-09-30 at 17:52 +0200, Niels de Vos wrote:
> > On Fri, Sep 30, 2016 at 08:50:12PM +0530, Ravishankar N wrote:
> > > On 09/30/2016 06:38 PM, Niels de Vos wrote:
> > > > On Fri, Sep 30, 2016 at 07:11:51AM +0530, Pranith Kumar Karampuri
> > > > wrote:
> > ...
> > > > Maybe we can add an additional tag that mentions all the people that
> > > > did do reviews of older versions of the patch. Not sure what the tag
> > > > would be, maybe just CC?
> > > It depends on what tags would be processed to obtain statistics on review
> > > contributions.
> > 
> > Real statistics would come from Gerrit, not from the 'git log' output.
> > We do have a ./extras/who-wrote-glusterfs/ in the sources, but that is
> > only to get an idea about the changes that were made and should not be
> > used for serious statistics.
> > 
> > It is possible to feed the Gerrit comment-stream into things like
> > Elasticsearch and get an accurate impression how many reviews people do
> > (and much more). I hope we can get some contribution diagrams from
> > someting like this at one point.
> > 
> > Would some kind of Gave-feedback tag for people that left a comment on
> > earlier versions of the patch be appreciated by others? It will show in
> > the 'git log' who was involved in some way or form.
> 
> I think this would be fair.
> 
> Reviewed-by tags should imho be reserved for the final
> incarnation of the patch. Those mean that the person named
> in the tag has aproved this version of the patch for getting
> into the official tree. A previous version of the patch can
> have been entirely different, so a reviewed-by for that
> previous version may not actually apply to the new version at all
> and hence create a false impression!
> 
> It is also difficult to track all activities by tags,
> and anyone who wants to measure performance and contributions
> only by looking at git commit tags will not be doing several
> people justice. We could add 'discussed-with' or 'designed-by'
> tags, etc ... ;-)
> 
> On a serious note, in Samba we use 'Pair-programmed-with' tags,
> because we do pair-programming a lot, but only one person can
> be an author of a git commit ...
> 
> The 'Gave-feedback' tag I do like. even though it does
> not quite match with the foobar-by pattern of other tags.
> 
> Michael
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] 'Reviewd-by' tag for commits

[Ira Cooper]

"Feedback-given-by: "

Cheers,

-IRa

- Original Message -
> On 2016-09-30 at 17:52 +0200, Niels de Vos wrote:
> > On Fri, Sep 30, 2016 at 08:50:12PM +0530, Ravishankar N wrote:
> > > On 09/30/2016 06:38 PM, Niels de Vos wrote:
> > > > On Fri, Sep 30, 2016 at 07:11:51AM +0530, Pranith Kumar Karampuri
> > > > wrote:
> > ...
> > > > Maybe we can add an additional tag that mentions all the people that
> > > > did do reviews of older versions of the patch. Not sure what the tag
> > > > would be, maybe just CC?
> > > It depends on what tags would be processed to obtain statistics on review
> > > contributions.
> > 
> > Real statistics would come from Gerrit, not from the 'git log' output.
> > We do have a ./extras/who-wrote-glusterfs/ in the sources, but that is
> > only to get an idea about the changes that were made and should not be
> > used for serious statistics.
> > 
> > It is possible to feed the Gerrit comment-stream into things like
> > Elasticsearch and get an accurate impression how many reviews people do
> > (and much more). I hope we can get some contribution diagrams from
> > someting like this at one point.
> > 
> > Would some kind of Gave-feedback tag for people that left a comment on
> > earlier versions of the patch be appreciated by others? It will show in
> > the 'git log' who was involved in some way or form.
> 
> I think this would be fair.
> 
> Reviewed-by tags should imho be reserved for the final
> incarnation of the patch. Those mean that the person named
> in the tag has aproved this version of the patch for getting
> into the official tree. A previous version of the patch can
> have been entirely different, so a reviewed-by for that
> previous version may not actually apply to the new version at all
> and hence create a false impression!
> 
> It is also difficult to track all activities by tags,
> and anyone who wants to measure performance and contributions
> only by looking at git commit tags will not be doing several
> people justice. We could add 'discussed-with' or 'designed-by'
> tags, etc ... ;-)
> 
> On a serious note, in Samba we use 'Pair-programmed-with' tags,
> because we do pair-programming a lot, but only one person can
> be an author of a git commit ...
> 
> The 'Gave-feedback' tag I do like. even though it does
> not quite match with the foobar-by pattern of other tags.
> 
> Michael
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] 'Reviewd-by' tag for commits

[Michael Adam]

On 2016-09-30 at 17:52 +0200, Niels de Vos wrote:
> On Fri, Sep 30, 2016 at 08:50:12PM +0530, Ravishankar N wrote:
> > On 09/30/2016 06:38 PM, Niels de Vos wrote:
> > > On Fri, Sep 30, 2016 at 07:11:51AM +0530, Pranith Kumar Karampuri wrote:
> ...
> > > Maybe we can add an additional tag that mentions all the people that
> > > did do reviews of older versions of the patch. Not sure what the tag
> > > would be, maybe just CC?
> > It depends on what tags would be processed to obtain statistics on review
> > contributions.
> 
> Real statistics would come from Gerrit, not from the 'git log' output.
> We do have a ./extras/who-wrote-glusterfs/ in the sources, but that is
> only to get an idea about the changes that were made and should not be
> used for serious statistics.
> 
> It is possible to feed the Gerrit comment-stream into things like
> Elasticsearch and get an accurate impression how many reviews people do
> (and much more). I hope we can get some contribution diagrams from
> someting like this at one point.
> 
> Would some kind of Gave-feedback tag for people that left a comment on
> earlier versions of the patch be appreciated by others? It will show in
> the 'git log' who was involved in some way or form.

I think this would be fair.

Reviewed-by tags should imho be reserved for the final
incarnation of the patch. Those mean that the person named
in the tag has aproved this version of the patch for getting
into the official tree. A previous version of the patch can
have been entirely different, so a reviewed-by for that
previous version may not actually apply to the new version at all
and hence create a false impression!

It is also difficult to track all activities by tags,
and anyone who wants to measure performance and contributions
only by looking at git commit tags will not be doing several
people justice. We could add 'discussed-with' or 'designed-by'
tags, etc ... ;-)

On a serious note, in Samba we use 'Pair-programmed-with' tags,
because we do pair-programming a lot, but only one person can
be an author of a git commit ...

The 'Gave-feedback' tag I do like. even though it does
not quite match with the foobar-by pattern of other tags.

Michael


signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Reminder: Weekly Gluster Community Meeting

[Ankit Raj]

Hi all,

The weekly Gluster community meeting is about to take place in ~30 minutes.

Meeting details:
- Location: #gluster-meeting on Freenode IRC
(https://webchat.freenode.net/?channels=gluster-meeting)
- Date: Every Wednesday
- Time: 12:00 UTC
(on your terminal, run: date -d "12:00 UTC")

Please find the agenda to be discussed in the meeting here:
https://public.pad.fsfe.org/p/gluster-community-meetings.

Currently the following topics are listed:
 - GlusterFS 4.0
 - GlusterFS 3.9
 - GlusterFS 3.8
 - GlusterFS 3.7
 - GlusterFS 3.6
 - Project infrastructure
 - NFS Ganesha
 - Samba
 - Last week's AI's
 - Open floor

If you have any other topic to be discussed please add so under the Open
floor section as a sub-topic.

Looking forward to your participation.

Thanks,

Ankit Raj
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Regression caused to gfapi applications with enabling client-io-threads by default

[Soumya Koduri]

Hi,

With http://review.gluster.org/#/c/15051/, performace/client-io-threads 
is enabled by default. But with that we see regression caused to 
nfs-ganesha application trying to un/re-export any glusterfs volume. 
This shall be the same case with any gfapi application using glfs_fini().


More details and the RCA can be found at [1].

In short, iot-worker threads spawned  (when the above option is enabled) 
are not cleaned up as part of io-threads-xlator->fini() and those 
threads could end up accessing invalid/freed memory post glfs_fini().


The actual fix is to address io-threads-xlator->fini() to cleanup those 
threads before exiting. But since those threads' IDs are currently not 
stored, the fix could be very intricate and take a while. So till then 
to avoid all existing applications crash, I suggest to keep this option 
disabled by default and update this known_issue with enabling this 
option in the release-notes.


I sent a patch to revert the commit - 
http://review.gluster.org/#/c/15616/ [2]


Comments/Suggestions are welcome.

Thanks,
Soumya

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1380619#c11
[2] http://review.gluster.org/#/c/15616/
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Duplicate entry in "gluster peer status" command

[Joe Julian]

On October 4, 2016 3:40:16 PM GMT+02:00, ABHISHEK PALIWAL 
 wrote:
>Hi,
>
>
>Again I am getting the duplicate peer entries in the peer status
>command
>while restarting system which causing sync failure in the gluster
>volume
>between both boards.

This is where you show the output so we can see what the problem is. 

>
>I am attaching logs from both the node could you please check this and
>help
>me in resolve the issue.

Good, but speaking for myself, some in-line context is better. I cannot open 
that tar file on my phone. 

>
>In logs we have BoardA and BoardB where BoardB showing duplicate
>entries in
>"gluster peer status" command.

Some background about how you got here would be helpful as well. 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel