Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

Hi Niels,

I have backported that patch on Gluster 3.7.6 and we haven't seen any other
issue due to that patch.

Everything is fine till now in our testing and its going on extensively.

Regards,
Abhishek

On Thu, Jun 1, 2017 at 1:46 PM, Niels de Vos  wrote:

> On Thu, Jun 01, 2017 at 01:03:25PM +0530, ABHISHEK PALIWAL wrote:
> > Hi Niels,
> >
> > No problem we wil try to backport that patch on 3.7.6.
> >
> > Could you please let me know in which release Gluster community is going
> to
> > provide this patch and date of that release?
>
> It really depends on when someone has time to work on it. Our releases
> are time based, and will happen even when a bugfix/feature is not merged
> or implemented. We can't give any guarantees about availability for
> final patche (or backports).
>
> The best you can do is help testing a potential fix, and work with the
> developer(s) of that patch to improve and get it accepted in the master
> branch. If developers do not have time to work on it, or progress is
> slow, you can ask them if you can take it over from if you are
> comfortable with writing the code.
>
> Niels
>
>
> >
> > Regards,
> > Abhishek
> >
> > On Wed, May 31, 2017 at 10:05 PM, Niels de Vos 
> wrote:
> >
> > > On Wed, May 31, 2017 at 04:08:06PM +0530, ABHISHEK PALIWAL wrote:
> > > > We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
> > > status
> > > > is "can't merge"
> > >
> > > Note that 3.7.x will not get any updates anymore. We currently maintain
> > > version 3.8.x, 3.10.x and 3.11.x. See the release schedele for more
> > > details:
> > >   https://www.gluster.org/community/release-schedule/
> > >
> > > Niels
> > >
> > >
> > > >
> > > > On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
> > > wrote:
> > > >
> > > > > This is already part of 3.11.0 release?
> > > > >
> > > > > On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
> > > abhishpali...@gmail.com
> > > > > > wrote:
> > > > >
> > > > >> Hi Atin,
> > > > >>
> > > > >> Could you please let us know any time plan for deliver of this
> patch.
> > > > >>
> > > > >> Regards,
> > > > >> Abhishek
> > > > >>
> > > > >> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
> > > abhishpali...@gmail.com
> > > > >> > wrote:
> > > > >>
> > > > >>> Actually it is very risky if it will reproduce in production
> thats is
> > > > >>> why I said it is on high priority as want to resolve it before
> > > production.
> > > > >>>
> > > > >>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee <
> amukh...@redhat.com>
> > > > >>> wrote:
> > > > >>>
> > > > 
> > > > 
> > > >  On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
> > > >  abhishpali...@gmail.com> wrote:
> > > > 
> > > > > Hi Atin,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > >
> > > > > Its urgent because this error is very rarely reproducible we
> have
> > > seen
> > > > > this 2 3 times in our system till now.
> > > > >
> > > > > We have delivery in near future so that we want it asap. Please
> > > try to
> > > > > review it internally.
> > > > >
> > > > 
> > > >  I don't think your statements justified the reason of urgency
> as (a)
> > > >  you have mentioned it to be *rarely* reproducible and (b) I am
> still
> > > >  waiting for a real use case where glusterd will go through
> multiple
> > > >  restarts in a loop?
> > > > 
> > > > 
> > > > > Regards,
> > > > > Abhishek
> > > > >
> > > > > On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee <
> > > amukh...@redhat.com>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >>
> > > > >> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> > > > >> abhishpali...@gmail.com> wrote:
> > > > >>
> > > > >>> + Muthu-vingeshwaran
> > > > >>>
> > > > >>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> > > > >>> abhishpali...@gmail.com> wrote:
> > > > >>>
> > > >  Hi Atin/Team,
> > > > 
> > > >  We are using gluster-3.7.6 with setup of two brick and while
> > > >  restart of system I have seen that the glusterd daemon is
> > > getting failed
> > > >  from start.
> > > > 
> > > > 
> > > >  At the time of analyzing the logs from
> etc-glusterfs...log
> > > file
> > > >  I have received the below logs
> > > > 
> > > > 
> > > >  [2017-05-06 03:33:39.798087] I [MSGID: 100030]
> > > >  [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started
> running
> > > >  /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd
> -p
> > > >  /var/run/glusterd.pid --log-level INFO)
> > > >  [2017-05-06 03:33:39.807859] I [MSGID: 106478]
> > > >  [glusterd.c:1350:init] 0-management: Maximum allowed open
> file
> > > descriptors
> > > >  set to 65536
> > > >  [2017-05-06 03:33:39.807974] I [MSGID: 

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[Niels de Vos]

On Thu, Jun 01, 2017 at 01:03:25PM +0530, ABHISHEK PALIWAL wrote:
> Hi Niels,
> 
> No problem we wil try to backport that patch on 3.7.6.
> 
> Could you please let me know in which release Gluster community is going to
> provide this patch and date of that release?

It really depends on when someone has time to work on it. Our releases
are time based, and will happen even when a bugfix/feature is not merged
or implemented. We can't give any guarantees about availability for
final patche (or backports).

The best you can do is help testing a potential fix, and work with the
developer(s) of that patch to improve and get it accepted in the master
branch. If developers do not have time to work on it, or progress is
slow, you can ask them if you can take it over from if you are
comfortable with writing the code.

Niels


> 
> Regards,
> Abhishek
> 
> On Wed, May 31, 2017 at 10:05 PM, Niels de Vos  wrote:
> 
> > On Wed, May 31, 2017 at 04:08:06PM +0530, ABHISHEK PALIWAL wrote:
> > > We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
> > status
> > > is "can't merge"
> >
> > Note that 3.7.x will not get any updates anymore. We currently maintain
> > version 3.8.x, 3.10.x and 3.11.x. See the release schedele for more
> > details:
> >   https://www.gluster.org/community/release-schedule/
> >
> > Niels
> >
> >
> > >
> > > On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
> > wrote:
> > >
> > > > This is already part of 3.11.0 release?
> > > >
> > > > On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
> > abhishpali...@gmail.com
> > > > > wrote:
> > > >
> > > >> Hi Atin,
> > > >>
> > > >> Could you please let us know any time plan for deliver of this patch.
> > > >>
> > > >> Regards,
> > > >> Abhishek
> > > >>
> > > >> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
> > abhishpali...@gmail.com
> > > >> > wrote:
> > > >>
> > > >>> Actually it is very risky if it will reproduce in production thats is
> > > >>> why I said it is on high priority as want to resolve it before
> > production.
> > > >>>
> > > >>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
> > > >>> wrote:
> > > >>>
> > > 
> > > 
> > >  On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
> > >  abhishpali...@gmail.com> wrote:
> > > 
> > > > Hi Atin,
> > > >
> > > > Thanks for your reply.
> > > >
> > > >
> > > > Its urgent because this error is very rarely reproducible we have
> > seen
> > > > this 2 3 times in our system till now.
> > > >
> > > > We have delivery in near future so that we want it asap. Please
> > try to
> > > > review it internally.
> > > >
> > > 
> > >  I don't think your statements justified the reason of urgency as (a)
> > >  you have mentioned it to be *rarely* reproducible and (b) I am still
> > >  waiting for a real use case where glusterd will go through multiple
> > >  restarts in a loop?
> > > 
> > > 
> > > > Regards,
> > > > Abhishek
> > > >
> > > > On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee <
> > amukh...@redhat.com>
> > > > wrote:
> > > >
> > > >>
> > > >>
> > > >> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> > > >> abhishpali...@gmail.com> wrote:
> > > >>
> > > >>> + Muthu-vingeshwaran
> > > >>>
> > > >>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> > > >>> abhishpali...@gmail.com> wrote:
> > > >>>
> > >  Hi Atin/Team,
> > > 
> > >  We are using gluster-3.7.6 with setup of two brick and while
> > >  restart of system I have seen that the glusterd daemon is
> > getting failed
> > >  from start.
> > > 
> > > 
> > >  At the time of analyzing the logs from etc-glusterfs...log
> > file
> > >  I have received the below logs
> > > 
> > > 
> > >  [2017-05-06 03:33:39.798087] I [MSGID: 100030]
> > >  [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
> > >  /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
> > >  /var/run/glusterd.pid --log-level INFO)
> > >  [2017-05-06 03:33:39.807859] I [MSGID: 106478]
> > >  [glusterd.c:1350:init] 0-management: Maximum allowed open file
> > descriptors
> > >  set to 65536
> > >  [2017-05-06 03:33:39.807974] I [MSGID: 106479]
> > >  [glusterd.c:1399:init] 0-management: Using /system/glusterd as
> > working
> > >  directory
> > >  [2017-05-06 03:33:39.826833] I [MSGID: 106513]
> > >  [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
> > >  retrieved op-version: 30706
> > >  [2017-05-06 03:33:39.827515] E [MSGID: 106206]
> > >  [glusterd-store.c:2562:glusterd_store_update_volinfo]
> > >  0-management: Failed to get next store iter
> > >  [2017-05-06 03:33:39.827563] E [MSGID: 106207]
> 

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

Hi Niels,

No problem we wil try to backport that patch on 3.7.6.

Could you please let me know in which release Gluster community is going to
provide this patch and date of that release?

Regards,
Abhishek

On Wed, May 31, 2017 at 10:05 PM, Niels de Vos  wrote:

> On Wed, May 31, 2017 at 04:08:06PM +0530, ABHISHEK PALIWAL wrote:
> > We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
> status
> > is "can't merge"
>
> Note that 3.7.x will not get any updates anymore. We currently maintain
> version 3.8.x, 3.10.x and 3.11.x. See the release schedele for more
> details:
>   https://www.gluster.org/community/release-schedule/
>
> Niels
>
>
> >
> > On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
> wrote:
> >
> > > This is already part of 3.11.0 release?
> > >
> > > On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com
> > > > wrote:
> > >
> > >> Hi Atin,
> > >>
> > >> Could you please let us know any time plan for deliver of this patch.
> > >>
> > >> Regards,
> > >> Abhishek
> > >>
> > >> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com
> > >> > wrote:
> > >>
> > >>> Actually it is very risky if it will reproduce in production thats is
> > >>> why I said it is on high priority as want to resolve it before
> production.
> > >>>
> > >>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
> > >>> wrote:
> > >>>
> > 
> > 
> >  On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
> >  abhishpali...@gmail.com> wrote:
> > 
> > > Hi Atin,
> > >
> > > Thanks for your reply.
> > >
> > >
> > > Its urgent because this error is very rarely reproducible we have
> seen
> > > this 2 3 times in our system till now.
> > >
> > > We have delivery in near future so that we want it asap. Please
> try to
> > > review it internally.
> > >
> > 
> >  I don't think your statements justified the reason of urgency as (a)
> >  you have mentioned it to be *rarely* reproducible and (b) I am still
> >  waiting for a real use case where glusterd will go through multiple
> >  restarts in a loop?
> > 
> > 
> > > Regards,
> > > Abhishek
> > >
> > > On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee <
> amukh...@redhat.com>
> > > wrote:
> > >
> > >>
> > >>
> > >> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> > >> abhishpali...@gmail.com> wrote:
> > >>
> > >>> + Muthu-vingeshwaran
> > >>>
> > >>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> > >>> abhishpali...@gmail.com> wrote:
> > >>>
> >  Hi Atin/Team,
> > 
> >  We are using gluster-3.7.6 with setup of two brick and while
> >  restart of system I have seen that the glusterd daemon is
> getting failed
> >  from start.
> > 
> > 
> >  At the time of analyzing the logs from etc-glusterfs...log
> file
> >  I have received the below logs
> > 
> > 
> >  [2017-05-06 03:33:39.798087] I [MSGID: 100030]
> >  [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
> >  /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
> >  /var/run/glusterd.pid --log-level INFO)
> >  [2017-05-06 03:33:39.807859] I [MSGID: 106478]
> >  [glusterd.c:1350:init] 0-management: Maximum allowed open file
> descriptors
> >  set to 65536
> >  [2017-05-06 03:33:39.807974] I [MSGID: 106479]
> >  [glusterd.c:1399:init] 0-management: Using /system/glusterd as
> working
> >  directory
> >  [2017-05-06 03:33:39.826833] I [MSGID: 106513]
> >  [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
> >  retrieved op-version: 30706
> >  [2017-05-06 03:33:39.827515] E [MSGID: 106206]
> >  [glusterd-store.c:2562:glusterd_store_update_volinfo]
> >  0-management: Failed to get next store iter
> >  [2017-05-06 03:33:39.827563] E [MSGID: 106207]
> >  [glusterd-store.c:2844:glusterd_store_retrieve_volume]
> >  0-management: Failed to update volinfo for c_glusterfs volume
> >  [2017-05-06 03:33:39.827625] E [MSGID: 106201]
> >  [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
> >  0-management: Unable to restore volume: c_glusterfs
> >  [2017-05-06 03:33:39.827722] E [MSGID: 101019]
> >  [xlator.c:428:xlator_init] 0-management: Initialization of
> volume
> >  'management' failed, review your volfile again
> >  [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_
> init]
> >  0-management: initializing translator failed
> >  [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_
> activate]
> >  0-graph: init failed
> >  [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_
> and_exit]
> 

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[Niels de Vos]

On Wed, May 31, 2017 at 04:08:06PM +0530, ABHISHEK PALIWAL wrote:
> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279 status
> is "can't merge"

Note that 3.7.x will not get any updates anymore. We currently maintain
version 3.8.x, 3.10.x and 3.11.x. See the release schedele for more
details:
  https://www.gluster.org/community/release-schedule/

Niels


> 
> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi  wrote:
> 
> > This is already part of 3.11.0 release?
> >
> > On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL  > > wrote:
> >
> >> Hi Atin,
> >>
> >> Could you please let us know any time plan for deliver of this patch.
> >>
> >> Regards,
> >> Abhishek
> >>
> >> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL  >> > wrote:
> >>
> >>> Actually it is very risky if it will reproduce in production thats is
> >>> why I said it is on high priority as want to resolve it before production.
> >>>
> >>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
> >>> wrote:
> >>>
> 
> 
>  On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
>  abhishpali...@gmail.com> wrote:
> 
> > Hi Atin,
> >
> > Thanks for your reply.
> >
> >
> > Its urgent because this error is very rarely reproducible we have seen
> > this 2 3 times in our system till now.
> >
> > We have delivery in near future so that we want it asap. Please try to
> > review it internally.
> >
> 
>  I don't think your statements justified the reason of urgency as (a)
>  you have mentioned it to be *rarely* reproducible and (b) I am still
>  waiting for a real use case where glusterd will go through multiple
>  restarts in a loop?
> 
> 
> > Regards,
> > Abhishek
> >
> > On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
> > wrote:
> >
> >>
> >>
> >> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> >> abhishpali...@gmail.com> wrote:
> >>
> >>> + Muthu-vingeshwaran
> >>>
> >>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> >>> abhishpali...@gmail.com> wrote:
> >>>
>  Hi Atin/Team,
> 
>  We are using gluster-3.7.6 with setup of two brick and while
>  restart of system I have seen that the glusterd daemon is getting 
>  failed
>  from start.
> 
> 
>  At the time of analyzing the logs from etc-glusterfs...log file
>  I have received the below logs
> 
> 
>  [2017-05-06 03:33:39.798087] I [MSGID: 100030]
>  [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
>  /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
>  /var/run/glusterd.pid --log-level INFO)
>  [2017-05-06 03:33:39.807859] I [MSGID: 106478]
>  [glusterd.c:1350:init] 0-management: Maximum allowed open file 
>  descriptors
>  set to 65536
>  [2017-05-06 03:33:39.807974] I [MSGID: 106479]
>  [glusterd.c:1399:init] 0-management: Using /system/glusterd as 
>  working
>  directory
>  [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>  [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>  retrieved op-version: 30706
>  [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>  [glusterd-store.c:2562:glusterd_store_update_volinfo]
>  0-management: Failed to get next store iter
>  [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>  [glusterd-store.c:2844:glusterd_store_retrieve_volume]
>  0-management: Failed to update volinfo for c_glusterfs volume
>  [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>  [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
>  0-management: Unable to restore volume: c_glusterfs
>  [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>  [xlator.c:428:xlator_init] 0-management: Initialization of volume
>  'management' failed, review your volfile again
>  [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>  0-management: initializing translator failed
>  [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>  0-graph: init failed
>  [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>  (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8)
>  [0x1000a648] -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210)
>  [0x1000a4d8] -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac)
>  [0x100097ac] ) 0-: received signum (0), shutting down
> 
> >>>
> >> Abhishek,
> >>
> >> This patch needs to be thoroughly reviewed to ensure that it doesn't
> >> cause any regression given this touches on the core store management
> >> functionality of 

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

So is there any one working on it to fix this issue either by this patch or
some other way? if yes then please provide the time plan.

On Wed, May 31, 2017 at 4:25 PM, Amar Tumballi  wrote:

>
>
> On Wed, May 31, 2017 at 4:08 PM, ABHISHEK PALIWAL  > wrote:
>
>> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
>> status is "can't merge"
>>
>> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
>> wrote:
>>
>>> This is already part of 3.11.0 release?
>>>
>>
> Sorry about confusion! I was thinking of another patch. This patch is not
> part of any releases yet.
>
> It says can't merge because it is failing regression tests and also looks
> like it needs a rebase to latest master branch too.
>
> -Amar
>
>
>>> On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin,

 Could you please let us know any time plan for deliver of this patch.

 Regards,
 Abhishek

 On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> Actually it is very risky if it will reproduce in production thats is
> why I said it is on high priority as want to resolve it before production.
>
>


-- 




Regards
Abhishek Paliwal
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[Amar Tumballi]

On Wed, May 31, 2017 at 4:08 PM, ABHISHEK PALIWAL 
wrote:

> We are using 3.7.6 and on link https://review.gluster.org/#/c/16279
> status is "can't merge"
>
> On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi 
> wrote:
>
>> This is already part of 3.11.0 release?
>>
>
Sorry about confusion! I was thinking of another patch. This patch is not
part of any releases yet.

It says can't merge because it is failing regression tests and also looks
like it needs a rebase to latest master branch too.

-Amar


>> On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> Hi Atin,
>>>
>>> Could you please let us know any time plan for deliver of this patch.
>>>
>>> Regards,
>>> Abhishek
>>>
>>> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Actually it is very risky if it will reproduce in production thats is
 why I said it is on high priority as want to resolve it before production.


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

We are using 3.7.6 and on link https://review.gluster.org/#/c/16279 status
is "can't merge"

On Wed, May 31, 2017 at 4:05 PM, Amar Tumballi  wrote:

> This is already part of 3.11.0 release?
>
> On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL  > wrote:
>
>> Hi Atin,
>>
>> Could you please let us know any time plan for deliver of this patch.
>>
>> Regards,
>> Abhishek
>>
>> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL > > wrote:
>>
>>> Actually it is very risky if it will reproduce in production thats is
>>> why I said it is on high priority as want to resolve it before production.
>>>
>>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
>>> wrote:
>>>


 On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> Hi Atin,
>
> Thanks for your reply.
>
>
> Its urgent because this error is very rarely reproducible we have seen
> this 2 3 times in our system till now.
>
> We have delivery in near future so that we want it asap. Please try to
> review it internally.
>

 I don't think your statements justified the reason of urgency as (a)
 you have mentioned it to be *rarely* reproducible and (b) I am still
 waiting for a real use case where glusterd will go through multiple
 restarts in a loop?


> Regards,
> Abhishek
>
> On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> + Muthu-vingeshwaran
>>>
>>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin/Team,

 We are using gluster-3.7.6 with setup of two brick and while
 restart of system I have seen that the glusterd daemon is getting 
 failed
 from start.


 At the time of analyzing the logs from etc-glusterfs...log file
 I have received the below logs


 [2017-05-06 03:33:39.798087] I [MSGID: 100030]
 [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
 /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
 /var/run/glusterd.pid --log-level INFO)
 [2017-05-06 03:33:39.807859] I [MSGID: 106478]
 [glusterd.c:1350:init] 0-management: Maximum allowed open file 
 descriptors
 set to 65536
 [2017-05-06 03:33:39.807974] I [MSGID: 106479]
 [glusterd.c:1399:init] 0-management: Using /system/glusterd as working
 directory
 [2017-05-06 03:33:39.826833] I [MSGID: 106513]
 [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
 retrieved op-version: 30706
 [2017-05-06 03:33:39.827515] E [MSGID: 106206]
 [glusterd-store.c:2562:glusterd_store_update_volinfo]
 0-management: Failed to get next store iter
 [2017-05-06 03:33:39.827563] E [MSGID: 106207]
 [glusterd-store.c:2844:glusterd_store_retrieve_volume]
 0-management: Failed to update volinfo for c_glusterfs volume
 [2017-05-06 03:33:39.827625] E [MSGID: 106201]
 [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
 0-management: Unable to restore volume: c_glusterfs
 [2017-05-06 03:33:39.827722] E [MSGID: 101019]
 [xlator.c:428:xlator_init] 0-management: Initialization of volume
 'management' failed, review your volfile again
 [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
 0-management: initializing translator failed
 [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
 0-graph: init failed
 [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
 (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8)
 [0x1000a648] -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210)
 [0x1000a4d8] -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac)
 [0x100097ac] ) 0-: received signum (0), shutting down

>>>
>> Abhishek,
>>
>> This patch needs to be thoroughly reviewed to ensure that it doesn't
>> cause any regression given this touches on the core store management
>> functionality of glusterd. AFAICT, we get into an empty info file only 
>> when
>> volume set operation is executed and in parallel one of the glusterd
>> instance in other nodes have been brought down and whole sequence of
>> operation happens in a loop. The test case through which you can get into
>> this situation is not something you'd hit in production. Please help me 
>> to
>> understand the urgency here.
>>
>> Also in one of the earlier thread, I did mention the workaround of
>> this issue back 

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[Amar Tumballi]

This is already part of 3.11.0 release?

On Wed, May 31, 2017 at 3:47 PM, ABHISHEK PALIWAL 
wrote:

> Hi Atin,
>
> Could you please let us know any time plan for deliver of this patch.
>
> Regards,
> Abhishek
>
> On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL 
> wrote:
>
>> Actually it is very risky if it will reproduce in production thats is why
>> I said it is on high priority as want to resolve it before production.
>>
>> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
>> wrote:
>>
>>>
>>>
>>> On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin,

 Thanks for your reply.


 Its urgent because this error is very rarely reproducible we have seen
 this 2 3 times in our system till now.

 We have delivery in near future so that we want it asap. Please try to
 review it internally.

>>>
>>> I don't think your statements justified the reason of urgency as (a) you
>>> have mentioned it to be *rarely* reproducible and (b) I am still waiting
>>> for a real use case where glusterd will go through multiple restarts in a
>>> loop?
>>>
>>>
 Regards,
 Abhishek

 On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
 wrote:

>
>
> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com> wrote:
>
>> + Muthu-vingeshwaran
>>
>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> Hi Atin/Team,
>>>
>>> We are using gluster-3.7.6 with setup of two brick and while restart
>>> of system I have seen that the glusterd daemon is getting failed from 
>>> start.
>>>
>>>
>>> At the time of analyzing the logs from etc-glusterfs...log file
>>> I have received the below logs
>>>
>>>
>>> [2017-05-06 03:33:39.798087] I [MSGID: 100030]
>>> [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
>>> /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
>>> /var/run/glusterd.pid --log-level INFO)
>>> [2017-05-06 03:33:39.807859] I [MSGID: 106478]
>>> [glusterd.c:1350:init] 0-management: Maximum allowed open file 
>>> descriptors
>>> set to 65536
>>> [2017-05-06 03:33:39.807974] I [MSGID: 106479]
>>> [glusterd.c:1399:init] 0-management: Using /system/glusterd as working
>>> directory
>>> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>>> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>>> retrieved op-version: 30706
>>> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>>> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
>>> Failed to get next store iter
>>> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>>> [glusterd-store.c:2844:glusterd_store_retrieve_volume]
>>> 0-management: Failed to update volinfo for c_glusterfs volume
>>> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>>> [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
>>> 0-management: Unable to restore volume: c_glusterfs
>>> [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>>> [xlator.c:428:xlator_init] 0-management: Initialization of volume
>>> 'management' failed, review your volfile again
>>> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>>> 0-management: initializing translator failed
>>> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>>> 0-graph: init failed
>>> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>>> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
>>> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
>>> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
>>> received signum (0), shutting down
>>>
>>
> Abhishek,
>
> This patch needs to be thoroughly reviewed to ensure that it doesn't
> cause any regression given this touches on the core store management
> functionality of glusterd. AFAICT, we get into an empty info file only 
> when
> volume set operation is executed and in parallel one of the glusterd
> instance in other nodes have been brought down and whole sequence of
> operation happens in a loop. The test case through which you can get into
> this situation is not something you'd hit in production. Please help me to
> understand the urgency here.
>
> Also in one of the earlier thread, I did mention the workaround of
> this issue back to Xin through http://lists.gluster.org/piper
> mail/gluster-users/2017-January/029600.html
>
> "If you end up in having a 0 byte info file you'd need to copy the same 
> info file from other node and put it there and restart glusterd"
>
>
>>>
>>> I have found one of the existing 

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

Hi Atin,

Could you please let us know any time plan for deliver of this patch.

Regards,
Abhishek

On Tue, May 9, 2017 at 6:37 PM, ABHISHEK PALIWAL 
wrote:

> Actually it is very risky if it will reproduce in production thats is why
> I said it is on high priority as want to resolve it before production.
>
> On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL > > wrote:
>>
>>> Hi Atin,
>>>
>>> Thanks for your reply.
>>>
>>>
>>> Its urgent because this error is very rarely reproducible we have seen
>>> this 2 3 times in our system till now.
>>>
>>> We have delivery in near future so that we want it asap. Please try to
>>> review it internally.
>>>
>>
>> I don't think your statements justified the reason of urgency as (a) you
>> have mentioned it to be *rarely* reproducible and (b) I am still waiting
>> for a real use case where glusterd will go through multiple restarts in a
>> loop?
>>
>>
>>> Regards,
>>> Abhishek
>>>
>>> On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
>>> wrote:
>>>


 On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> + Muthu-vingeshwaran
>
> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com> wrote:
>
>> Hi Atin/Team,
>>
>> We are using gluster-3.7.6 with setup of two brick and while restart
>> of system I have seen that the glusterd daemon is getting failed from 
>> start.
>>
>>
>> At the time of analyzing the logs from etc-glusterfs...log file I
>> have received the below logs
>>
>>
>> [2017-05-06 03:33:39.798087] I [MSGID: 100030]
>> [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
>> /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
>> /var/run/glusterd.pid --log-level INFO)
>> [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
>> 0-management: Maximum allowed open file descriptors set to 65536
>> [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
>> 0-management: Using /system/glusterd as working directory
>> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>> retrieved op-version: 30706
>> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
>> Failed to get next store iter
>> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>> [glusterd-store.c:2844:glusterd_store_retrieve_volume] 0-management:
>> Failed to update volinfo for c_glusterfs volume
>> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>> [glusterd-store.c:3042:glusterd_store_retrieve_volumes]
>> 0-management: Unable to restore volume: c_glusterfs
>> [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>> [xlator.c:428:xlator_init] 0-management: Initialization of volume
>> 'management' failed, review your volfile again
>> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>> 0-management: initializing translator failed
>> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>> 0-graph: init failed
>> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
>> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
>> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
>> received signum (0), shutting down
>>
>
 Abhishek,

 This patch needs to be thoroughly reviewed to ensure that it doesn't
 cause any regression given this touches on the core store management
 functionality of glusterd. AFAICT, we get into an empty info file only when
 volume set operation is executed and in parallel one of the glusterd
 instance in other nodes have been brought down and whole sequence of
 operation happens in a loop. The test case through which you can get into
 this situation is not something you'd hit in production. Please help me to
 understand the urgency here.

 Also in one of the earlier thread, I did mention the workaround of this
 issue back to Xin through http://lists.gluster.org/piper
 mail/gluster-users/2017-January/029600.html

 "If you end up in having a 0 byte info file you'd need to copy the same 
 info file from other node and put it there and restart glusterd"


>>
>> I have found one of the existing case is there and also solution
>> patch is available but the status of that patch in "cannot merge". Also 
>> the
>> "info" file is empty and "info.tmp" file present in "lib/glusterd/vol"
>> directory.
>>
>> Below is the link of the existing case.
>>

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

Actually it is very risky if it will reproduce in production thats is why I
said it is on high priority as want to resolve it before production.

On Tue, May 9, 2017 at 6:20 PM, Atin Mukherjee  wrote:

>
>
> On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL 
> wrote:
>
>> Hi Atin,
>>
>> Thanks for your reply.
>>
>>
>> Its urgent because this error is very rarely reproducible we have seen
>> this 2 3 times in our system till now.
>>
>> We have delivery in near future so that we want it asap. Please try to
>> review it internally.
>>
>
> I don't think your statements justified the reason of urgency as (a) you
> have mentioned it to be *rarely* reproducible and (b) I am still waiting
> for a real use case where glusterd will go through multiple restarts in a
> loop?
>
>
>> Regards,
>> Abhishek
>>
>> On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
>> wrote:
>>
>>>
>>>
>>> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 + Muthu-vingeshwaran

 On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
 abhishpali...@gmail.com> wrote:

> Hi Atin/Team,
>
> We are using gluster-3.7.6 with setup of two brick and while restart
> of system I have seen that the glusterd daemon is getting failed from 
> start.
>
>
> At the time of analyzing the logs from etc-glusterfs...log file I
> have received the below logs
>
>
> [2017-05-06 03:33:39.798087] I [MSGID: 100030]
> [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running
> /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p
> /var/run/glusterd.pid --log-level INFO)
> [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
> 0-management: Maximum allowed open file descriptors set to 65536
> [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
> 0-management: Using /system/glusterd as working directory
> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
> retrieved op-version: 30706
> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
> Failed to get next store iter
> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
> [glusterd-store.c:2844:glusterd_store_retrieve_volume] 0-management:
> Failed to update volinfo for c_glusterfs volume
> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
> [glusterd-store.c:3042:glusterd_store_retrieve_volumes] 0-management:
> Unable to restore volume: c_glusterfs
> [2017-05-06 03:33:39.827722] E [MSGID: 101019]
> [xlator.c:428:xlator_init] 0-management: Initialization of volume
> 'management' failed, review your volfile again
> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
> 0-graph: init failed
> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
> received signum (0), shutting down
>

>>> Abhishek,
>>>
>>> This patch needs to be thoroughly reviewed to ensure that it doesn't
>>> cause any regression given this touches on the core store management
>>> functionality of glusterd. AFAICT, we get into an empty info file only when
>>> volume set operation is executed and in parallel one of the glusterd
>>> instance in other nodes have been brought down and whole sequence of
>>> operation happens in a loop. The test case through which you can get into
>>> this situation is not something you'd hit in production. Please help me to
>>> understand the urgency here.
>>>
>>> Also in one of the earlier thread, I did mention the workaround of this
>>> issue back to Xin through http://lists.gluster.org/piper
>>> mail/gluster-users/2017-January/029600.html
>>>
>>> "If you end up in having a 0 byte info file you'd need to copy the same 
>>> info file from other node and put it there and restart glusterd"
>>>
>>>
>
> I have found one of the existing case is there and also solution patch
> is available but the status of that patch in "cannot merge". Also the
> "info" file is empty and "info.tmp" file present in "lib/glusterd/vol"
> directory.
>
> Below is the link of the existing case.
>
> https://review.gluster.org/#/c/16279/5
>
> please let me know what is the plan of community to provide the
> solution of this problem and in which version.
>
> Regards
> Abhishek Paliwal
>



 --




 Regards
 Abhishek Paliwal

>>>
>>>
>>

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[Atin Mukherjee]

On Tue, May 9, 2017 at 6:10 PM, ABHISHEK PALIWAL 
wrote:

> Hi Atin,
>
> Thanks for your reply.
>
>
> Its urgent because this error is very rarely reproducible we have seen
> this 2 3 times in our system till now.
>
> We have delivery in near future so that we want it asap. Please try to
> review it internally.
>

I don't think your statements justified the reason of urgency as (a) you
have mentioned it to be *rarely* reproducible and (b) I am still waiting
for a real use case where glusterd will go through multiple restarts in a
loop?


> Regards,
> Abhishek
>
> On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL > > wrote:
>>
>>> + Muthu-vingeshwaran
>>>
>>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
 Hi Atin/Team,

 We are using gluster-3.7.6 with setup of two brick and while restart of
 system I have seen that the glusterd daemon is getting failed from start.


 At the time of analyzing the logs from etc-glusterfs...log file I
 have received the below logs


 [2017-05-06 03:33:39.798087] I [MSGID: 100030] [glusterfsd.c:2348:main]
 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.6
 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
 [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
 0-management: Maximum allowed open file descriptors set to 65536
 [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
 0-management: Using /system/glusterd as working directory
 [2017-05-06 03:33:39.826833] I [MSGID: 106513]
 [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
 retrieved op-version: 30706
 [2017-05-06 03:33:39.827515] E [MSGID: 106206]
 [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
 Failed to get next store iter
 [2017-05-06 03:33:39.827563] E [MSGID: 106207]
 [glusterd-store.c:2844:glusterd_store_retrieve_volume] 0-management:
 Failed to update volinfo for c_glusterfs volume
 [2017-05-06 03:33:39.827625] E [MSGID: 106201]
 [glusterd-store.c:3042:glusterd_store_retrieve_volumes] 0-management:
 Unable to restore volume: c_glusterfs
 [2017-05-06 03:33:39.827722] E [MSGID: 101019]
 [xlator.c:428:xlator_init] 0-management: Initialization of volume
 'management' failed, review your volfile again
 [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
 0-management: initializing translator failed
 [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
 0-graph: init failed
 [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
 (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
 -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
 -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
 received signum (0), shutting down

>>>
>> Abhishek,
>>
>> This patch needs to be thoroughly reviewed to ensure that it doesn't
>> cause any regression given this touches on the core store management
>> functionality of glusterd. AFAICT, we get into an empty info file only when
>> volume set operation is executed and in parallel one of the glusterd
>> instance in other nodes have been brought down and whole sequence of
>> operation happens in a loop. The test case through which you can get into
>> this situation is not something you'd hit in production. Please help me to
>> understand the urgency here.
>>
>> Also in one of the earlier thread, I did mention the workaround of this
>> issue back to Xin through http://lists.gluster.org/piper
>> mail/gluster-users/2017-January/029600.html
>>
>> "If you end up in having a 0 byte info file you'd need to copy the same info 
>> file from other node and put it there and restart glusterd"
>>
>>

 I have found one of the existing case is there and also solution patch
 is available but the status of that patch in "cannot merge". Also the
 "info" file is empty and "info.tmp" file present in "lib/glusterd/vol"
 directory.

 Below is the link of the existing case.

 https://review.gluster.org/#/c/16279/5

 please let me know what is the plan of community to provide the
 solution of this problem and in which version.

 Regards
 Abhishek Paliwal

>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>> Regards
>>> Abhishek Paliwal
>>>
>>
>>
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

Hi Atin,

Thanks for your reply.


Its urgent because this error is very rarely reproducible we have seen this
2 3 times in our system till now.

We have delivery in near future so that we want it asap. Please try to
review it internally.

Regards,
Abhishek

On Tue, May 9, 2017 at 5:58 PM, Atin Mukherjee  wrote:

>
>
> On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL 
> wrote:
>
>> + Muthu-vingeshwaran
>>
>> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <
>> abhishpali...@gmail.com> wrote:
>>
>>> Hi Atin/Team,
>>>
>>> We are using gluster-3.7.6 with setup of two brick and while restart of
>>> system I have seen that the glusterd daemon is getting failed from start.
>>>
>>>
>>> At the time of analyzing the logs from etc-glusterfs...log file I
>>> have received the below logs
>>>
>>>
>>> [2017-05-06 03:33:39.798087] I [MSGID: 100030] [glusterfsd.c:2348:main]
>>> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.6
>>> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
>>> [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
>>> 0-management: Maximum allowed open file descriptors set to 65536
>>> [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
>>> 0-management: Using /system/glusterd as working directory
>>> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>>> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>>> retrieved op-version: 30706
>>> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>>> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
>>> Failed to get next store iter
>>> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>>> [glusterd-store.c:2844:glusterd_store_retrieve_volume] 0-management:
>>> Failed to update volinfo for c_glusterfs volume
>>> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>>> [glusterd-store.c:3042:glusterd_store_retrieve_volumes] 0-management:
>>> Unable to restore volume: c_glusterfs
>>> [2017-05-06 03:33:39.827722] E [MSGID: 101019]
>>> [xlator.c:428:xlator_init] 0-management: Initialization of volume
>>> 'management' failed, review your volfile again
>>> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>>> 0-management: initializing translator failed
>>> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>>> 0-graph: init failed
>>> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>>> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
>>> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
>>> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
>>> received signum (0), shutting down
>>>
>>
> Abhishek,
>
> This patch needs to be thoroughly reviewed to ensure that it doesn't cause
> any regression given this touches on the core store management
> functionality of glusterd. AFAICT, we get into an empty info file only when
> volume set operation is executed and in parallel one of the glusterd
> instance in other nodes have been brought down and whole sequence of
> operation happens in a loop. The test case through which you can get into
> this situation is not something you'd hit in production. Please help me to
> understand the urgency here.
>
> Also in one of the earlier thread, I did mention the workaround of this
> issue back to Xin through http://lists.gluster.org/
> pipermail/gluster-users/2017-January/029600.html
>
> "If you end up in having a 0 byte info file you'd need to copy the same info 
> file from other node and put it there and restart glusterd"
>
>
>>>
>>> I have found one of the existing case is there and also solution patch
>>> is available but the status of that patch in "cannot merge". Also the
>>> "info" file is empty and "info.tmp" file present in "lib/glusterd/vol"
>>> directory.
>>>
>>> Below is the link of the existing case.
>>>
>>> https://review.gluster.org/#/c/16279/5
>>>
>>> please let me know what is the plan of community to provide the solution
>>> of this problem and in which version.
>>>
>>> Regards
>>> Abhishek Paliwal
>>>
>>
>>
>>
>> --
>>
>>
>>
>>
>> Regards
>> Abhishek Paliwal
>>
>
>


-- 




Regards
Abhishek Paliwal
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[Atin Mukherjee]

On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL 
wrote:

> + Muthu-vingeshwaran
>
> On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL  > wrote:
>
>> Hi Atin/Team,
>>
>> We are using gluster-3.7.6 with setup of two brick and while restart of
>> system I have seen that the glusterd daemon is getting failed from start.
>>
>>
>> At the time of analyzing the logs from etc-glusterfs...log file I
>> have received the below logs
>>
>>
>> [2017-05-06 03:33:39.798087] I [MSGID: 100030] [glusterfsd.c:2348:main]
>> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.6
>> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
>> [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
>> 0-management: Maximum allowed open file descriptors set to 65536
>> [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
>> 0-management: Using /system/glusterd as working directory
>> [2017-05-06 03:33:39.826833] I [MSGID: 106513]
>> [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd:
>> retrieved op-version: 30706
>> [2017-05-06 03:33:39.827515] E [MSGID: 106206]
>> [glusterd-store.c:2562:glusterd_store_update_volinfo] 0-management:
>> Failed to get next store iter
>> [2017-05-06 03:33:39.827563] E [MSGID: 106207]
>> [glusterd-store.c:2844:glusterd_store_retrieve_volume] 0-management:
>> Failed to update volinfo for c_glusterfs volume
>> [2017-05-06 03:33:39.827625] E [MSGID: 106201]
>> [glusterd-store.c:3042:glusterd_store_retrieve_volumes] 0-management:
>> Unable to restore volume: c_glusterfs
>> [2017-05-06 03:33:39.827722] E [MSGID: 101019] [xlator.c:428:xlator_init]
>> 0-management: Initialization of volume 'management' failed, review your
>> volfile again
>> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
>> 0-management: initializing translator failed
>> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
>> 0-graph: init failed
>> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
>> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
>> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
>> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
>> received signum (0), shutting down
>>
>
Abhishek,

This patch needs to be thoroughly reviewed to ensure that it doesn't cause
any regression given this touches on the core store management
functionality of glusterd. AFAICT, we get into an empty info file only when
volume set operation is executed and in parallel one of the glusterd
instance in other nodes have been brought down and whole sequence of
operation happens in a loop. The test case through which you can get into
this situation is not something you'd hit in production. Please help me to
understand the urgency here.

Also in one of the earlier thread, I did mention the workaround of this
issue back to Xin through
http://lists.gluster.org/pipermail/gluster-users/2017-January/029600.html

"If you end up in having a 0 byte info file you'd need to copy the
same info file from other node and put it there and restart glusterd"


>>
>> I have found one of the existing case is there and also solution patch is
>> available but the status of that patch in "cannot merge". Also the "info"
>> file is empty and "info.tmp" file present in "lib/glusterd/vol" directory.
>>
>> Below is the link of the existing case.
>>
>> https://review.gluster.org/#/c/16279/5
>>
>> please let me know what is the plan of community to provide the solution
>> of this problem and in which version.
>>
>> Regards
>> Abhishek Paliwal
>>
>
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Empty info file preventing glusterd from starting

[ABHISHEK PALIWAL]

+ Muthu-vingeshwaran

On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL 
wrote:

> Hi Atin/Team,
>
> We are using gluster-3.7.6 with setup of two brick and while restart of
> system I have seen that the glusterd daemon is getting failed from start.
>
>
> At the time of analyzing the logs from etc-glusterfs...log file I have
> received the below logs
>
>
> [2017-05-06 03:33:39.798087] I [MSGID: 100030] [glusterfsd.c:2348:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.6
> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
> [2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init]
> 0-management: Maximum allowed open file descriptors set to 65536
> [2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init]
> 0-management: Using /system/glusterd as working directory
> [2017-05-06 03:33:39.826833] I [MSGID: 106513] 
> [glusterd-store.c:2047:glusterd_restore_op_version]
> 0-glusterd: retrieved op-version: 30706
> [2017-05-06 03:33:39.827515] E [MSGID: 106206] 
> [glusterd-store.c:2562:glusterd_store_update_volinfo]
> 0-management: Failed to get next store iter
> [2017-05-06 03:33:39.827563] E [MSGID: 106207] [glusterd-store.c:2844:
> glusterd_store_retrieve_volume] 0-management: Failed to update volinfo
> for c_glusterfs volume
> [2017-05-06 03:33:39.827625] E [MSGID: 106201] [glusterd-store.c:3042:
> glusterd_store_retrieve_volumes] 0-management: Unable to restore volume:
> c_glusterfs
> [2017-05-06 03:33:39.827722] E [MSGID: 101019] [xlator.c:428:xlator_init]
> 0-management: Initialization of volume 'management' failed, review your
> volfile again
> [2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_activate]
> 0-graph: init failed
> [2017-05-06 03:33:39.828396] W [glusterfsd.c:1238:cleanup_and_exit]
> (-->/usr/sbin/glusterd(glusterfs_volumes_init-0x1b0b8) [0x1000a648]
> -->/usr/sbin/glusterd(glusterfs_process_volfp-0x1b210) [0x1000a4d8]
> -->/usr/sbin/glusterd(cleanup_and_exit-0x1beac) [0x100097ac] ) 0-:
> received signum (0), shutting down
>
>
> I have found one of the existing case is there and also solution patch is
> available but the status of that patch in "cannot merge". Also the "info"
> file is empty and "info.tmp" file present in "lib/glusterd/vol" directory.
>
> Below is the link of the existing case.
>
> https://review.gluster.org/#/c/16279/5
>
> please let me know what is the plan of community to provide the solution
> of this problem and in which version.
>
> Regards
> Abhishek Paliwal
>



-- 




Regards
Abhishek Paliwal
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel