Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[Atin Mukherjee]

On 05/04/2016 06:18 PM, ABHISHEK PALIWAL wrote:
> I am talking about the time taken by the GlusterD to mark the process
> offline because
> here GlusterD is responsible to making brick online/offline.
> 
> is it configurable?
No, there is no such configuration
> 
> On Wed, May 4, 2016 at 5:53 PM, Atin Mukherjee  > wrote:
> 
> Abhishek,
> 
> See the response inline.
> 
> 
> On 05/04/2016 05:43 PM, ABHISHEK PALIWAL wrote:
> > Hi Atin,
> >
> > please reply, is there any configurable time out parameter for brick
> > process to go offline which we can increase?
> >
> > Regards,
> > Abhishek
> >
> > On Thu, Apr 21, 2016 at 12:34 PM, ABHISHEK PALIWAL
> > 
> >>
> wrote:
> >
> > Hi Atin,
> >
> > Please answer following doubts as well:
> >
> > 1 .If there is a temporary glitch in the network , will that affect
> > the gluster brick process in anyway, Is there any timeout for the
> > brick process to go offline in case of the glitch in the network.
>   If there is disconnection, GlusterD will receive it and mark the
> brick as disconnected even if the brick process is online. So answer to
> this question is both yes and no. From process perspective they are
> still up but not to the other components/layers and that may impact the
> operations (both mgmt & I/O given there is a disconnect between client
> and brick processes too)
> >
> > 2. Is there is any configurable time out parameter which we can
> > increase ?
> I don't get this question. What time out are you talking about?
> >
> > 3.Brick and glusterd connected by unix domain socket.It is just a
> > local socket then why it is disconnect in below logs:
>   This is not true, its over TCP socket.
> >
> >  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> > [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
> > Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
> > glusterd.
> >  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> > [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: 
> Setting
> > brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
> >
> > Regards,
> > Abhishek
> >
> >
> > On Tue, Apr 19, 2016 at 1:12 PM, ABHISHEK PALIWAL
> > 
> >>
> wrote:
> >
> > Hi Atin,
> >
> > Thanks.
> >
> > Have more doubts here.
> >
> > Brick and glusterd connected by unix domain socket.It is just a
> > local socket then why it is disconnect in below logs:
> >
> >  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> > [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 
> 0-management:
> > Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected 
> from
> > glusterd.
> >  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> > [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd:
> > Setting
> > brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
> >
> >
> > Regards,
> > Abhishek
> >
> >
> > On Fri, Apr 15, 2016 at 9:14 AM, Atin Mukherjee
> > 
> >> wrote:
> >
> >
> >
> > On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
> > >
> > >
> > > On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee 
> 
> >
> > >     > >
> > >
> > >
> > > On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
> > > >
> > > >
> > > > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee
> 
> >
> >    >>
> > > >  
> >  

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[ABHISHEK PALIWAL]

I am talking about the time taken by the GlusterD to mark the process
offline because
here GlusterD is responsible to making brick online/offline.

is it configurable?

On Wed, May 4, 2016 at 5:53 PM, Atin Mukherjee  wrote:

> Abhishek,
>
> See the response inline.
>
>
> On 05/04/2016 05:43 PM, ABHISHEK PALIWAL wrote:
> > Hi Atin,
> >
> > please reply, is there any configurable time out parameter for brick
> > process to go offline which we can increase?
> >
> > Regards,
> > Abhishek
> >
> > On Thu, Apr 21, 2016 at 12:34 PM, ABHISHEK PALIWAL
> > > wrote:
> >
> > Hi Atin,
> >
> > Please answer following doubts as well:
> >
> > 1 .If there is a temporary glitch in the network , will that affect
> > the gluster brick process in anyway, Is there any timeout for the
> > brick process to go offline in case of the glitch in the network.
>   If there is disconnection, GlusterD will receive it and mark the
> brick as disconnected even if the brick process is online. So answer to
> this question is both yes and no. From process perspective they are
> still up but not to the other components/layers and that may impact the
> operations (both mgmt & I/O given there is a disconnect between client
> and brick processes too)
> >
> > 2. Is there is any configurable time out parameter which we can
> > increase ?
> I don't get this question. What time out are you talking about?
> >
> > 3.Brick and glusterd connected by unix domain socket.It is just a
> > local socket then why it is disconnect in below logs:
>   This is not true, its over TCP socket.
> >
> >  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> > [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
> > Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
> > glusterd.
> >  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> > [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
> > brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
> >
> > Regards,
> > Abhishek
> >
> >
> > On Tue, Apr 19, 2016 at 1:12 PM, ABHISHEK PALIWAL
> > > wrote:
> >
> > Hi Atin,
> >
> > Thanks.
> >
> > Have more doubts here.
> >
> > Brick and glusterd connected by unix domain socket.It is just a
> > local socket then why it is disconnect in below logs:
> >
> >  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> > [glusterd-handler.c:4908:__glusterd_brick_rpc_notify]
> 0-management:
> > Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected
> from
> > glusterd.
> >  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> > [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd:
> > Setting
> > brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
> >
> >
> > Regards,
> > Abhishek
> >
> >
> > On Fri, Apr 15, 2016 at 9:14 AM, Atin Mukherjee
> > > wrote:
> >
> >
> >
> > On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
> > >
> > >
> > > On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee <
> amukh...@redhat.com 
> > > >>
> wrote:
> > >
> > >
> > >
> > > On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
> > > >
> > > >
> > > > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee <
> amukh...@redhat.com 
> > >
> > > >  >   >  > > >
> > > >
> > > >
> > > > On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
> > > > > Hi Team,
> > > > >
> > > > > We are using Gluster 3.7.6 and facing one
> > problem in which
> > > brick is not
> > > > > comming online after restart the board.
> > > > >
> > > > > To understand our setup, please look the
> > following steps:
> > > > > 1. We have two boards A and B on which Gluster
> > volume is
> > > running in
> > > > > replicated mode having one brick on each board.
> > > > > 2. Gluster mount point is present on the Board
> > A which is
> > > sharable
> >

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[Atin Mukherjee]

Abhishek,

See the response inline.


On 05/04/2016 05:43 PM, ABHISHEK PALIWAL wrote:
> Hi Atin,
> 
> please reply, is there any configurable time out parameter for brick
> process to go offline which we can increase?
> 
> Regards,
> Abhishek
> 
> On Thu, Apr 21, 2016 at 12:34 PM, ABHISHEK PALIWAL
> > wrote:
> 
> Hi Atin,
> 
> Please answer following doubts as well:
> 
> 1 .If there is a temporary glitch in the network , will that affect
> the gluster brick process in anyway, Is there any timeout for the
> brick process to go offline in case of the glitch in the network.
  If there is disconnection, GlusterD will receive it and mark the
brick as disconnected even if the brick process is online. So answer to
this question is both yes and no. From process perspective they are
still up but not to the other components/layers and that may impact the
operations (both mgmt & I/O given there is a disconnect between client
and brick processes too)
> 
> 2. Is there is any configurable time out parameter which we can
> increase ?
I don't get this question. What time out are you talking about?
> 
> 3.Brick and glusterd connected by unix domain socket.It is just a
> local socket then why it is disconnect in below logs:
  This is not true, its over TCP socket.
> 
>  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
> Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
> glusterd.
>  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
> brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
> 
> Regards,
> Abhishek
> 
> 
> On Tue, Apr 19, 2016 at 1:12 PM, ABHISHEK PALIWAL
> > wrote:
> 
> Hi Atin,
> 
> Thanks.
> 
> Have more doubts here.
> 
> Brick and glusterd connected by unix domain socket.It is just a
> local socket then why it is disconnect in below logs:
> 
>  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
> Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
> glusterd.
>  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd:
> Setting
> brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
> 
> 
> Regards,
> Abhishek
> 
> 
> On Fri, Apr 15, 2016 at 9:14 AM, Atin Mukherjee
> > wrote:
> 
> 
> 
> On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
> >
> >
> > On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee 
> 
> > >> 
> wrote:
> >
> >
> >
> > On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
> > >
> > >
> > > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee 
> 
> >
> > >     > >
> > >
> > >
> > > On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
> > > > Hi Team,
> > > >
> > > > We are using Gluster 3.7.6 and facing one
> problem in which
> > brick is not
> > > > comming online after restart the board.
> > > >
> > > > To understand our setup, please look the
> following steps:
> > > > 1. We have two boards A and B on which Gluster
> volume is
> > running in
> > > > replicated mode having one brick on each board.
> > > > 2. Gluster mount point is present on the Board
> A which is
> > sharable
> > > > between number of processes.
> > > > 3. Till now our volume is in sync and
> everthing is working fine.
> > > > 4. Now we have test case in which we'll stop
> the glusterd,
> > reboot the
> > > > Board B and when this board comes up, starts
> the glusterd
> > again on it.
> > > > 5. We repeated 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[ABHISHEK PALIWAL]

Hi Atin,

please reply, is there any configurable time out parameter for brick
process to go offline which we can increase?

Regards,
Abhishek

On Thu, Apr 21, 2016 at 12:34 PM, ABHISHEK PALIWAL 
wrote:

> Hi Atin,
>
> Please answer following doubts as well:
>
> 1 .If there is a temporary glitch in the network , will that affect the
> gluster brick process in anyway, Is there any timeout for the brick process
> to go offline in case of the glitch in the network.
>
> 2. Is there is any configurable time out parameter which we can increase ?
>
> 3.Brick and glusterd connected by unix domain socket.It is just a local
> socket then why it is disconnect in below logs:
>
>  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
> Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
> glusterd.
>  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
> brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
>
> Regards,
> Abhishek
>
>
> On Tue, Apr 19, 2016 at 1:12 PM, ABHISHEK PALIWAL  > wrote:
>
>> Hi Atin,
>>
>> Thanks.
>>
>> Have more doubts here.
>>
>> Brick and glusterd connected by unix domain socket.It is just a local
>> socket then why it is disconnect in below logs:
>>
>>  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
>> [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
>> Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
>> glusterd.
>>  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
>> [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
>> brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
>>
>>
>> Regards,
>> Abhishek
>>
>>
>> On Fri, Apr 15, 2016 at 9:14 AM, Atin Mukherjee 
>> wrote:
>>
>>>
>>>
>>> On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
>>> >
>>> >
>>> > On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee >> > > wrote:
>>> >
>>> >
>>> >
>>> > On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
>>> > >
>>> > >
>>> > > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee <
>>> amukh...@redhat.com 
>>> > > >>
>>> wrote:
>>> > >
>>> > >
>>> > >
>>> > > On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
>>> > > > Hi Team,
>>> > > >
>>> > > > We are using Gluster 3.7.6 and facing one problem in which
>>> > brick is not
>>> > > > comming online after restart the board.
>>> > > >
>>> > > > To understand our setup, please look the following steps:
>>> > > > 1. We have two boards A and B on which Gluster volume is
>>> > running in
>>> > > > replicated mode having one brick on each board.
>>> > > > 2. Gluster mount point is present on the Board A which is
>>> > sharable
>>> > > > between number of processes.
>>> > > > 3. Till now our volume is in sync and everthing is working
>>> fine.
>>> > > > 4. Now we have test case in which we'll stop the glusterd,
>>> > reboot the
>>> > > > Board B and when this board comes up, starts the glusterd
>>> > again on it.
>>> > > > 5. We repeated Steps 4 multiple times to check the
>>> > reliability of system.
>>> > > > 6. After the Step 4, sometimes system comes in working
>>> state
>>> > (i.e. in
>>> > > > sync) but sometime we faces that brick of Board B is
>>> present in
>>> > > > “gluster volume status” command but not be online even
>>> > waiting for
>>> > > > more than a minute.
>>> > > As I mentioned in another email thread until and unless the
>>> > log shows
>>> > > the evidence that there was a reboot nothing can be
>>> concluded.
>>> > The last
>>> > > log what you shared with us few days back didn't give any
>>> > indication
>>> > > that brick process wasn't running.
>>> > >
>>> > > How can we identify that the brick process is running in brick
>>> logs?
>>> > >
>>> > > > 7. When the Step 4 is executing at the same time on Board
>>> A some
>>> > > > processes are started accessing the files from the Gluster
>>> > mount point.
>>> > > >
>>> > > > As a solution to make this brick online, we found some
>>> > existing issues
>>> > > > in gluster mailing list giving suggestion to use “gluster
>>> > volume start
>>> > > >  force” to make the brick 'offline' to 'online'.
>>> > > >
>>> > > > If we use “gluster volume start  force” command.
>>> > It will kill
>>> > > > the existing volume process and started the new process
>>> then
>>> > what will
>>> > > > happen if other processes 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[ABHISHEK PALIWAL]

Hi Atin,

Please answer following doubts as well:

1 .If there is a temporary glitch in the network , will that affect the
gluster brick process in anyway, Is there any timeout for the brick process
to go offline in case of the glitch in the network.

2. Is there is any configurable time out parameter which we can increase ?

3.Brick and glusterd connected by unix domain socket.It is just a local
socket then why it is disconnect in below logs:

 1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
[glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
glusterd.
 1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
[glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped

Regards,
Abhishek


On Tue, Apr 19, 2016 at 1:12 PM, ABHISHEK PALIWAL 
wrote:

> Hi Atin,
>
> Thanks.
>
> Have more doubts here.
>
> Brick and glusterd connected by unix domain socket.It is just a local
> socket then why it is disconnect in below logs:
>
>  1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
> [glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
> Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
> glusterd.
>  1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
> [glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
> brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped
>
>
> Regards,
> Abhishek
>
>
> On Fri, Apr 15, 2016 at 9:14 AM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
>> >
>> >
>> > On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee > > > wrote:
>> >
>> >
>> >
>> > On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
>> > >
>> > >
>> > > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee <
>> amukh...@redhat.com 
>> > > >> wrote:
>> > >
>> > >
>> > >
>> > > On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
>> > > > Hi Team,
>> > > >
>> > > > We are using Gluster 3.7.6 and facing one problem in which
>> > brick is not
>> > > > comming online after restart the board.
>> > > >
>> > > > To understand our setup, please look the following steps:
>> > > > 1. We have two boards A and B on which Gluster volume is
>> > running in
>> > > > replicated mode having one brick on each board.
>> > > > 2. Gluster mount point is present on the Board A which is
>> > sharable
>> > > > between number of processes.
>> > > > 3. Till now our volume is in sync and everthing is working
>> fine.
>> > > > 4. Now we have test case in which we'll stop the glusterd,
>> > reboot the
>> > > > Board B and when this board comes up, starts the glusterd
>> > again on it.
>> > > > 5. We repeated Steps 4 multiple times to check the
>> > reliability of system.
>> > > > 6. After the Step 4, sometimes system comes in working state
>> > (i.e. in
>> > > > sync) but sometime we faces that brick of Board B is
>> present in
>> > > > “gluster volume status” command but not be online even
>> > waiting for
>> > > > more than a minute.
>> > > As I mentioned in another email thread until and unless the
>> > log shows
>> > > the evidence that there was a reboot nothing can be concluded.
>> > The last
>> > > log what you shared with us few days back didn't give any
>> > indication
>> > > that brick process wasn't running.
>> > >
>> > > How can we identify that the brick process is running in brick
>> logs?
>> > >
>> > > > 7. When the Step 4 is executing at the same time on Board A
>> some
>> > > > processes are started accessing the files from the Gluster
>> > mount point.
>> > > >
>> > > > As a solution to make this brick online, we found some
>> > existing issues
>> > > > in gluster mailing list giving suggestion to use “gluster
>> > volume start
>> > > >  force” to make the brick 'offline' to 'online'.
>> > > >
>> > > > If we use “gluster volume start  force” command.
>> > It will kill
>> > > > the existing volume process and started the new process then
>> > what will
>> > > > happen if other processes are accessing the same volume at
>> > the time when
>> > > > volume process is killed by this command internally. Will it
>> > impact any
>> > > > failure on these processes?
>> > > This is not true, volume start force will start the brick
>> > processes only
>> > > if they are not running. Running brick processes will not be
>> > > 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[ABHISHEK PALIWAL]

Hi Atin,

Thanks.

Have more doubts here.

Brick and glusterd connected by unix domain socket.It is just a local
socket then why it is disconnect in below logs:

 1667 [2016-04-03 10:12:32.984331] I [MSGID: 106005]
[glusterd-handler.c:4908:__glusterd_brick_rpc_notify] 0-management:
Brick 10.32.   1.144:/opt/lvmdir/c2/brick has disconnected from
glusterd.
 1668 [2016-04-03 10:12:32.984366] D [MSGID: 0]
[glusterd-utils.c:4872:glusterd_set_brick_status] 0-glusterd: Setting
brick 10.32.1.144:/opt/lvmdir/c2/brick status to stopped


Regards,
Abhishek

On Fri, Apr 15, 2016 at 9:14 AM, Atin Mukherjee  wrote:

>
>
> On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
> >
> >
> > On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee  > > wrote:
> >
> >
> >
> > On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
> > >
> > >
> > > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee <
> amukh...@redhat.com 
> > > >> wrote:
> > >
> > >
> > >
> > > On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
> > > > Hi Team,
> > > >
> > > > We are using Gluster 3.7.6 and facing one problem in which
> > brick is not
> > > > comming online after restart the board.
> > > >
> > > > To understand our setup, please look the following steps:
> > > > 1. We have two boards A and B on which Gluster volume is
> > running in
> > > > replicated mode having one brick on each board.
> > > > 2. Gluster mount point is present on the Board A which is
> > sharable
> > > > between number of processes.
> > > > 3. Till now our volume is in sync and everthing is working
> fine.
> > > > 4. Now we have test case in which we'll stop the glusterd,
> > reboot the
> > > > Board B and when this board comes up, starts the glusterd
> > again on it.
> > > > 5. We repeated Steps 4 multiple times to check the
> > reliability of system.
> > > > 6. After the Step 4, sometimes system comes in working state
> > (i.e. in
> > > > sync) but sometime we faces that brick of Board B is present
> in
> > > > “gluster volume status” command but not be online even
> > waiting for
> > > > more than a minute.
> > > As I mentioned in another email thread until and unless the
> > log shows
> > > the evidence that there was a reboot nothing can be concluded.
> > The last
> > > log what you shared with us few days back didn't give any
> > indication
> > > that brick process wasn't running.
> > >
> > > How can we identify that the brick process is running in brick
> logs?
> > >
> > > > 7. When the Step 4 is executing at the same time on Board A
> some
> > > > processes are started accessing the files from the Gluster
> > mount point.
> > > >
> > > > As a solution to make this brick online, we found some
> > existing issues
> > > > in gluster mailing list giving suggestion to use “gluster
> > volume start
> > > >  force” to make the brick 'offline' to 'online'.
> > > >
> > > > If we use “gluster volume start  force” command.
> > It will kill
> > > > the existing volume process and started the new process then
> > what will
> > > > happen if other processes are accessing the same volume at
> > the time when
> > > > volume process is killed by this command internally. Will it
> > impact any
> > > > failure on these processes?
> > > This is not true, volume start force will start the brick
> > processes only
> > > if they are not running. Running brick processes will not be
> > > interrupted.
> > >
> > > we have tried and check the pid of process before force start and
> > after
> > > force start.
> > > the pid has been changed after force start.
> > >
> > > Please find the logs at the time of failure attached once again
> with
> > > log-level=debug.
> > >
> > > if you can give me the exact line where you are able to find out
> that
> > > the brick process
> > > is running in brick log file please give me the line number of
> > that file.
> >
> > Here is the sequence at which glusterd and respective brick process
> is
> > restarted.
> >
> > 1. glusterd restart trigger - line number 1014 in glusterd.log file:
> >
> > [2016-04-03 10:12:29.051735] I [MSGID: 100030]
> [glusterfsd.c:2318:main]
> > 0-/usr/sbin/glusterd: Started running /usr/sbin/
> glusterd
> > version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid
> > --log-level DEBUG)
> >
> > 2. brick start trigger - line number 190 in opt-lvmdir-c2-brick.log
> >
> > [2016-04-03 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[Atin Mukherjee]

On 04/14/2016 04:07 PM, ABHISHEK PALIWAL wrote:
> 
> 
> On Thu, Apr 14, 2016 at 2:33 PM, Atin Mukherjee  > wrote:
> 
> 
> 
> On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
> >
> >
> > On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee  
> > >> wrote:
> >
> >
> >
> > On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
> > > Hi Team,
> > >
> > > We are using Gluster 3.7.6 and facing one problem in which
> brick is not
> > > comming online after restart the board.
> > >
> > > To understand our setup, please look the following steps:
> > > 1. We have two boards A and B on which Gluster volume is
> running in
> > > replicated mode having one brick on each board.
> > > 2. Gluster mount point is present on the Board A which is
> sharable
> > > between number of processes.
> > > 3. Till now our volume is in sync and everthing is working fine.
> > > 4. Now we have test case in which we'll stop the glusterd,
> reboot the
> > > Board B and when this board comes up, starts the glusterd
> again on it.
> > > 5. We repeated Steps 4 multiple times to check the
> reliability of system.
> > > 6. After the Step 4, sometimes system comes in working state
> (i.e. in
> > > sync) but sometime we faces that brick of Board B is present in
> > > “gluster volume status” command but not be online even
> waiting for
> > > more than a minute.
> > As I mentioned in another email thread until and unless the
> log shows
> > the evidence that there was a reboot nothing can be concluded.
> The last
> > log what you shared with us few days back didn't give any
> indication
> > that brick process wasn't running.
> >
> > How can we identify that the brick process is running in brick logs?
> >
> > > 7. When the Step 4 is executing at the same time on Board A some
> > > processes are started accessing the files from the Gluster
> mount point.
> > >
> > > As a solution to make this brick online, we found some
> existing issues
> > > in gluster mailing list giving suggestion to use “gluster
> volume start
> > >  force” to make the brick 'offline' to 'online'.
> > >
> > > If we use “gluster volume start  force” command.
> It will kill
> > > the existing volume process and started the new process then
> what will
> > > happen if other processes are accessing the same volume at
> the time when
> > > volume process is killed by this command internally. Will it
> impact any
> > > failure on these processes?
> > This is not true, volume start force will start the brick
> processes only
> > if they are not running. Running brick processes will not be
> > interrupted.
> >
> > we have tried and check the pid of process before force start and
> after
> > force start.
> > the pid has been changed after force start.
> >
> > Please find the logs at the time of failure attached once again with
> > log-level=debug.
> >
> > if you can give me the exact line where you are able to find out that
> > the brick process
> > is running in brick log file please give me the line number of
> that file.
> 
> Here is the sequence at which glusterd and respective brick process is
> restarted.
> 
> 1. glusterd restart trigger - line number 1014 in glusterd.log file:
> 
> [2016-04-03 10:12:29.051735] I [MSGID: 100030] [glusterfsd.c:2318:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/  glusterd
> version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid
> --log-level DEBUG)
> 
> 2. brick start trigger - line number 190 in opt-lvmdir-c2-brick.log
> 
> [2016-04-03 10:14:25.268833] I [MSGID: 100030] [glusterfsd.c:2318:main]
> 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd
> version 3.7.6 (args: /usr/sbin/glusterfsd -s 10.32.1.144 --volfile-id
> c_glusterfs.10.32.1.144.opt-lvmdir-c2-brick -p /
> system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid
> -S /var/run/gluster/697c0e4a16ebc734cd06fd9150723005.socket
> --brick-name /opt/lvmdir/c2/brick -l
> /var/log/glusterfs/bricks/opt-lvmdir-c2-brick.log --xlator-option
> *-posix.glusterd-   uuid=2d576ff8-0cea-4f75-9e34-a5674fbf7256
> --brick-port 49329 --xlator-option c_glusterfs-server.listen-port=49329)
> 
> 3. The following log indicates that brick is up and is now started.
> Refer to line 16123 in glusterd.log
> 
> [2016-04-03 10:14:25.336855] D 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[Atin Mukherjee]

On 04/05/2016 03:35 PM, ABHISHEK PALIWAL wrote:
> 
> 
> On Tue, Apr 5, 2016 at 2:22 PM, Atin Mukherjee  > wrote:
> 
> 
> 
> On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
> > Hi Team,
> >
> > We are using Gluster 3.7.6 and facing one problem in which brick is not
> > comming online after restart the board.
> >
> > To understand our setup, please look the following steps:
> > 1. We have two boards A and B on which Gluster volume is running in
> > replicated mode having one brick on each board.
> > 2. Gluster mount point is present on the Board A which is sharable
> > between number of processes.
> > 3. Till now our volume is in sync and everthing is working fine.
> > 4. Now we have test case in which we'll stop the glusterd, reboot the
> > Board B and when this board comes up, starts the glusterd again on it.
> > 5. We repeated Steps 4 multiple times to check the reliability of 
> system.
> > 6. After the Step 4, sometimes system comes in working state (i.e. in
> > sync) but sometime we faces that brick of Board B is present in
> > “gluster volume status” command but not be online even waiting for
> > more than a minute.
> As I mentioned in another email thread until and unless the log shows
> the evidence that there was a reboot nothing can be concluded. The last
> log what you shared with us few days back didn't give any indication
> that brick process wasn't running.
> 
> How can we identify that the brick process is running in brick logs?
> 
> > 7. When the Step 4 is executing at the same time on Board A some
> > processes are started accessing the files from the Gluster mount point.
> >
> > As a solution to make this brick online, we found some existing issues
> > in gluster mailing list giving suggestion to use “gluster volume start
> >  force” to make the brick 'offline' to 'online'.
> >
> > If we use “gluster volume start  force” command. It will kill
> > the existing volume process and started the new process then what will
> > happen if other processes are accessing the same volume at the time when
> > volume process is killed by this command internally. Will it impact any
> > failure on these processes?
> This is not true, volume start force will start the brick processes only
> if they are not running. Running brick processes will not be
> interrupted.
> 
> we have tried and check the pid of process before force start and after
> force start.
> the pid has been changed after force start.
> 
> Please find the logs at the time of failure attached once again with
> log-level=debug.
> 
> if you can give me the exact line where you are able to find out that
> the brick process
> is running in brick log file please give me the line number of that file.

Here is the sequence at which glusterd and respective brick process is
restarted.

1. glusterd restart trigger - line number 1014 in glusterd.log file:

[2016-04-03 10:12:29.051735] I [MSGID: 100030] [glusterfsd.c:2318:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/  glusterd
version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid
--log-level DEBUG)

2. brick start trigger - line number 190 in opt-lvmdir-c2-brick.log

[2016-04-03 10:14:25.268833] I [MSGID: 100030] [glusterfsd.c:2318:main]
0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd
version 3.7.6 (args: /usr/sbin/glusterfsd -s 10.32.1.144 --volfile-id
c_glusterfs.10.32.1.144.opt-lvmdir-c2-brick -p /
system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid
-S /var/run/gluster/697c0e4a16ebc734cd06fd9150723005.socket
--brick-name /opt/lvmdir/c2/brick -l
/var/log/glusterfs/bricks/opt-lvmdir-c2-brick.log --xlator-option
*-posix.glusterd-   uuid=2d576ff8-0cea-4f75-9e34-a5674fbf7256
--brick-port 49329 --xlator-option c_glusterfs-server.listen-port=49329)

3. The following log indicates that brick is up and is now started.
Refer to line 16123 in glusterd.log

[2016-04-03 10:14:25.336855] D [MSGID: 0]
[glusterd-handler.c:4897:__glusterd_brick_rpc_notify] 0-management:
Connected to 10.32.1.144:/opt/lvmdir/c2/brick

This clearly indicates that the brick is up and running as after that I
do not see any disconnect event been processed by glusterd for the brick
process.

Please note that all the logs referred and pasted are from 002500.

~Atin
> 
> 002500 - Board B that brick is offline
> 00300 - Board A logs
> 
> >
> > *Question : What could be contributing to brick offline?*
> >
> >
> > --
> >
> > Regards
> > Abhishek Paliwal
> >
> >
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org 
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> >
> 
> 
> 
> 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[Atin Mukherjee]

Abhishek,

I will go through the attachments you sent last time and get back.
Apologies for missing that out.

~Atin

On 04/14/2016 07:13 AM, ABHISHEK PALIWAL wrote:
> Hi,
> 
> I have still facing this problem and analyzing the system as well as
> gluster logs.
> 
> From the system logs /var/log/messages I found that time stamp of
> temporary n/w failure logs is very near and between to “volume status”
> command and before the remove-brick command where our brick is offline.
> 
> Please suggest me based on below logs is this could be possible reason
> for and stopping Brick to come online?
> 
> we have collect logs from two setup
> 
> Setup 1:
> 
> 
> 002500> cat /var/log/messages | grep -r "name lookup failed for"
> Apr  8 06:52:08 2016 oamhost daemon.info 
> rsyncd[2361]: name lookup failed for 10.32.0.48 :
> Temporary failure in name resolution
> Apr  8 06:52:09 2016 oamhost daemon.info 
> rsyncd[2639]: name lookup failed for 10.32.0.48 :
> Temporary failure in name resolution
> Apr  8 06:52:09 2016 oamhost daemon.info 
> rsyncd[2657]: name lookup failed for 10.32.0.48 :
> Temporary failure in name resolution
> Apr  8 06:52:09 2016 oamhost daemon.info 
> rsyncd[2677]: name lookup failed for 10.32.0.48 :
> Temporary failure in name resolution
> Apr  8 06:52:10 2016 oamhost daemon.info 
> rsyncd[2878]: name lookup failed for 10.32.0.48 :
> Temporary failure in name resolution
>  
> # cat /var/log/glusterfs/cmd_history.log | grep -r "SUCCESS"
> [2016-04-08 06:52:13.687779]  : volume status : SUCCESS
> [2016-04-08 06:52:13.703139]  : volume status : SUCCESS
> [2016-04-08 06:52:14.041347]  : volume remove-brick c_glusterfs replica
> 1 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> [2016-04-08 06:52:14.135737]  : peer detach 10.32.1.144 : SUCCESS
> [2016-04-08 06:52:15.415575]  : peer probe 10.32.1.144 : SUCCESS
> [2016-04-08 06:52:17.941650]  : volume add-brick c_glusterfs replica 2
> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> 
> 
> Setup 2:
> 
> # cat /var/log/glusterfs/cmd_history.log | grep -r "SUCCESS"
> [2016-04-07 23:48:00.544269]  : volume status : SUCCESS
> [2016-04-07 23:48:00.564007]  : volume status : SUCCESS
> [2016-04-07 23:48:01.624280]  : volume status : SUCCESS
> [2016-04-07 23:48:01.642542]  : volume status : SUCCESS
> [2016-04-07 23:48:02.699085]  : volume status : SUCCESS
> [2016-04-07 23:48:02.716108]  : volume status : SUCCESS
> [2016-04-07 23:48:03.782118]  : volume status : SUCCESS
> [2016-04-07 23:48:03.832059]  : volume status : SUCCESS
> [2016-04-07 23:56:22.709733]  : volume set c_glusterfs nfs.disable on :
> SUCCESS
> [2016-04-07 23:56:24.037270]  : volume start c_glusterfs force : SUCCESS
> [2016-04-07 23:56:34.326782]  : volume status : SUCCESS
> [2016-04-07 23:56:34.352975]  : volume status : SUCCESS
> [2016-04-07 23:56:35.441763]  : volume status : SUCCESS
> [2016-04-07 23:56:35.467474]  : volume status : SUCCESS
> [2016-04-07 23:56:36.544532]  : volume status : SUCCESS
> [2016-04-07 23:56:36.563667]  : volume status : SUCCESS
> [2016-04-07 23:56:37.633660]  : volume status : SUCCESS
> [2016-04-07 23:56:37.653251]  : volume status : SUCCESS
> [2016-04-07 23:56:38.726406]  : volume status : SUCCESS
> [2016-04-07 23:56:38.746097]  : volume status : SUCCESS
> [2016-04-07 23:56:39.805968]  : volume status : SUCCESS
> [2016-04-07 23:56:39.824601]  : volume status : SUCCESS
> [2016-04-07 23:56:40.886599]  : volume status : SUCCESS
> [2016-04-07 23:56:40.905728]  : volume status : SUCCESS
> [2016-04-07 23:56:41.963659]  : volume status : SUCCESS
> [2016-04-07 23:56:41.980006]  : volume status : SUCCESS
> [2016-04-07 23:56:43.037351]  : volume status : SUCCESS
> [2016-04-07 23:56:43.053252]  : volume status : SUCCESS
> [2016-04-07 23:56:44.112666]  : volume status : SUCCESS
> [2016-04-07 23:56:44.129299]  : volume status : SUCCESS
> [2016-04-07 23:56:45.186047]  : volume status : SUCCESS
> [2016-04-07 23:56:45.201741]  : volume status : SUCCESS
> [2016-04-07 23:56:46.259575]  : volume status : SUCCESS
> [2016-04-07 23:56:46.274823]  : volume status : SUCCESS
> [2016-04-07 23:56:47.332110]  : volume status : SUCCESS
> [2016-04-07 23:56:47.347350]  : volume status : SUCCESS
> [2016-04-07 23:56:48.408752]  : volume status : SUCCESS
> [2016-04-07 23:56:48.424115]  : volume status : SUCCESS
> [2016-04-07 23:56:49.481196]  : volume status : SUCCESS
> [2016-04-07 23:56:49.496417]  : volume status : SUCCESS
> [2016-04-07 23:56:50.556596]  : volume status : SUCCESS
> [2016-04-07 23:56:50.572201]  : volume status : SUCCESS
> [2016-04-07 23:56:51.645649]  : volume status : SUCCESS
> [2016-04-07 23:56:51.660980]  : volume status : SUCCESS
> [2016-04-07 23:56:52.718777]  : volume status : SUCCESS
> [2016-04-07 23:56:52.734715]  : volume status : SUCCESS
> [2016-04-07 23:56:53.798818]  

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[ABHISHEK PALIWAL]

Hi,

I have still facing this problem and analyzing the system as well as
gluster logs.

>From the system logs /var/log/messages I found that time stamp of temporary
n/w failure logs is very near and between to “volume status” command and
before the remove-brick command where our brick is offline.

Please suggest me based on below logs is this could be possible reason for
and stopping Brick to come online?

we have collect logs from two setup

Setup 1:


002500> cat /var/log/messages | grep -r "name lookup failed for"
Apr  8 06:52:08 2016 oamhost daemon.info rsyncd[2361]: name lookup failed
for 10.32.0.48: Temporary failure in name resolution
Apr  8 06:52:09 2016 oamhost daemon.info rsyncd[2639]: name lookup failed
for 10.32.0.48: Temporary failure in name resolution
Apr  8 06:52:09 2016 oamhost daemon.info rsyncd[2657]: name lookup failed
for 10.32.0.48: Temporary failure in name resolution
Apr  8 06:52:09 2016 oamhost daemon.info rsyncd[2677]: name lookup failed
for 10.32.0.48: Temporary failure in name resolution
Apr  8 06:52:10 2016 oamhost daemon.info rsyncd[2878]: name lookup failed
for 10.32.0.48: Temporary failure in name resolution

# cat /var/log/glusterfs/cmd_history.log | grep -r "SUCCESS"
[2016-04-08 06:52:13.687779]  : volume status : SUCCESS
[2016-04-08 06:52:13.703139]  : volume status : SUCCESS
[2016-04-08 06:52:14.041347]  : volume remove-brick c_glusterfs replica 1
10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
[2016-04-08 06:52:14.135737]  : peer detach 10.32.1.144 : SUCCESS
[2016-04-08 06:52:15.415575]  : peer probe 10.32.1.144 : SUCCESS
[2016-04-08 06:52:17.941650]  : volume add-brick c_glusterfs replica 2
10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS


Setup 2:

# cat /var/log/glusterfs/cmd_history.log | grep -r "SUCCESS"
[2016-04-07 23:48:00.544269]  : volume status : SUCCESS
[2016-04-07 23:48:00.564007]  : volume status : SUCCESS
[2016-04-07 23:48:01.624280]  : volume status : SUCCESS
[2016-04-07 23:48:01.642542]  : volume status : SUCCESS
[2016-04-07 23:48:02.699085]  : volume status : SUCCESS
[2016-04-07 23:48:02.716108]  : volume status : SUCCESS
[2016-04-07 23:48:03.782118]  : volume status : SUCCESS
[2016-04-07 23:48:03.832059]  : volume status : SUCCESS
[2016-04-07 23:56:22.709733]  : volume set c_glusterfs nfs.disable on :
SUCCESS
[2016-04-07 23:56:24.037270]  : volume start c_glusterfs force : SUCCESS
[2016-04-07 23:56:34.326782]  : volume status : SUCCESS
[2016-04-07 23:56:34.352975]  : volume status : SUCCESS
[2016-04-07 23:56:35.441763]  : volume status : SUCCESS
[2016-04-07 23:56:35.467474]  : volume status : SUCCESS
[2016-04-07 23:56:36.544532]  : volume status : SUCCESS
[2016-04-07 23:56:36.563667]  : volume status : SUCCESS
[2016-04-07 23:56:37.633660]  : volume status : SUCCESS
[2016-04-07 23:56:37.653251]  : volume status : SUCCESS
[2016-04-07 23:56:38.726406]  : volume status : SUCCESS
[2016-04-07 23:56:38.746097]  : volume status : SUCCESS
[2016-04-07 23:56:39.805968]  : volume status : SUCCESS
[2016-04-07 23:56:39.824601]  : volume status : SUCCESS
[2016-04-07 23:56:40.886599]  : volume status : SUCCESS
[2016-04-07 23:56:40.905728]  : volume status : SUCCESS
[2016-04-07 23:56:41.963659]  : volume status : SUCCESS
[2016-04-07 23:56:41.980006]  : volume status : SUCCESS
[2016-04-07 23:56:43.037351]  : volume status : SUCCESS
[2016-04-07 23:56:43.053252]  : volume status : SUCCESS
[2016-04-07 23:56:44.112666]  : volume status : SUCCESS
[2016-04-07 23:56:44.129299]  : volume status : SUCCESS
[2016-04-07 23:56:45.186047]  : volume status : SUCCESS
[2016-04-07 23:56:45.201741]  : volume status : SUCCESS
[2016-04-07 23:56:46.259575]  : volume status : SUCCESS
[2016-04-07 23:56:46.274823]  : volume status : SUCCESS
[2016-04-07 23:56:47.332110]  : volume status : SUCCESS
[2016-04-07 23:56:47.347350]  : volume status : SUCCESS
[2016-04-07 23:56:48.408752]  : volume status : SUCCESS
[2016-04-07 23:56:48.424115]  : volume status : SUCCESS
[2016-04-07 23:56:49.481196]  : volume status : SUCCESS
[2016-04-07 23:56:49.496417]  : volume status : SUCCESS
[2016-04-07 23:56:50.556596]  : volume status : SUCCESS
[2016-04-07 23:56:50.572201]  : volume status : SUCCESS
[2016-04-07 23:56:51.645649]  : volume status : SUCCESS
[2016-04-07 23:56:51.660980]  : volume status : SUCCESS
[2016-04-07 23:56:52.718777]  : volume status : SUCCESS
[2016-04-07 23:56:52.734715]  : volume status : SUCCESS
[2016-04-07 23:56:53.798818]  : volume status : SUCCESS
[2016-04-07 23:56:53.814556]  : volume status : SUCCESS
[2016-04-07 23:56:54.873792]  : volume status : SUCCESS
[2016-04-07 23:56:54.889273]  : volume status : SUCCESS
[2016-04-07 23:56:55.949193]  : volume status : SUCCESS
[2016-04-07 23:56:55.964678]  : volume status : SUCCESS
[2016-04-07 23:56:57.026010]  : volume status : SUCCESS
[2016-04-07 23:56:57.042384]  : volume status : SUCCESS
[2016-04-07 23:56:58.104969]  : volume status : SUCCESS
[2016-04-07 23:56:58.121515]  : volume status : SUCCESS
[2016-04-07 23:56:59.177334]  : volume 

Re: [Gluster-devel] Gluster Brick Offline after reboot!!

[Atin Mukherjee]

On 04/05/2016 01:04 PM, ABHISHEK PALIWAL wrote:
> Hi Team,
> 
> We are using Gluster 3.7.6 and facing one problem in which brick is not
> comming online after restart the board.
> 
> To understand our setup, please look the following steps:
> 1. We have two boards A and B on which Gluster volume is running in
> replicated mode having one brick on each board.
> 2. Gluster mount point is present on the Board A which is sharable
> between number of processes.
> 3. Till now our volume is in sync and everthing is working fine.
> 4. Now we have test case in which we'll stop the glusterd, reboot the
> Board B and when this board comes up, starts the glusterd again on it.
> 5. We repeated Steps 4 multiple times to check the reliability of system.
> 6. After the Step 4, sometimes system comes in working state (i.e. in
> sync) but sometime we faces that brick of Board B is present in
> “gluster volume status” command but not be online even waiting for
> more than a minute.
As I mentioned in another email thread until and unless the log shows
the evidence that there was a reboot nothing can be concluded. The last
log what you shared with us few days back didn't give any indication
that brick process wasn't running.
> 7. When the Step 4 is executing at the same time on Board A some
> processes are started accessing the files from the Gluster mount point.
> 
> As a solution to make this brick online, we found some existing issues
> in gluster mailing list giving suggestion to use “gluster volume start
>  force” to make the brick 'offline' to 'online'.
> 
> If we use “gluster volume start  force” command. It will kill
> the existing volume process and started the new process then what will
> happen if other processes are accessing the same volume at the time when
> volume process is killed by this command internally. Will it impact any
> failure on these processes?
This is not true, volume start force will start the brick processes only
if they are not running. Running brick processes will not be interrupted.
> 
> *Question : What could be contributing to brick offline?*
> 
> 
> -- 
> 
> Regards
> Abhishek Paliwal
> 
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel