Hi Atin, Please tell me the line number where you are seeing that glusterd has restored value from the disk files in Board B file.
Regards, Abhishek On Tue, Mar 15, 2016 at 11:31 AM, ABHISHEK PALIWAL <abhishpali...@gmail.com> wrote: > > > On Tue, Mar 15, 2016 at 11:10 AM, Atin Mukherjee <amukh...@redhat.com> > wrote: > >> >> >> On 03/15/2016 10:54 AM, ABHISHEK PALIWAL wrote: >> > Hi Atin, >> > >> > Is these files are ok? or you need some other files. >> I just started going through the log files you shared. I've few >> questions for you looking at the log: >> 1. Are you sure the log what you have provided from board B is post a >> reboot? If you claim that a reboot wipes of /var/lib/glusterd/ then why >> am I seeing that glusterd has restored value from the disk files? >> > > Yes these logs from Board B after reboot. Could you please explain me the > line number where you are seeing that glusterd has restored value from the > disk files. > >> >> 2. From the content of glusterd configurations which you shared earlier >> the peer UUIDs are 4bf982c0-b21b-415c-b870-e72f36c7f2e7, >> 4bf982c0-b21b-415c-b870-e72f36c7f2e7 002500/glusterd/peers & >> c6b64e36-76da-4e98-a616-48e0e52c7006 from 000300/glusterd/peers. They >> don't even exist in glusterd.log. >> >> Somehow I have a feeling that the sequence of log and configurations >> files you shared don't match! >> > > There is two UUID file present in 002500/glusterd/peers > 1. 4bf982c0-b21b-415c-b870-e72f36c7f2e7 > Content of this file is: > uuid=4bf982c0-b21b-415c-b870-e72f36c7f2e7 > state=10 > hostname1=10.32.0.48 > I have a question from where this UUID is coming? > > 2. 98a28041-f853-48ac-bee0-34c592eeb827 > Content of this file is: > uuid=f4ebe3c5-b6a4-4795-98e0-732337f76faf //This uuid is belogs to > 000300(10.32.0.48) board you can check this in both of the glusterd log file > state=4 //what this state field display in this file? > hostname1=10.32.0.48 > > > There is only one UUID file is present on 00030/glusterd/peers > > c6b64e36-76da-4e98-a616-48e0e52c7006 //This is the old UUID of the 002500 > board before reboot > > content of this file is: > > uuid=267a92c3-fd28-4811-903c-c1d54854bda9 //This is new UUID generated by > the 002500 board after reboot you can check this as well in glusterd file > of 00030 board. > state=3 > hostname1=10.32.1.144 > >> >> ~Atin >> >> > >> > Regards, >> > Abhishek >> > >> > On Mon, Mar 14, 2016 at 6:12 PM, ABHISHEK PALIWAL >> > <abhishpali...@gmail.com <mailto:abhishpali...@gmail.com>> wrote: >> > >> > You mean etc*-glusterd-*.log file from both of the boards? >> > >> > if yes please find the attachment for the same. >> > >> > On Mon, Mar 14, 2016 at 5:27 PM, Atin Mukherjee < >> amukh...@redhat.com >> > <mailto:amukh...@redhat.com>> wrote: >> > >> > >> > >> > On 03/14/2016 05:09 PM, ABHISHEK PALIWAL wrote: >> > > I am not getting you which glusterd directory you are asking. >> if you are >> > > asking about the /var/lib/glusterd directory then which I >> shared earlier >> > > is the same. >> > 1. Go to /var/log/glusterfs directory >> > 2. Look for glusterd log file >> > 3. attach the log >> > Do it for both the boards. >> > > >> > > I have two directories related to gluster >> > > >> > > 1. /var/log/glusterfs >> > > 2./var/lib/glusterd >> > > >> > > On Mon, Mar 14, 2016 at 4:12 PM, Atin Mukherjee < >> amukh...@redhat.com <mailto:amukh...@redhat.com> >> > > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com>>> >> wrote: >> > > >> > > >> > > >> > > On 03/14/2016 03:59 PM, ABHISHEK PALIWAL wrote: >> > > > I have only these glusterd files available on the nodes >> > > Look for etc-*-glusterd*.log in /var/log/glusterfs, that >> represents the >> > > glusterd log file. >> > > > >> > > > Regards, >> > > > Abhishek >> > > > >> > > > On Mon, Mar 14, 2016 at 3:43 PM, Atin Mukherjee < >> amukh...@redhat.com <mailto:amukh...@redhat.com> >> > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com>> >> > > > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com >> > >> > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com>>>> >> wrote: >> > > > >> > > > >> > > > >> > > > On 03/14/2016 02:18 PM, ABHISHEK PALIWAL wrote: >> > > > > >> > > > > >> > > > > On Mon, Mar 14, 2016 at 12:12 PM, Atin Mukherjee >> > <amukh...@redhat.com <mailto:amukh...@redhat.com> >> > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com>> >> > > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com> >> > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com>>> >> > > > > <mailto:amukh...@redhat.com >> > <mailto:amukh...@redhat.com> <mailto:amukh...@redhat.com >> > <mailto:amukh...@redhat.com>> >> > > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com> >> > <mailto:amukh...@redhat.com <mailto:amukh...@redhat.com>>>>> >> wrote: >> > > > > >> > > > > >> > > > > >> > > > > On 03/14/2016 10:52 AM, ABHISHEK PALIWAL >> wrote: >> > > > > > Hi Team, >> > > > > > >> > > > > > I am facing some issue with peer status and >> > because of >> > > that >> > > > remove-brick >> > > > > > on replica volume is getting failed. >> > > > > > >> > > > > > Here. is the scenario what I am doing with >> > gluster: >> > > > > > >> > > > > > 1. I have two boards A & B and gluster is >> > running on >> > > both of >> > > > the boards. >> > > > > > 2. On board I have created a replicated >> > volume with one >> > > > brick on each >> > > > > > board. >> > > > > > 3. Created one glusterfs mount point where >> > both of >> > > brick are >> > > > mounted. >> > > > > > 4. start the volume with nfs.disable=true. >> > > > > > 5. Till now everything is in sync between >> > both of bricks. >> > > > > > >> > > > > > Now when I manually plug-out the board B >> > from the slot and >> > > > plug-in it again. >> > > > > > >> > > > > > 1. After bootup the board B I have started >> > the glusted on >> > > > the board B. >> > > > > > >> > > > > > Following are the some gluster command >> > output on Board B >> > > > after the step 1. >> > > > > > >> > > > > > # gluster peer status >> > > > > > Number of Peers: 2 >> > > > > > >> > > > > > Hostname: 10.32.0.48 >> > > > > > Uuid: f4ebe3c5-b6a4-4795-98e0-732337f76faf >> > > > > > State: Accepted peer request (Connected) >> > > > > > >> > > > > > Hostname: 10.32.0.48 >> > > > > > Uuid: 4bf982c0-b21b-415c-b870-e72f36c7f2e7 >> > > > > > State: Peer is connected and Accepted >> > (Connected) >> > > > > > >> > > > > > Why this peer status is showing two peer >> with >> > > different UUID? >> > > > > GlusterD doesn't generate a new UUID on init >> > if it has >> > > already >> > > > generated >> > > > > an UUID earlier. This clearly indicates that >> > on reboot >> > > of board B >> > > > > content of /var/lib/glusterd were wiped off. >> > I've asked this >> > > > question to >> > > > > you multiple times that is it the case? >> > > > > >> > > > > >> > > > > Yes I am following the same which is mentioned in >> > the link: >> > > > > >> > > > > >> > > > >> > > >> > >> http://www.gluster.org/community/documentation/index.php/Resolving_Peer_Rejected >> > > > > >> > > > > but why it is showing two peer enteries? >> > > > > >> > > > > > >> > > > > > # gluster volume info >> > > > > > >> > > > > > Volume Name: c_glusterfs >> > > > > > Type: Replicate >> > > > > > Volume ID: >> c11f1f13-64a0-4aca-98b5-91d609a4a18d >> > > > > > Status: Started >> > > > > > Number of Bricks: 1 x 2 = 2 >> > > > > > Transport-type: tcp >> > > > > > Bricks: >> > > > > > Brick1: 10.32.0.48:/opt/lvmdir/c2/brick >> > > > > > Brick2: 10.32.1.144:/opt/lvmdir/c2/brick >> > > > > > Options Reconfigured: >> > > > > > performance.readdir-ahead: on >> > > > > > network.ping-timeout: 4 >> > > > > > nfs.disable: on >> > > > > > # gluster volume heal c_glusterfs info >> > > > > > c_glusterfs: Not able to fetch volfile from >> > glusterd >> > > > > > Volume heal failed. >> > > > > > # gluster volume status c_glusterfs >> > > > > > Status of volume: c_glusterfs >> > > > > > Gluster process >> > TCP Port >> > > RDMA Port >> > > > > Online >> > > > > > Pid >> > > > > > >> > > > > >> > > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > > > > >> > > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick >> > N/A N/A >> > > > N >> > > > > > N/A >> > > > > > Self-heal Daemon on localhost >> > N/A N/A >> > > > Y >> > > > > > 3922 >> > > > > > >> > > > > > Task Status of Volume c_glusterfs >> > > > > > >> > > > > >> > > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > > > > >> > > > > > There are no active volume tasks >> > > > > > -- >> > > > > > >> > > > > > At the same time Board A have the following >> > gluster >> > > commands >> > > > outcome: >> > > > > > >> > > > > > # gluster peer status >> > > > > > Number of Peers: 1 >> > > > > > >> > > > > > Hostname: 10.32.1.144 >> > > > > > Uuid: c6b64e36-76da-4e98-a616-48e0e52c7006 >> > > > > > State: Peer in Cluster (Connected) >> > > > > > >> > > > > > Why it is showing the older UUID of host >> > 10.32.1.144 >> > > when this >> > > > > UUID has >> > > > > > been changed and new UUID is >> > > > 267a92c3-fd28-4811-903c-c1d54854bda9 >> > > > > > >> > > > > > >> > > > > > # gluster volume heal c_glusterfs info >> > > > > > c_glusterfs: Not able to fetch volfile from >> > glusterd >> > > > > > Volume heal failed. >> > > > > > # gluster volume status c_glusterfs >> > > > > > Status of volume: c_glusterfs >> > > > > > Gluster process >> > TCP Port >> > > RDMA Port >> > > > > Online >> > > > > > Pid >> > > > > > >> > > > > >> > > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > > > > >> > > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick >> > 49169 0 >> > > > Y >> > > > > > 2427 >> > > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick >> > N/A N/A >> > > > N >> > > > > > N/A >> > > > > > Self-heal Daemon on localhost >> > N/A N/A >> > > > Y >> > > > > > 3388 >> > > > > > Self-heal Daemon on 10.32.1.144 >> > N/A N/A >> > > > Y >> > > > > > 3922 >> > > > > > >> > > > > > Task Status of Volume c_glusterfs >> > > > > > >> > > > > >> > > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > > > > >> > > > > > There are no active volume tasks >> > > > > > >> > > > > > As you see in the "gluster volume status" >> > showing that >> > > Brick >> > > > > > "10.32.1.144:/opt/lvmdir/c2/brick " is >> > offline so We have >> > > > tried to >> > > > > > remove it but getting "volume remove-brick >> > c_glusterfs >> > > replica 1 >> > > > > > 10.32.1.144:/opt/lvmdir/c2/brick force : >> > FAILED : >> > > Incorrect >> > > > brick >> > > > > > 10.32.1.144:/opt/lvmdir/c2/brick for volume >> > c_glusterfs" >> > > > error on the >> > > > > > Board A. >> > > > > > >> > > > > > Please reply on this post because I am >> > always getting >> > > this error >> > > > > in this >> > > > > > scenario. >> > > > > > >> > > > > > For more detail I am also adding the logs of >> > both of the >> > > > board which >> > > > > > having some manual created file in which you >> > can find the >> > > > output of >> > > > > > glulster command from both of the boards >> > > > > > >> > > > > > in logs >> > > > > > 00030 is board A >> > > > > > 00250 is board B. >> > > > > This attachment doesn't help much. Could you >> > attach full >> > > > glusterd log >> > > > > files from both the nodes? >> > > > > > >> > > > > >> > > > > inside this attachment you will found full >> > glusterd log file >> > > > > 00300/glusterd/ and 002500/glusterd/ >> > > > No, that contains the configuration files. >> > > > > >> > > > > > Thanks in advance waiting for the reply. >> > > > > > >> > > > > > Regards, >> > > > > > Abhishek >> > > > > > >> > > > > > >> > > > > > Regards >> > > > > > Abhishek Paliwal >> > > > > > >> > > > > > >> > > > > > >> _______________________________________________ >> > > > > > Gluster-devel mailing list >> > > > > > Gluster-devel@gluster.org >> > <mailto:Gluster-devel@gluster.org> >> > > <mailto:Gluster-devel@gluster.org >> > <mailto:Gluster-devel@gluster.org>> >> > <mailto:Gluster-devel@gluster.org <mailto: >> Gluster-devel@gluster.org> >> > > <mailto:Gluster-devel@gluster.org >> > <mailto:Gluster-devel@gluster.org>>> >> > > > <mailto:Gluster-devel@gluster.org >> > <mailto:Gluster-devel@gluster.org> >> > > <mailto:Gluster-devel@gluster.org >> > <mailto:Gluster-devel@gluster.org>> >> > <mailto:Gluster-devel@gluster.org <mailto: >> Gluster-devel@gluster.org> >> > > <mailto:Gluster-devel@gluster.org >> > <mailto:Gluster-devel@gluster.org>>>> >> > > > > > >> > http://www.gluster.org/mailman/listinfo/gluster-devel >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > Regards >> > > > > Abhishek Paliwal >> > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > >> > > > >> > > > >> > > > >> > > > Regards >> > > > Abhishek Paliwal >> > > >> > > >> > > >> > > >> > > -- >> > > >> > > >> > > >> > > >> > > Regards >> > > Abhishek Paliwal >> > >> > >> > >> > >> > -- >> > >> > >> > >> > >> > Regards >> > Abhishek Paliwal >> > >> > >> > >> > >> > -- >> > >> > >> > >> > >> > Regards >> > Abhishek Paliwal >> > > > > -- > > > > > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
