Re: [Gluster-users] Fwd: Replica brick not working

2016-12-19 Thread Miloš Čučulović - MDPI 

All was fixed now, thank you very much.

I had to completely purge the storage2 server for all glusterfs files.
The entire procedure is as follows:

How to add a new replica volume

1. Purge all glusterfs packages on new server
2. Remove all from /var/log/glusterfs and /var/lib/glusterfs
3. Install fresh glusterfs-server installation
4. On new server, start glusterfs-server
5. On old server, run gluster peer probe storage2
6. On both servers, check gluster peer status
7. If it is 'Peer in cluster' on both, then you're good.
8. On old server, run sudo gluster volume add-brick storage replica 2 
storage2:/data/data-cluster

9. Return message is volume add-brick: success
10. Healing will start automatically, check the status with gluster 
volume heal storage info


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 18:58, Pranith Kumar Karampuri wrote:



On Thu, Dec 8, 2016 at 11:25 PM, Pranith Kumar Karampuri
mailto:pkara...@redhat.com>> wrote:



On Thu, Dec 8, 2016 at 11:17 PM, Pranith Kumar Karampuri
mailto:pkara...@redhat.com>> wrote:



On Thu, Dec 8, 2016 at 10:22 PM, Ravishankar N
mailto:ravishan...@redhat.com>> wrote:

On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:

I was able to fix the sync by rsync-ing all the
directories, then the hale started. The next problem :),
as soon as there are files on the new brick, the gluster
mount will render also this one for mounts, and the new
brick is not ready yet, as the sync is not yet done, so
it results on missing files on client side. I temporary
removed the new brick, now I am running a manual rsync
and will add the brick again, hope this could work.

What mechanism is managing this issue, I guess there is
something per built to make a replica brick available
only once the data is completely synced.

This mechanism was introduced in  3.7.9 or 3.7.10
(http://review.gluster.org/#/c/13806/
). Before that
version, you manually needed to set some xattrs on the
bricks so that healing could happen in parallel while the
client still would server reads from the original brick.  I
can't find the link to the doc which describes these steps
for setting xattrs.:-(



https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-brick




Oh this is addition of bricks?
Just do the following:
1) Bring the new brick down by killing it.
2) On the root of the mount directory(Let's call it /mnt) do:

mkdir /mnt/
rmdir /mnt/
setfattr -n trusted.non-existent-key -v abc /mnt
setfattr -x trusted.non-existent-key  /mnt

3) Start the volume using: "gluster volume start  force"

This will trigger the heal which will make sure everything is healed
and the application will only see the correct data.

Since you did an explicit rsync, there is no gurantee that things
should work as expected. We will be adding the steps above to
documentation.


Please note that you need to do these steps exactly, If you do the
mkdir/rmdir/setfattr steps by bringing the good brick, reverse heal will
happen and the data will be removed.





Calling it a day,
Ravi


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:17, Ravishankar N wrote:

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović
- MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

Ah, damn! I found the issue. On the storage
server, the storage2
IP address was wrong, I inversed two digits
in the /etc/hosts
file, sorry for that :(

I was able to add the brick

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-14 Thread Atin Mukherjee 
On Wed, Dec 14, 2016 at 1:34 PM, Miloš Čučulović - MDPI 
wrote:

> Atin,
>
> I was able to move forward a bit. Initially, I had this:
>
> sudo gluster peer status
> Number of Peers: 1
>
> Hostname: storage2
> Uuid: 32bef70a-9e31-403e-b9f3-ec9e1bd162ad
> State: Peer Rejected (Connected)
>
> Then, on storage2 I removed all from /var/lib/glusterd except the info
> file.
>
> Now I am getting another error message:
>
> sudo gluster peer status
> Number of Peers: 1
>
> Hostname: storage2
> Uuid: 32bef70a-9e31-403e-b9f3-ec9e1bd162ad
> State: Sent and Received peer request (Connected)
>


Please edit /var/lib/glusterd/peers/32bef70a-9e31-403e-b9f3-ec9e1bd162ad
file and set the state to 3 in storage1 and restart glusterd instance.


> But the add brick is still not working. I checked the hosts file and all
> seems ok, ping is also working well.
>
> The think I also need to know, when adding a new replicated brick, do I
> need to first sync all files, or the new brick server needs to be empty?
> Also, do I first need to create the same volume on the new server or adding
> it to the volume of server1 will do it automatically?
>
>
> - Kindest regards,
>
> Milos Cuculovic
> IT Manager
>
> ---
> MDPI AG
> Postfach, CH-4020 Basel, Switzerland
> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
> Tel. +41 61 683 77 35
> Fax +41 61 302 89 18
> Email: cuculo...@mdpi.com
> Skype: milos.cuculovic.mdpi
>
> On 14.12.2016 05:13, Atin Mukherjee wrote:
>
>> Milos,
>>
>> I just managed to take a look into a similar issue and my analysis is at
>> [1]. I remember you mentioning about some incorrect /etc/hosts entries
>> which lead to this same problem in earlier case, do you mind to recheck
>> the same?
>>
>> [1]
>> http://www.gluster.org/pipermail/gluster-users/2016-December/029443.html
>>
>> On Wed, Dec 14, 2016 at 2:57 AM, Miloš Čučulović - MDPI
>> mailto:cuculo...@mdpi.com>> wrote:
>>
>> Hi All,
>>
>> Moving forward with my issue, sorry for the late reply!
>>
>> I had some issues with the storage2 server (original volume), then
>> decided to use 3.9.0, si I have the latest version.
>>
>> For that, I synced manually all the files to the storage server. I
>> installed there gluster 3.9.0, started it, created new volume called
>> storage and all seems to work ok.
>>
>> Now, I need to create my replicated volume (add new brick on
>> storage2 server). Almost all the files are there. So, I was adding
>> on storage server:
>>
>> * sudo gluter peer probe storage2
>> * sudo gluster volume add-brick storage replica 2
>> storage2:/data/data-cluster force
>>
>> But there I am receiving "volume add-brick: failed: Host storage2 is
>> not in 'Peer in Cluster' state"
>>
>> Any idea?
>>
>> - Kindest regards,
>>
>> Milos Cuculovic
>> IT Manager
>>
>> ---
>> MDPI AG
>> Postfach, CH-4020 Basel, Switzerland
>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>> Tel. +41 61 683 77 35
>> Fax +41 61 302 89 18
>> Email: cuculo...@mdpi.com 
>> Skype: milos.cuculovic.mdpi
>>
>> On 08.12.2016 17:52, Ravishankar N wrote:
>>
>> On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:
>>
>> I was able to fix the sync by rsync-ing all the directories,
>> then the
>> hale started. The next problem :), as soon as there are
>> files on the
>> new brick, the gluster mount will render also this one for
>> mounts, and
>> the new brick is not ready yet, as the sync is not yet done,
>> so it
>> results on missing files on client side. I temporary removed
>> the new
>> brick, now I am running a manual rsync and will add the
>> brick again,
>> hope this could work.
>>
>> What mechanism is managing this issue, I guess there is
>> something per
>> built to make a replica brick available only once the data is
>> completely synced.
>>
>> This mechanism was introduced in  3.7.9 or 3.7.10
>> (http://review.gluster.org/#/c/13806/
>> ). Before that version, you
>> manually needed to set some xattrs on the bricks so that healing
>> could
>> happen in parallel while the client still would server reads
>> from the
>> original brick.  I can't find the link to the doc which
>> describes these
>> steps for setting xattrs.:-(
>>
>> Calling it a day,
>> Ravi
>>
>>
>> - Kindest regards,
>>
>> Milos Cuculovic
>> IT Manager
>>
>> ---
>> MDPI AG
>> Postfach, CH-4020 Basel, Switzerland
>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>> Tel. +41 61 683 77 35
>> Fax +41 61 302 89 18
>> Em

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-14 Thread Miloš Čučulović - MDPI 

Atin,

I was able to move forward a bit. Initially, I had this:

sudo gluster peer status
Number of Peers: 1

Hostname: storage2
Uuid: 32bef70a-9e31-403e-b9f3-ec9e1bd162ad
State: Peer Rejected (Connected)

Then, on storage2 I removed all from /var/lib/glusterd except the info file.

Now I am getting another error message:

sudo gluster peer status
Number of Peers: 1

Hostname: storage2
Uuid: 32bef70a-9e31-403e-b9f3-ec9e1bd162ad
State: Sent and Received peer request (Connected)

But the add brick is still not working. I checked the hosts file and all 
seems ok, ping is also working well.


The think I also need to know, when adding a new replicated brick, do I 
need to first sync all files, or the new brick server needs to be empty? 
Also, do I first need to create the same volume on the new server or 
adding it to the volume of server1 will do it automatically?



- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 14.12.2016 05:13, Atin Mukherjee wrote:

Milos,

I just managed to take a look into a similar issue and my analysis is at
[1]. I remember you mentioning about some incorrect /etc/hosts entries
which lead to this same problem in earlier case, do you mind to recheck
the same?

[1]
http://www.gluster.org/pipermail/gluster-users/2016-December/029443.html

On Wed, Dec 14, 2016 at 2:57 AM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>> wrote:

Hi All,

Moving forward with my issue, sorry for the late reply!

I had some issues with the storage2 server (original volume), then
decided to use 3.9.0, si I have the latest version.

For that, I synced manually all the files to the storage server. I
installed there gluster 3.9.0, started it, created new volume called
storage and all seems to work ok.

Now, I need to create my replicated volume (add new brick on
storage2 server). Almost all the files are there. So, I was adding
on storage server:

* sudo gluter peer probe storage2
* sudo gluster volume add-brick storage replica 2
storage2:/data/data-cluster force

But there I am receiving "volume add-brick: failed: Host storage2 is
not in 'Peer in Cluster' state"

Any idea?

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 17:52, Ravishankar N wrote:

On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:

I was able to fix the sync by rsync-ing all the directories,
then the
hale started. The next problem :), as soon as there are
files on the
new brick, the gluster mount will render also this one for
mounts, and
the new brick is not ready yet, as the sync is not yet done,
so it
results on missing files on client side. I temporary removed
the new
brick, now I am running a manual rsync and will add the
brick again,
hope this could work.

What mechanism is managing this issue, I guess there is
something per
built to make a replica brick available only once the data is
completely synced.

This mechanism was introduced in  3.7.9 or 3.7.10
(http://review.gluster.org/#/c/13806/
). Before that version, you
manually needed to set some xattrs on the bricks so that healing
could
happen in parallel while the client still would server reads
from the
original brick.  I can't find the link to the doc which
describes these
steps for setting xattrs.:-(

Calling it a day,
Ravi


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:17, Ravishankar N wrote:

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

Ah, damn! I found the issue. On the storage
server, the storage2

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-13 Thread Atin Mukherjee 
Milos,

I just managed to take a look into a similar issue and my analysis is at
[1]. I remember you mentioning about some incorrect /etc/hosts entries
which lead to this same problem in earlier case, do you mind to recheck the
same?

[1] http://www.gluster.org/pipermail/gluster-users/2016-December/029443.html

On Wed, Dec 14, 2016 at 2:57 AM, Miloš Čučulović - MDPI 
wrote:

> Hi All,
>
> Moving forward with my issue, sorry for the late reply!
>
> I had some issues with the storage2 server (original volume), then decided
> to use 3.9.0, si I have the latest version.
>
> For that, I synced manually all the files to the storage server. I
> installed there gluster 3.9.0, started it, created new volume called
> storage and all seems to work ok.
>
> Now, I need to create my replicated volume (add new brick on storage2
> server). Almost all the files are there. So, I was adding on storage server:
>
> * sudo gluter peer probe storage2
> * sudo gluster volume add-brick storage replica 2
> storage2:/data/data-cluster force
>
> But there I am receiving "volume add-brick: failed: Host storage2 is not
> in 'Peer in Cluster' state"
>
> Any idea?
>
> - Kindest regards,
>
> Milos Cuculovic
> IT Manager
>
> ---
> MDPI AG
> Postfach, CH-4020 Basel, Switzerland
> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
> Tel. +41 61 683 77 35
> Fax +41 61 302 89 18
> Email: cuculo...@mdpi.com
> Skype: milos.cuculovic.mdpi
>
> On 08.12.2016 17:52, Ravishankar N wrote:
>
>> On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:
>>
>>> I was able to fix the sync by rsync-ing all the directories, then the
>>> hale started. The next problem :), as soon as there are files on the
>>> new brick, the gluster mount will render also this one for mounts, and
>>> the new brick is not ready yet, as the sync is not yet done, so it
>>> results on missing files on client side. I temporary removed the new
>>> brick, now I am running a manual rsync and will add the brick again,
>>> hope this could work.
>>>
>>> What mechanism is managing this issue, I guess there is something per
>>> built to make a replica brick available only once the data is
>>> completely synced.
>>>
>> This mechanism was introduced in  3.7.9 or 3.7.10
>> (http://review.gluster.org/#/c/13806/). Before that version, you
>> manually needed to set some xattrs on the bricks so that healing could
>> happen in parallel while the client still would server reads from the
>> original brick.  I can't find the link to the doc which describes these
>> steps for setting xattrs.:-(
>>
>> Calling it a day,
>> Ravi
>>
>>>
>>> - Kindest regards,
>>>
>>> Milos Cuculovic
>>> IT Manager
>>>
>>> ---
>>> MDPI AG
>>> Postfach, CH-4020 Basel, Switzerland
>>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>>> Tel. +41 61 683 77 35
>>> Fax +41 61 302 89 18
>>> Email: cuculo...@mdpi.com
>>> Skype: milos.cuculovic.mdpi
>>>
>>> On 08.12.2016 16:17, Ravishankar N wrote:
>>>
 On 12/08/2016 06:53 PM, Atin Mukherjee wrote:

>
>
> On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
> mailto:cuculo...@mdpi.com>> wrote:
>
> Ah, damn! I found the issue. On the storage server, the storage2
> IP address was wrong, I inversed two digits in the /etc/hosts
> file, sorry for that :(
>
> I was able to add the brick now, I started the heal, but still no
> data transfer visible.
>
> 1. Are the files getting created on the new brick though?
 2. Can you provide the output of `getfattr -d -m . -e hex
 /data/data-cluster` on both bricks?
 3. Is it possible to attach gdb to the self-heal daemon on the original
 (old) brick and get a backtrace?
 `gdb -p `
  thread apply all bt  -->share this output
 quit gdb.


 -Ravi

>
> @Ravi/Pranith - can you help here?
>
>
>
> By doing gluster volume status, I have
>
> Status of volume: storage
> Gluster process   TCP Port  RDMA Port
> Online  Pid
> 
> --
>
> Brick storage2:/data/data-cluster 49152 0 Y
>  23101
> Brick storage:/data/data-cluster  49152 0 Y
>  30773
> Self-heal Daemon on localhost N/A   N/A Y
>  30050
> Self-heal Daemon on storage   N/A   N/A Y
>  30792
>
>
> Any idea?
>
> On storage I have:
> Number of Peers: 1
>
> Hostname: 195.65.194.217
> Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
> State: Peer in Cluster (Connected)
>
>
> - Kindest regards,
>
> Milos Cuculovic
> IT Manager
>
> ---
> MDPI AG
> Postfach, CH-4020 Basel, Switzerland
> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
> Tel. +41 61 683 77 35
> Fax +41 61

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-13 Thread Miloš Čučulović - MDPI 

Hi All,

Moving forward with my issue, sorry for the late reply!

I had some issues with the storage2 server (original volume), then 
decided to use 3.9.0, si I have the latest version.


For that, I synced manually all the files to the storage server. I 
installed there gluster 3.9.0, started it, created new volume called 
storage and all seems to work ok.


Now, I need to create my replicated volume (add new brick on storage2 
server). Almost all the files are there. So, I was adding on storage server:


* sudo gluter peer probe storage2
* sudo gluster volume add-brick storage replica 2 
storage2:/data/data-cluster force


But there I am receiving "volume add-brick: failed: Host storage2 is not 
in 'Peer in Cluster' state"


Any idea?

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 17:52, Ravishankar N wrote:

On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:

I was able to fix the sync by rsync-ing all the directories, then the
hale started. The next problem :), as soon as there are files on the
new brick, the gluster mount will render also this one for mounts, and
the new brick is not ready yet, as the sync is not yet done, so it
results on missing files on client side. I temporary removed the new
brick, now I am running a manual rsync and will add the brick again,
hope this could work.

What mechanism is managing this issue, I guess there is something per
built to make a replica brick available only once the data is
completely synced.

This mechanism was introduced in  3.7.9 or 3.7.10
(http://review.gluster.org/#/c/13806/). Before that version, you
manually needed to set some xattrs on the bricks so that healing could
happen in parallel while the client still would server reads from the
original brick.  I can't find the link to the doc which describes these
steps for setting xattrs.:-(

Calling it a day,
Ravi


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:17, Ravishankar N wrote:

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>> wrote:

Ah, damn! I found the issue. On the storage server, the storage2
IP address was wrong, I inversed two digits in the /etc/hosts
file, sorry for that :(

I was able to add the brick now, I started the heal, but still no
data transfer visible.


1. Are the files getting created on the new brick though?
2. Can you provide the output of `getfattr -d -m . -e hex
/data/data-cluster` on both bricks?
3. Is it possible to attach gdb to the self-heal daemon on the original
(old) brick and get a backtrace?
`gdb -p `
 thread apply all bt  -->share this output
quit gdb.


-Ravi


@Ravi/Pranith - can you help here?



By doing gluster volume status, I have

Status of volume: storage
Gluster process   TCP Port  RDMA Port
Online  Pid
--

Brick storage2:/data/data-cluster 49152 0 Y
 23101
Brick storage:/data/data-cluster  49152 0 Y
 30773
Self-heal Daemon on localhost N/A   N/A Y
 30050
Self-heal Daemon on storage   N/A   N/A Y
 30792


Any idea?

On storage I have:
Number of Peers: 1

Hostname: 195.65.194.217
Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
State: Peer in Cluster (Connected)


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 13:55, Atin Mukherjee wrote:

Can you resend the attachment as zip? I am unable to extract
the
content? We shouldn't have 0 info file. What does gluster peer
status
output say?

On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

I hope you received my last email Atin, thank you!

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Pranith Kumar Karampuri 
On Thu, Dec 8, 2016 at 11:25 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Thu, Dec 8, 2016 at 11:17 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Thu, Dec 8, 2016 at 10:22 PM, Ravishankar N 
>> wrote:
>>
>>> On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:
>>>
 I was able to fix the sync by rsync-ing all the directories, then the
 hale started. The next problem :), as soon as there are files on the new
 brick, the gluster mount will render also this one for mounts, and the new
 brick is not ready yet, as the sync is not yet done, so it results on
 missing files on client side. I temporary removed the new brick, now I am
 running a manual rsync and will add the brick again, hope this could work.

 What mechanism is managing this issue, I guess there is something per
 built to make a replica brick available only once the data is completely
 synced.

>>> This mechanism was introduced in  3.7.9 or 3.7.10 (
>>> http://review.gluster.org/#/c/13806/). Before that version, you
>>> manually needed to set some xattrs on the bricks so that healing could
>>> happen in parallel while the client still would server reads from the
>>> original brick.  I can't find the link to the doc which describes these
>>> steps for setting xattrs.:-(
>>>
>>
>> https://gluster.readthedocs.io/en/latest/Administrator%20Gui
>> de/Managing%20Volumes/#replace-brick
>>
>
> Oh this is addition of bricks?
> Just do the following:
> 1) Bring the new brick down by killing it.
> 2) On the root of the mount directory(Let's call it /mnt) do:
>
> mkdir /mnt/
> rmdir /mnt/
> setfattr -n trusted.non-existent-key -v abc /mnt
> setfattr -x trusted.non-existent-key  /mnt
>
> 3) Start the volume using: "gluster volume start  force"
>
> This will trigger the heal which will make sure everything is healed and
> the application will only see the correct data.
>
> Since you did an explicit rsync, there is no gurantee that things should
> work as expected. We will be adding the steps above to documentation.
>

Please note that you need to do these steps exactly, If you do the
mkdir/rmdir/setfattr steps by bringing the good brick, reverse heal will
happen and the data will be removed.


>
>>
>>
>>> Calling it a day,
>>> Ravi
>>>
>>>
 - Kindest regards,

 Milos Cuculovic
 IT Manager

 ---
 MDPI AG
 Postfach, CH-4020 Basel, Switzerland
 Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
 Tel. +41 61 683 77 35
 Fax +41 61 302 89 18
 Email: cuculo...@mdpi.com
 Skype: milos.cuculovic.mdpi

 On 08.12.2016 16:17, Ravishankar N wrote:

> On 12/08/2016 06:53 PM, Atin Mukherjee wrote:
>
>>
>>
>> On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
>> mailto:cuculo...@mdpi.com>> wrote:
>>
>> Ah, damn! I found the issue. On the storage server, the storage2
>> IP address was wrong, I inversed two digits in the /etc/hosts
>> file, sorry for that :(
>>
>> I was able to add the brick now, I started the heal, but still no
>> data transfer visible.
>>
>> 1. Are the files getting created on the new brick though?
> 2. Can you provide the output of `getfattr -d -m . -e hex
> /data/data-cluster` on both bricks?
> 3. Is it possible to attach gdb to the self-heal daemon on the original
> (old) brick and get a backtrace?
> `gdb -p `
>  thread apply all bt  -->share this output
> quit gdb.
>
>
> -Ravi
>
>>
>> @Ravi/Pranith - can you help here?
>>
>>
>>
>> By doing gluster volume status, I have
>>
>> Status of volume: storage
>> Gluster process   TCP Port  RDMA Port Online
>> Pid
>> 
>> --
>> Brick storage2:/data/data-cluster 49152 0 Y
>>  23101
>> Brick storage:/data/data-cluster  49152 0 Y
>>  30773
>> Self-heal Daemon on localhost N/A   N/A Y
>>  30050
>> Self-heal Daemon on storage   N/A   N/A Y
>>  30792
>>
>>
>> Any idea?
>>
>> On storage I have:
>> Number of Peers: 1
>>
>> Hostname: 195.65.194.217
>> Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
>> State: Peer in Cluster (Connected)
>>
>>
>> - Kindest regards,
>>
>> Milos Cuculovic
>> IT Manager
>>
>> ---
>> MDPI AG
>> Postfach, CH-4020 Basel, Switzerland
>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>> Tel. +41 61 683 77 35
>> Fax +41 61 302 89 18
>> Email: cuculo...@mdpi.com 
>> Skype: milos.cuculovic.mdpi
>>
>> On 08.12.2016 13:55, Atin Mukherjee wrote:
>>

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Pranith Kumar Karampuri 
On Thu, Dec 8, 2016 at 11:17 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Thu, Dec 8, 2016 at 10:22 PM, Ravishankar N 
> wrote:
>
>> On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:
>>
>>> I was able to fix the sync by rsync-ing all the directories, then the
>>> hale started. The next problem :), as soon as there are files on the new
>>> brick, the gluster mount will render also this one for mounts, and the new
>>> brick is not ready yet, as the sync is not yet done, so it results on
>>> missing files on client side. I temporary removed the new brick, now I am
>>> running a manual rsync and will add the brick again, hope this could work.
>>>
>>> What mechanism is managing this issue, I guess there is something per
>>> built to make a replica brick available only once the data is completely
>>> synced.
>>>
>> This mechanism was introduced in  3.7.9 or 3.7.10 (
>> http://review.gluster.org/#/c/13806/). Before that version, you manually
>> needed to set some xattrs on the bricks so that healing could happen in
>> parallel while the client still would server reads from the original
>> brick.  I can't find the link to the doc which describes these steps for
>> setting xattrs.:-(
>>
>
> https://gluster.readthedocs.io/en/latest/Administrator%
> 20Guide/Managing%20Volumes/#replace-brick
>

Oh this is addition of bricks?
Just do the following:
1) Bring the new brick down by killing it.
2) On the root of the mount directory(Let's call it /mnt) do:

mkdir /mnt/
rmdir /mnt/
setfattr -n trusted.non-existent-key -v abc /mnt
setfattr -x trusted.non-existent-key  /mnt

3) Start the volume using: "gluster volume start  force"

This will trigger the heal which will make sure everything is healed and
the application will only see the correct data.

Since you did an explicit rsync, there is no gurantee that things should
work as expected. We will be adding the steps above to documentation.


>
>
>> Calling it a day,
>> Ravi
>>
>>
>>> - Kindest regards,
>>>
>>> Milos Cuculovic
>>> IT Manager
>>>
>>> ---
>>> MDPI AG
>>> Postfach, CH-4020 Basel, Switzerland
>>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>>> Tel. +41 61 683 77 35
>>> Fax +41 61 302 89 18
>>> Email: cuculo...@mdpi.com
>>> Skype: milos.cuculovic.mdpi
>>>
>>> On 08.12.2016 16:17, Ravishankar N wrote:
>>>
 On 12/08/2016 06:53 PM, Atin Mukherjee wrote:

>
>
> On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
> mailto:cuculo...@mdpi.com>> wrote:
>
> Ah, damn! I found the issue. On the storage server, the storage2
> IP address was wrong, I inversed two digits in the /etc/hosts
> file, sorry for that :(
>
> I was able to add the brick now, I started the heal, but still no
> data transfer visible.
>
> 1. Are the files getting created on the new brick though?
 2. Can you provide the output of `getfattr -d -m . -e hex
 /data/data-cluster` on both bricks?
 3. Is it possible to attach gdb to the self-heal daemon on the original
 (old) brick and get a backtrace?
 `gdb -p `
  thread apply all bt  -->share this output
 quit gdb.


 -Ravi

>
> @Ravi/Pranith - can you help here?
>
>
>
> By doing gluster volume status, I have
>
> Status of volume: storage
> Gluster process   TCP Port  RDMA Port Online
> Pid
> 
> --
> Brick storage2:/data/data-cluster 49152 0 Y
>  23101
> Brick storage:/data/data-cluster  49152 0 Y
>  30773
> Self-heal Daemon on localhost N/A   N/A Y
>  30050
> Self-heal Daemon on storage   N/A   N/A Y
>  30792
>
>
> Any idea?
>
> On storage I have:
> Number of Peers: 1
>
> Hostname: 195.65.194.217
> Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
> State: Peer in Cluster (Connected)
>
>
> - Kindest regards,
>
> Milos Cuculovic
> IT Manager
>
> ---
> MDPI AG
> Postfach, CH-4020 Basel, Switzerland
> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
> Tel. +41 61 683 77 35
> Fax +41 61 302 89 18
> Email: cuculo...@mdpi.com 
> Skype: milos.cuculovic.mdpi
>
> On 08.12.2016 13:55, Atin Mukherjee wrote:
>
> Can you resend the attachment as zip? I am unable to extract
> the
> content? We shouldn't have 0 info file. What does gluster peer
> status
> output say?
>
> On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
> mailto:cuculo...@mdpi.com>
> >>
> wrote:
>
> I hope

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Pranith Kumar Karampuri 
On Thu, Dec 8, 2016 at 10:22 PM, Ravishankar N 
wrote:

> On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:
>
>> I was able to fix the sync by rsync-ing all the directories, then the
>> hale started. The next problem :), as soon as there are files on the new
>> brick, the gluster mount will render also this one for mounts, and the new
>> brick is not ready yet, as the sync is not yet done, so it results on
>> missing files on client side. I temporary removed the new brick, now I am
>> running a manual rsync and will add the brick again, hope this could work.
>>
>> What mechanism is managing this issue, I guess there is something per
>> built to make a replica brick available only once the data is completely
>> synced.
>>
> This mechanism was introduced in  3.7.9 or 3.7.10 (
> http://review.gluster.org/#/c/13806/). Before that version, you manually
> needed to set some xattrs on the bricks so that healing could happen in
> parallel while the client still would server reads from the original
> brick.  I can't find the link to the doc which describes these steps for
> setting xattrs.:-(
>

https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-brick


> Calling it a day,
> Ravi
>
>
>> - Kindest regards,
>>
>> Milos Cuculovic
>> IT Manager
>>
>> ---
>> MDPI AG
>> Postfach, CH-4020 Basel, Switzerland
>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>> Tel. +41 61 683 77 35
>> Fax +41 61 302 89 18
>> Email: cuculo...@mdpi.com
>> Skype: milos.cuculovic.mdpi
>>
>> On 08.12.2016 16:17, Ravishankar N wrote:
>>
>>> On 12/08/2016 06:53 PM, Atin Mukherjee wrote:
>>>


 On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
 mailto:cuculo...@mdpi.com>> wrote:

 Ah, damn! I found the issue. On the storage server, the storage2
 IP address was wrong, I inversed two digits in the /etc/hosts
 file, sorry for that :(

 I was able to add the brick now, I started the heal, but still no
 data transfer visible.

 1. Are the files getting created on the new brick though?
>>> 2. Can you provide the output of `getfattr -d -m . -e hex
>>> /data/data-cluster` on both bricks?
>>> 3. Is it possible to attach gdb to the self-heal daemon on the original
>>> (old) brick and get a backtrace?
>>> `gdb -p `
>>>  thread apply all bt  -->share this output
>>> quit gdb.
>>>
>>>
>>> -Ravi
>>>

 @Ravi/Pranith - can you help here?



 By doing gluster volume status, I have

 Status of volume: storage
 Gluster process   TCP Port  RDMA Port Online
 Pid
 
 --
 Brick storage2:/data/data-cluster 49152 0 Y
  23101
 Brick storage:/data/data-cluster  49152 0 Y
  30773
 Self-heal Daemon on localhost N/A   N/A Y
  30050
 Self-heal Daemon on storage   N/A   N/A Y
  30792


 Any idea?

 On storage I have:
 Number of Peers: 1

 Hostname: 195.65.194.217
 Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
 State: Peer in Cluster (Connected)


 - Kindest regards,

 Milos Cuculovic
 IT Manager

 ---
 MDPI AG
 Postfach, CH-4020 Basel, Switzerland
 Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
 Tel. +41 61 683 77 35
 Fax +41 61 302 89 18
 Email: cuculo...@mdpi.com 
 Skype: milos.cuculovic.mdpi

 On 08.12.2016 13:55, Atin Mukherjee wrote:

 Can you resend the attachment as zip? I am unable to extract the
 content? We shouldn't have 0 info file. What does gluster peer
 status
 output say?

 On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
 mailto:cuculo...@mdpi.com>
 >> wrote:

 I hope you received my last email Atin, thank you!

 - Kindest regards,

 Milos Cuculovic
 IT Manager

 ---
 MDPI AG
 Postfach, CH-4020 Basel, Switzerland
 Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
 Tel. +41 61 683 77 35
 Fax +41 61 302 89 18
 Email: cuculo...@mdpi.com 
 >
 Skype: milos.cuculovic.mdpi

 On 08.12.2016 10:28, Atin Mukherjee wrote:


 -- Forwarded message --
 From: *Atin Mukherjee* >>>

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Ravishankar N 

On 12/08/2016 09:44 PM, Miloš Čučulović - MDPI wrote:
I was able to fix the sync by rsync-ing all the directories, then the 
hale started. The next problem :), as soon as there are files on the 
new brick, the gluster mount will render also this one for mounts, and 
the new brick is not ready yet, as the sync is not yet done, so it 
results on missing files on client side. I temporary removed the new 
brick, now I am running a manual rsync and will add the brick again, 
hope this could work.


What mechanism is managing this issue, I guess there is something per 
built to make a replica brick available only once the data is 
completely synced.
This mechanism was introduced in  3.7.9 or 3.7.10 
(http://review.gluster.org/#/c/13806/). Before that version, you 
manually needed to set some xattrs on the bricks so that healing could 
happen in parallel while the client still would server reads from the 
original brick.  I can't find the link to the doc which describes these 
steps for setting xattrs.:-(


Calling it a day,
Ravi


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:17, Ravishankar N wrote:

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>> wrote:

Ah, damn! I found the issue. On the storage server, the storage2
IP address was wrong, I inversed two digits in the /etc/hosts
file, sorry for that :(

I was able to add the brick now, I started the heal, but still no
data transfer visible.


1. Are the files getting created on the new brick though?
2. Can you provide the output of `getfattr -d -m . -e hex
/data/data-cluster` on both bricks?
3. Is it possible to attach gdb to the self-heal daemon on the original
(old) brick and get a backtrace?
`gdb -p `
 thread apply all bt  -->share this output
quit gdb.


-Ravi


@Ravi/Pranith - can you help here?



By doing gluster volume status, I have

Status of volume: storage
Gluster process   TCP Port  RDMA Port 
Online  Pid

--
Brick storage2:/data/data-cluster 49152 0 Y
 23101
Brick storage:/data/data-cluster  49152 0 Y
 30773
Self-heal Daemon on localhost N/A   N/A Y
 30050
Self-heal Daemon on storage   N/A   N/A Y
 30792


Any idea?

On storage I have:
Number of Peers: 1

Hostname: 195.65.194.217
Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
State: Peer in Cluster (Connected)


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 13:55, Atin Mukherjee wrote:

Can you resend the attachment as zip? I am unable to extract 
the

content? We shouldn't have 0 info file. What does gluster peer
status
output say?

On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

I hope you received my last email Atin, thank you!

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
>
Skype: milos.cuculovic.mdpi

On 08.12.2016 10:28, Atin Mukherjee wrote:


-- Forwarded message --
From: *Atin Mukherjee* mailto:amukh...@redhat.com>
> 

>

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Miloš Čučulović - MDPI 
I was able to fix the sync by rsync-ing all the directories, then the 
hale started. The next problem :), as soon as there are files on the new 
brick, the gluster mount will render also this one for mounts, and the 
new brick is not ready yet, as the sync is not yet done, so it results 
on missing files on client side. I temporary removed the new brick, now 
I am running a manual rsync and will add the brick again, hope this 
could work.


What mechanism is managing this issue, I guess there is something per 
built to make a replica brick available only once the data is completely 
synced.


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:17, Ravishankar N wrote:

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>> wrote:

Ah, damn! I found the issue. On the storage server, the storage2
IP address was wrong, I inversed two digits in the /etc/hosts
file, sorry for that :(

I was able to add the brick now, I started the heal, but still no
data transfer visible.


1. Are the files getting created on the new brick though?
2. Can you provide the output of `getfattr -d -m . -e hex
/data/data-cluster` on both bricks?
3. Is it possible to attach gdb to the self-heal daemon on the original
(old) brick and get a backtrace?
`gdb -p `
 thread apply all bt  -->share this output
quit gdb.


-Ravi


@Ravi/Pranith - can you help here?



By doing gluster volume status, I have

Status of volume: storage
Gluster process   TCP Port  RDMA Port  Online  Pid

--
Brick storage2:/data/data-cluster 49152 0  Y
 23101
Brick storage:/data/data-cluster  49152 0  Y
 30773
Self-heal Daemon on localhost N/A   N/AY
 30050
Self-heal Daemon on storage   N/A   N/AY
 30792


Any idea?

On storage I have:
Number of Peers: 1

Hostname: 195.65.194.217
Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
State: Peer in Cluster (Connected)


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 13:55, Atin Mukherjee wrote:

Can you resend the attachment as zip? I am unable to extract the
content? We shouldn't have 0 info file. What does gluster peer
status
output say?

On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

I hope you received my last email Atin, thank you!

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
>
Skype: milos.cuculovic.mdpi

On 08.12.2016 10:28, Atin Mukherjee wrote:


-- Forwarded message --
From: *Atin Mukherjee* mailto:amukh...@redhat.com>
> 

>


>


>

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Miloš Čučulović - MDPI 

Additional info, there are warning / errors in the new brick:

[2016-12-08 15:37:05.053615] E [MSGID: 115056] 
[server-rpc-fops.c:509:server_mkdir_cbk] 0-storage-server: 12636867: 
MKDIR /dms (----0001/dms) ==> (Permission 
denied) [Permission denied]
[2016-12-08 15:37:05.135607] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 12636895: 
FSTAT -2 (e9481d78-9094-45a7-ac7e-e1feeb7055df) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.163610] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523605: 
FSTAT -2 (2bb87992-5f24-44bd-ba7c-70c84510942b) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.163633] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523604: 
FSTAT -2 (2bb87992-5f24-44bd-ba7c-70c84510942b) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.166590] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523619: 
FSTAT -2 (616028b7-a2c2-40e3-998a-68329daf7b07) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.166659] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523620: 
FSTAT -2 (616028b7-a2c2-40e3-998a-68329daf7b07) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.241276] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3451382: 
FSTAT -2 (f00e597e-7ae4-4d3a-986e-bbeb6cc07339) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.268583] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523823: 
FSTAT -2 (a8a343c1-512f-4ad1-a3db-de9fc8ed990c) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.268771] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523824: 
FSTAT -2 (a8a343c1-512f-4ad1-a3db-de9fc8ed990c) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.302501] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523868: 
FSTAT -2 (eb0c4500-f9ae-408a-85e6-6e67ec6466a9) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.302558] I [MSGID: 115081] 
[server-rpc-fops.c:1280:server_fstat_cbk] 0-storage-server: 3523869: 
FSTAT -2 (eb0c4500-f9ae-408a-85e6-6e67ec6466a9) ==> (No such file or 
directory) [No such file or directory]
[2016-12-08 15:37:05.365428] E [MSGID: 115056] 
[server-rpc-fops.c:509:server_mkdir_cbk] 0-storage-server: 12637038: 
MKDIR /files (----0001/files) ==> 
(Permission denied) [Permission denied]
[2016-12-08 15:37:05.414486] E [MSGID: 115056] 
[server-rpc-fops.c:509:server_mkdir_cbk] 0-storage-server: 3451430: 
MKDIR /files (----0001/files) ==> 
(Permission denied) [Permission denied]



- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:32, Miloš Čučulović - MDPI wrote:

1. No, atm the old server (storage2) volume is mounted on some other
servers, so all files are created there. If I check the new brick, there
is no files.


2. On storage2 server (old brick)
getfattr: Removing leading '/' from absolute path names
# file: data/data-cluster
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x0226135726f346bcb3f8cb73365ed382

On storage server (new brick)
getfattr: Removing leading '/' from absolute path names
# file: data/data-cluster
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x0226135726f346bcb3f8cb73365ed382


3.
Thread 8 (Thread 0x7fad832dd700 (LWP 30057)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x7fad88834f3e in __afr_shd_healer_wait () from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/cluster/replicate.so
#2  0x7fad88834fad in afr_shd_healer_wait () from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/cluster/replicate.so
#3  0x7fad88835aa0 in afr_shd_index_healer () from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/cluster/replicate.so
#4  0x7fad8df4270a in start_thread (arg=0x7fad832dd700) at
pthread_create.c:333
#5  0x7fad8dc7882d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7fad83ade700 (LWP 30056)):
#0  0x7fad8dc78e23 in epoll_wait () at
../sysdeps/unix/syscall-template.S:84
#1  0x7fad8e808a58 in ?? () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2  0x7fad8df4270a in start_thread (arg=0x7fad83ade700) at
pthrea

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Miloš Čučulović - MDPI 
1. No, atm the old server (storage2) volume is mounted on some other 
servers, so all files are created there. If I check the new brick, there 
is no files.



2. On storage2 server (old brick)
getfattr: Removing leading '/' from absolute path names
# file: data/data-cluster
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x0226135726f346bcb3f8cb73365ed382

On storage server (new brick)
getfattr: Removing leading '/' from absolute path names
# file: data/data-cluster
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x0226135726f346bcb3f8cb73365ed382


3.
Thread 8 (Thread 0x7fad832dd700 (LWP 30057)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x7fad88834f3e in __afr_shd_healer_wait () from 
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/cluster/replicate.so
#2  0x7fad88834fad in afr_shd_healer_wait () from 
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/cluster/replicate.so
#3  0x7fad88835aa0 in afr_shd_index_healer () from 
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/cluster/replicate.so
#4  0x7fad8df4270a in start_thread (arg=0x7fad832dd700) at 
pthread_create.c:333
#5  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 7 (Thread 0x7fad83ade700 (LWP 30056)):
#0  0x7fad8dc78e23 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84
#1  0x7fad8e808a58 in ?? () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2  0x7fad8df4270a in start_thread (arg=0x7fad83ade700) at 
pthread_create.c:333
#3  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 6 (Thread 0x7fad894a5700 (LWP 30055)):
#0  0x7fad8dc78e23 in epoll_wait () at 
../sysdeps/unix/syscall-template.S:84
#1  0x7fad8e808a58 in ?? () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2  0x7fad8df4270a in start_thread (arg=0x7fad894a5700) at 
pthread_create.c:333
#3  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 5 (Thread 0x7fad8a342700 (LWP 30054)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x7fad8e7ecd98 in syncenv_task () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2  0x7fad8e7ed970 in syncenv_processor () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#3  0x7fad8df4270a in start_thread (arg=0x7fad8a342700) at 
pthread_create.c:333
#4  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 4 (Thread 0x7fad8ab43700 (LWP 30053)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x7fad8e7ecd98 in syncenv_task () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2  0x7fad8e7ed970 in syncenv_processor () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#3  0x7fad8df4270a in start_thread (arg=0x7fad8ab43700) at 
pthread_create.c:333
#4  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 3 (Thread 0x7fad8b344700 (LWP 30052)):
#0  do_sigwait (sig=0x7fad8b343e3c, set=) at 
../sysdeps/unix/sysv/linux/sigwait.c:64
#1  __sigwait (set=, sig=0x7fad8b343e3c) at 
../sysdeps/unix/sysv/linux/sigwait.c:96

#2  0x004080bf in glusterfs_sigwaiter ()
#3  0x7fad8df4270a in start_thread (arg=0x7fad8b344700) at 
pthread_create.c:333
#4  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 2 (Thread 0x7fad8bb45700 (LWP 30051)):
#0  0x7fad8df4bc6d in nanosleep () at 
../sysdeps/unix/syscall-template.S:84
#1  0x7fad8e7ca744 in gf_timer_proc () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2  0x7fad8df4270a in start_thread (arg=0x7fad8bb45700) at 
pthread_create.c:333
#3  0x7fad8dc7882d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Thread 1 (Thread 0x7fad8ec66780 (LWP 30050)):
#0  0x7fad8df439dd in pthread_join (threadid=140383309420288, 
thread_return=0x0) at pthread_join.c:90
#1  0x7fad8e808eeb in ?? () from 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0

#2  0x00405501 in main ()


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 16:17, Ravishankar N wrote:

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>> wrote:

Ah, damn! I found the issue. On the storage server, the storage2
IP address was wrong, I inversed two digits in the /etc/hosts
file, sorry for that :(

I was able to add the brick now, I started the heal, but still

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Ravishankar N 

On 12/08/2016 06:53 PM, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI 
mailto:cuculo...@mdpi.com>> wrote:


Ah, damn! I found the issue. On the storage server, the storage2
IP address was wrong, I inversed two digits in the /etc/hosts
file, sorry for that :(

I was able to add the brick now, I started the heal, but still no
data transfer visible.


1. Are the files getting created on the new brick though?
2. Can you provide the output of `getfattr -d -m . -e hex 
/data/data-cluster` on both bricks?
3. Is it possible to attach gdb to the self-heal daemon on the original 
(old) brick and get a backtrace?

  `gdb -p `
 thread apply all bt  -->share this output
quit gdb.


-Ravi


@Ravi/Pranith - can you help here?


By doing gluster volume status, I have

Status of volume: storage
Gluster process   TCP Port  RDMA Port  Online  Pid

--
Brick storage2:/data/data-cluster 49152 0 Y   23101
Brick storage:/data/data-cluster  49152 0 Y   30773
Self-heal Daemon on localhost N/A   N/A Y   30050
Self-heal Daemon on storage   N/A   N/A Y   30792


Any idea?

On storage I have:
Number of Peers: 1

Hostname: 195.65.194.217
Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
State: Peer in Cluster (Connected)


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 13:55, Atin Mukherjee wrote:

Can you resend the attachment as zip? I am unable to extract the
content? We shouldn't have 0 info file. What does gluster peer
status
output say?

On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

I hope you received my last email Atin, thank you!

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
>
Skype: milos.cuculovic.mdpi

On 08.12.2016 10:28, Atin Mukherjee wrote:


-- Forwarded message --
From: *Atin Mukherjee* mailto:amukh...@redhat.com>
> 

>


>


>
 
>

 >
 From the log snippet:

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Miloš Čučulović - MDPI 

A note to add, when checking the
sudo gluster volume heal storage info

I am  getting more than 2k Number of entries, but no file on storage 
server (replica).


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

On 08.12.2016 14:23, Atin Mukherjee wrote:



On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>> wrote:

Ah, damn! I found the issue. On the storage server, the storage2 IP
address was wrong, I inversed two digits in the /etc/hosts file,
sorry for that :(

I was able to add the brick now, I started the heal, but still no
data transfer visible.


@Ravi/Pranith - can you help here?



By doing gluster volume status, I have

Status of volume: storage
Gluster process   TCP Port  RDMA Port  Online  Pid

--
Brick storage2:/data/data-cluster 49152 0  Y   23101
Brick storage:/data/data-cluster  49152 0  Y   30773
Self-heal Daemon on localhost N/A   N/AY   30050
Self-heal Daemon on storage   N/A   N/AY   30792


Any idea?

On storage I have:
Number of Peers: 1

Hostname: 195.65.194.217
Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
State: Peer in Cluster (Connected)


- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
Skype: milos.cuculovic.mdpi

On 08.12.2016 13:55, Atin Mukherjee wrote:

Can you resend the attachment as zip? I am unable to extract the
content? We shouldn't have 0 info file. What does gluster peer
status
output say?

On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
mailto:cuculo...@mdpi.com>
>> wrote:

I hope you received my last email Atin, thank you!

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com 
>
Skype: milos.cuculovic.mdpi

On 08.12.2016 10:28, Atin Mukherjee wrote:


-- Forwarded message --
From: *Atin Mukherjee* mailto:amukh...@redhat.com>
> 

> 

>


>


>


>

Re: [Gluster-users] Fwd: Replica brick not working

2016-12-08 Thread Atin Mukherjee 
On Thu, Dec 8, 2016 at 6:44 PM, Miloš Čučulović - MDPI 
wrote:

> Ah, damn! I found the issue. On the storage server, the storage2 IP
> address was wrong, I inversed two digits in the /etc/hosts file, sorry for
> that :(
>
> I was able to add the brick now, I started the heal, but still no data
> transfer visible.
>

@Ravi/Pranith - can you help here?


>
> By doing gluster volume status, I have
>
> Status of volume: storage
> Gluster process   TCP Port  RDMA Port  Online  Pid
> 
> --
> Brick storage2:/data/data-cluster 49152 0  Y   23101
> Brick storage:/data/data-cluster  49152 0  Y   30773
> Self-heal Daemon on localhost N/A   N/AY   30050
> Self-heal Daemon on storage   N/A   N/AY   30792
>
>
> Any idea?
>
> On storage I have:
> Number of Peers: 1
>
> Hostname: 195.65.194.217
> Uuid: 7c988af2-9f76-4843-8e6f-d94866d57bb0
> State: Peer in Cluster (Connected)
>
>
> - Kindest regards,
>
> Milos Cuculovic
> IT Manager
>
> ---
> MDPI AG
> Postfach, CH-4020 Basel, Switzerland
> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
> Tel. +41 61 683 77 35
> Fax +41 61 302 89 18
> Email: cuculo...@mdpi.com
> Skype: milos.cuculovic.mdpi
>
> On 08.12.2016 13:55, Atin Mukherjee wrote:
>
>> Can you resend the attachment as zip? I am unable to extract the
>> content? We shouldn't have 0 info file. What does gluster peer status
>> output say?
>>
>> On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI
>> mailto:cuculo...@mdpi.com>> wrote:
>>
>> I hope you received my last email Atin, thank you!
>>
>> - Kindest regards,
>>
>> Milos Cuculovic
>> IT Manager
>>
>> ---
>> MDPI AG
>> Postfach, CH-4020 Basel, Switzerland
>> Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
>> Tel. +41 61 683 77 35
>> Fax +41 61 302 89 18
>> Email: cuculo...@mdpi.com 
>> Skype: milos.cuculovic.mdpi
>>
>> On 08.12.2016 10:28, Atin Mukherjee wrote:
>>
>>
>> -- Forwarded message --
>> From: *Atin Mukherjee* >  > >>
>> Date: Thu, Dec 8, 2016 at 11:56 AM
>> Subject: Re: [Gluster-users] Replica brick not working
>> To: Ravishankar N >  > >>
>> Cc: Miloš Čučulović - MDPI > 
>> >>,
>> Pranith Kumar Karampuri
>> mailto:pkara...@redhat.com>
>> >>,
>> gluster-users
>> mailto:gluster-users@gluster.org>
>> > >>
>>
>>
>>
>>
>> On Thu, Dec 8, 2016 at 11:11 AM, Ravishankar N
>> mailto:ravishan...@redhat.com>
>> >>
>>
>> wrote:
>>
>> On 12/08/2016 10:43 AM, Atin Mukherjee wrote:
>>
>> >From the log snippet:
>>
>> [2016-12-07 09:15:35.677645] I [MSGID: 106482]
>> [glusterd-brick-ops.c:442:__glusterd_handle_add_brick]
>> 0-management: Received add brick req
>> [2016-12-07 09:15:35.677708] I [MSGID: 106062]
>> [glusterd-brick-ops.c:494:__glusterd_handle_add_brick]
>> 0-management: replica-count is 2
>> [2016-12-07 09:15:35.677735] E [MSGID: 106291]
>> [glusterd-brick-ops.c:614:__glusterd_handle_add_brick]
>> 0-management:
>>
>> The last log entry indicates that we hit the code path in
>> gd_addbr_validate_replica_count ()
>>
>> if (replica_count ==
>> volinfo->replica_count) {
>> if (!(total_bricks %
>> volinfo->dist_leaf_count)) {
>> ret = 1;
>> goto out;
>> }
>> }
>>
>>
>> It seems unlikely that this snippet was hit because we print
>> the E
>> [MSGID: 106291] in the above message only if ret==-1.
>> gd_addbr_validate_replica_count() returns -1 and yet not
>> populates
>> err_str only when in volinfo->type doesn't match any of the
>> known
>> volume types, so volinfo->type is corrupted perhaps?
>>
>>
>> You are right, I missed that ret is set to 1 here in the above
>> snippet.
>>
>> @Milos - Can you please provide us the volume in