Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Can you send logs as Pranith requested?
On Fri, May 20, 2011 at 11:06 AM, Martin Schenker
wrote:
> No, all files are running VMs. No-one alters them manually (which would kill
> the VM...)
>
> So, all was done by the replicate mechanism and the sync. We have to reboot
> servers from time to time for upgrades, but we do bring the back up with the
> Gluster running before tackling a second server.
>
> Best, Martin
>
> -Original Message-
> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
> Sent: Thursday, May 19, 2011 7:05 PM
> To: Pranith Kumar. Karampuri; Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>
> What's more interesting is that pserver3 shows "0" bytes and rest 3 of
> them show the same "size". While pserver12 & 13 has
> "trusted.glusterfs.dht.linkto="storage0-replicate-0" set.
>
> Was there every any manual operation done with these files?
>
> On Thu, May 19, 2011 at 5:16 AM, Pranith Kumar. Karampuri
> wrote:
>> Need the logs from May 13th to 17th.
>>
>> Pranith.
>> - Original Message -
>> From: "Martin Schenker"
>> To: "Pranith Kumar. Karampuri"
>> Cc: gluster-users@gluster.org
>> Sent: Thursday, May 19, 2011 5:28:06 PM
>> Subject: RE: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>>
>> Hi Pranith!
>>
>> That's what I would have expected as well! The files should be on one
> brick. But they appear on both.
>> I'm quite stumped WHY the files show up on the other brick, this isn't
> what I understood from the manual/setup! The vol-file doesn't seem to be
> wrong so any ideas?
>>
>> Best, Martin
>>
>>
>>
>> -Original Message-
>> From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com]
>> Sent: Thursday, May 19, 2011 1:52 PM
>> To: Martin Schenker
>> Cc: gluster-users@gluster.org
>> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>>
>> Martin,
>> The output suggests that there are 2 replicas per 1 volume. So it
> should be present on only 2 bricks. Why is the file present in 4 bricks?. It
> should either be present on pserver12&13 or pserver3 & 5. I am not sure why
> you are expecting it to be there on 4 bricks.
>> Am I missing any info here?.
>>
>> Pranith
>>
>> - Original Message -
>> From: "Martin Schenker"
>> To: gluster-users@gluster.org
>> Sent: Wednesday, May 18, 2011 2:23:09 PM
>> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>>
>> Here is another occurrence:
>>
>> The file 20819 is shown twice, different timestamps and attributes. 0
>> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in
> sync.
>> So what's going on?
>>
>>
>> 0 root@de-dc1-c1-pserver13:~ # ls -al
>>
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/2081*
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>>
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/20819
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>>
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/20819
>>
>> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 |
> xargs
>> -i ls -al {}
>> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
>> hdd-images/20819
>> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>>
>> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
> -al
>> {}
>> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@pserver5:~ # getfattr
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
No, all files are running VMs. No-one alters them manually (which would kill
the VM...)
So, all was done by the replicate mechanism and the sync. We have to reboot
servers from time to time for upgrades, but we do bring the back up with the
Gluster running before tackling a second server.
Best, Martin
-Original Message-
From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
Sent: Thursday, May 19, 2011 7:05 PM
To: Pranith Kumar. Karampuri; Martin Schenker
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
What's more interesting is that pserver3 shows "0" bytes and rest 3 of
them show the same "size". While pserver12 & 13 has
"trusted.glusterfs.dht.linkto="storage0-replicate-0" set.
Was there every any manual operation done with these files?
On Thu, May 19, 2011 at 5:16 AM, Pranith Kumar. Karampuri
wrote:
> Need the logs from May 13th to 17th.
>
> Pranith.
> - Original Message -
> From: "Martin Schenker"
> To: "Pranith Kumar. Karampuri"
> Cc: gluster-users@gluster.org
> Sent: Thursday, May 19, 2011 5:28:06 PM
> Subject: RE: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
>
> Hi Pranith!
>
> That's what I would have expected as well! The files should be on one
brick. But they appear on both.
> I'm quite stumped WHY the files show up on the other brick, this isn't
what I understood from the manual/setup! The vol-file doesn't seem to be
wrong so any ideas?
>
> Best, Martin
>
>
>
> -Original Message-
> From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com]
> Sent: Thursday, May 19, 2011 1:52 PM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
>
> Martin,
> The output suggests that there are 2 replicas per 1 volume. So it
should be present on only 2 bricks. Why is the file present in 4 bricks?. It
should either be present on pserver12&13 or pserver3 & 5. I am not sure why
you are expecting it to be there on 4 bricks.
> Am I missing any info here?.
>
> Pranith
>
> ----- Original Message -
> From: "Martin Schenker"
> To: gluster-users@gluster.org
> Sent: Wednesday, May 18, 2011 2:23:09 PM
> Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
>
> Here is another occurrence:
>
> The file 20819 is shown twice, different timestamps and attributes. 0
> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in
sync.
> So what's going on?
>
>
> 0 root@de-dc1-c1-pserver13:~ # ls -al
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/2081*
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
>
> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 |
xargs
> -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
>
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
-al
> {}
> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver5:~ # getfattr -dm -
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
>
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-0=0sAgIA
> trusted.afr.storage0-client-1=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
-al
> {}
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
What's more interesting is that pserver3 shows "0" bytes and rest 3 of
them show the same "size". While pserver12 & 13 has
"trusted.glusterfs.dht.linkto="storage0-replicate-0" set.
Was there every any manual operation done with these files?
On Thu, May 19, 2011 at 5:16 AM, Pranith Kumar. Karampuri
wrote:
> Need the logs from May 13th to 17th.
>
> Pranith.
> - Original Message -
> From: "Martin Schenker"
> To: "Pranith Kumar. Karampuri"
> Cc: gluster-users@gluster.org
> Sent: Thursday, May 19, 2011 5:28:06 PM
> Subject: RE: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>
> Hi Pranith!
>
> That's what I would have expected as well! The files should be on one brick.
> But they appear on both.
> I'm quite stumped WHY the files show up on the other brick, this isn't what I
> understood from the manual/setup! The vol-file doesn't seem to be wrong so
> any ideas?
>
> Best, Martin
>
>
>
> -Original Message-
> From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com]
> Sent: Thursday, May 19, 2011 1:52 PM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>
> Martin,
> The output suggests that there are 2 replicas per 1 volume. So it should
> be present on only 2 bricks. Why is the file present in 4 bricks?. It should
> either be present on pserver12&13 or pserver3 & 5. I am not sure why you are
> expecting it to be there on 4 bricks.
> Am I missing any info here?.
>
> Pranith
>
> ----- Original Message -
> From: "Martin Schenker"
> To: gluster-users@gluster.org
> Sent: Wednesday, May 18, 2011 2:23:09 PM
> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>
> Here is another occurrence:
>
> The file 20819 is shown twice, different timestamps and attributes. 0
> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in sync.
> So what's going on?
>
>
> 0 root@de-dc1-c1-pserver13:~ # ls -al
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/2081*
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
>
> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
> -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
> {}
> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver5:~ # getfattr -dm -
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-0=0sAgIA
> trusted.afr.storage0-client-1=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
> {}
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver12:~ # getfattr -dm -
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-6=0s
> trusted.afr.storage0-client-7=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf3
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Need the logs from May 13th to 17th.
Pranith.
- Original Message -
From: "Martin Schenker"
To: "Pranith Kumar. Karampuri"
Cc: gluster-users@gluster.org
Sent: Thursday, May 19, 2011 5:28:06 PM
Subject: RE: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
Hi Pranith!
That's what I would have expected as well! The files should be on one brick.
But they appear on both.
I'm quite stumped WHY the files show up on the other brick, this isn't what I
understood from the manual/setup! The vol-file doesn't seem to be wrong so any
ideas?
Best, Martin
-Original Message-
From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com]
Sent: Thursday, May 19, 2011 1:52 PM
To: Martin Schenker
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Client and server file "view", different results?!
Client can't see the right file.
Martin,
The output suggests that there are 2 replicas per 1 volume. So it should
be present on only 2 bricks. Why is the file present in 4 bricks?. It should
either be present on pserver12&13 or pserver3 & 5. I am not sure why you are
expecting it to be there on 4 bricks.
Am I missing any info here?.
Pranith
- Original Message -
From: "Martin Schenker"
To: gluster-users@gluster.org
Sent: Wednesday, May 18, 2011 2:23:09 PM
Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
Here is another occurrence:
The file 20819 is shown twice, different timestamps and attributes. 0
filesize on pserver3, outdated on pserver5, just 12&13 seems to be in sync.
So what's going on?
0 root@de-dc1-c1-pserver13:~ # ls -al
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/2081*
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver5:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-0=0sAgIA
trusted.afr.storage0-client-1=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver12:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:39
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver13:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
Only entrance in log file on pserver5, no references in t
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Hi Pranith!
That's what I would have expected as well! The files should be on one brick.
But they appear on both.
I'm quite stumped WHY the files show up on the other brick, this isn't what I
understood from the manual/setup! The vol-file doesn't seem to be wrong so any
ideas?
Best, Martin
-Original Message-
From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com]
Sent: Thursday, May 19, 2011 1:52 PM
To: Martin Schenker
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Client and server file "view", different results?!
Client can't see the right file.
Martin,
The output suggests that there are 2 replicas per 1 volume. So it should
be present on only 2 bricks. Why is the file present in 4 bricks?. It should
either be present on pserver12&13 or pserver3 & 5. I am not sure why you are
expecting it to be there on 4 bricks.
Am I missing any info here?.
Pranith
- Original Message -
From: "Martin Schenker"
To: gluster-users@gluster.org
Sent: Wednesday, May 18, 2011 2:23:09 PM
Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
Here is another occurrence:
The file 20819 is shown twice, different timestamps and attributes. 0
filesize on pserver3, outdated on pserver5, just 12&13 seems to be in sync.
So what's going on?
0 root@de-dc1-c1-pserver13:~ # ls -al
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/2081*
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver5:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-0=0sAgIA
trusted.afr.storage0-client-1=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver12:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:39
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver13:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
Only entrance in log file on pserver5, no references in the other three
logs/servers:
0 root@pserver5:~ # grep 20819
/var/log/glusterfs/opt-profitbricks-storage.log
[2011-05-17 20:37:30.52535] I [client-handshake.c:407:client3_1_reopen_cbk]
0-storage0-client-7: reopen on
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819 succeeded
(remote-fd = 6)
[2011-05-17 20:37:34.824934] I [afr-open.c:435:afr_openfd_
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Martin,
The output suggests that there are 2 replicas per 1 volume. So it should
be present on only 2 bricks. Why is the file present in 4 bricks?. It should
either be present on pserver12&13 or pserver3 & 5. I am not sure why you are
expecting it to be there on 4 bricks.
Am I missing any info here?.
Pranith
- Original Message -
From: "Martin Schenker"
To: gluster-users@gluster.org
Sent: Wednesday, May 18, 2011 2:23:09 PM
Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
Here is another occurrence:
The file 20819 is shown twice, different timestamps and attributes. 0
filesize on pserver3, outdated on pserver5, just 12&13 seems to be in sync.
So what's going on?
0 root@de-dc1-c1-pserver13:~ # ls -al
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/2081*
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver5:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-0=0sAgIA
trusted.afr.storage0-client-1=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver12:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:39
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver13:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
Only entrance in log file on pserver5, no references in the other three
logs/servers:
0 root@pserver5:~ # grep 20819
/var/log/glusterfs/opt-profitbricks-storage.log
[2011-05-17 20:37:30.52535] I [client-handshake.c:407:client3_1_reopen_cbk]
0-storage0-client-7: reopen on
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819 succeeded
(remote-fd = 6)
[2011-05-17 20:37:34.824934] I [afr-open.c:435:afr_openfd_sh]
0-storage0-replicate-3: data self-heal triggered. path:
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819, reason:
Replicate up down flush, data lock is held
[2011-05-17 20:37:34.825557] E
[afr-self-heal-common.c:1214:sh_missing_entries_create]
0-storage0-replicate-3: no missing files -
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819.
proceeding to metadata check
[2011-05-17 21:08:59.241203] I
[afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]
0-storage0-replicate-3: diff self-heal on
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819: 6 blocks
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
hi Martin, Manually copying the file may lead to same files with different gfids, which will again lead to the same problem. Make sure that the "trusted.gfid" attribute values are same on this file on both the replicas. Pranith - Original Message - From: "Martin Schenker" To: "Pranith Kumar. Karampuri" Cc: gluster-users@gluster.org Sent: Tuesday, May 17, 2011 3:49:30 PM Subject: RE: [Gluster-users] Client and server file "view", different results?! Client can't see the right file. It version 3.1.3 (we tried 3.2.0 for about 10h and rolled back) Unfortunateltly the file view was "repaired" already by brutally copying manually from the correct /mnt (server) mountpoint to the /opt (client) mount which fixed the situation for now. We needed the files accessible ASAP. Best, Martin > -Original Message- > From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com] > Sent: Tuesday, May 17, 2011 10:41 AM > To: Martin Schenker > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Client and server file "view", > different results?! Client can't see the right file. > > > hi Martin, > Could you please gather the following outputs so that we > can debug as to what is happening: > 1) whats the version of the gluster. > 2) backend "ls -l" of files in question on all bricks that > file is replicated on. > 3) 'ls -l" o/p from mnt point for that file. > > Thanks > Pranith > - Original Message - > From: "Martin Schenker" > To: "Pranith Kumar. Karampuri" > Cc: gluster-users@gluster.org > Sent: Tuesday, May 17, 2011 11:13:32 AM > Subject: RE: [Gluster-users] Client and server file "view", > different results?! Client can't see the right file. > > Yes, it is! > > Here's the volfile: > > cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol: > > volume storage0-client-0 > type protocol/client > option remote-host de-dc1-c1-pserver3 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-1 > type protocol/client > option remote-host de-dc1-c1-pserver5 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-2 > type protocol/client > option remote-host de-dc1-c1-pserver3 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-3 > type protocol/client > option remote-host de-dc1-c1-pserver5 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-4 > type protocol/client > option remote-host de-dc1-c1-pserver12 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-5 > type protocol/client > option remote-host de-dc1-c1-pserver13 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-6 > type protocol/client > option remote-host de-dc1-c1-pserver12 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-7 > type protocol/client > option remote-host de-dc1-c1-pserver13 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-replicate-0 > type cluster/replicate > subvolumes storage0-client-0 storage0-client-1 > end-volume > > volume storage0-replicate-1 > type cluster/replicate > subvolumes storage0-client-2 storage0-client-3 > end-volume > > volume storage0-replicate-2 > type cluster/replicate > subvolumes storage0-client-4 storage0-client-5 > end-volume > > volume storage0-replicate-3 > type cluster/replicate > subvolumes storage0-client-6 storage0-client-7 > end-volume > > volume storage0-dht > type cluster/distribute > subvolumes storage0-replicate-0 storage0-replicate-1 > storage0-replicate-2 storage0-replicate-3 end-volume > > volume storage0-write-behind > type performance/write-behind > subvolumes storage0-dht > end-vo
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
No, we had these issues before already, running on 3.1.3
Only the system load has gone up a lot in the meantime...
Best, Martin
-Original Message-
From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
Sent: Wednesday, May 18, 2011 10:36 PM
To: Martin Schenker
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
So you started seeing this issue after rolling it back to 3.1.3?
On Wed, May 18, 2011 at 1:30 PM, Martin Schenker
wrote:
> We're running 3.1.3. we had a brief test of 3.2.0 and rolled back to 3.1.3
> by reinstalling the Debian package.
>
> 0 root@pserver12:~ # gluster volume info all
>
> Volume Name: storage0
> Type: Distributed-Replicate
> Status: Started
> Number of Bricks: 4 x 2 = 8
> Transport-type: rdma
> Bricks:
> Brick1: de-dc1-c1-pserver3:/mnt/gluster/brick0/storage
> Brick2: de-dc1-c1-pserver5:/mnt/gluster/brick0/storage
> Brick3: de-dc1-c1-pserver3:/mnt/gluster/brick1/storage
> Brick4: de-dc1-c1-pserver5:/mnt/gluster/brick1/storage
> Brick5: de-dc1-c1-pserver12:/mnt/gluster/brick0/storage
> Brick6: de-dc1-c1-pserver13:/mnt/gluster/brick0/storage
> Brick7: de-dc1-c1-pserver12:/mnt/gluster/brick1/storage
> Brick8: de-dc1-c1-pserver13:/mnt/gluster/brick1/storage
> Options Reconfigured:
> network.ping-timeout: 5
> nfs.disable: on
> performance.cache-size: 4096MB
>
> Best, Martin
>
> -Original Message-
> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
> Sent: Wednesday, May 18, 2011 9:43 PM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>
> Which version are you running? Can you also post output from volume info?
>
> Meanwhile, anyone from dev want to answer??
>
> On Wed, May 18, 2011 at 1:53 AM, Martin Schenker
> wrote:
>> Here is another occurrence:
>>
>> The file 20819 is shown twice, different timestamps and attributes. 0
>> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in
> sync.
>> So what's going on?
>>
>>
>> 0 root@de-dc1-c1-pserver13:~ # ls -al
>>
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/2081*
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>>
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/20819
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>>
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/20819
>>
>> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 |
> xargs
>> -i ls -al {}
>> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
>>
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
>>
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
>
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
>> hdd-images/20819
>> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>>
>> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
> -al
>> {}
>> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
>>
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@pserver5:~ # getfattr -dm -
>>
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
>
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
>> hdd-images/20819
>> trusted.afr.storage0-client-0=0sAgIA
>> trusted.afr.storage0-client-1=0s
>> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>>
>> 0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
> -al
>> {}
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
>>
>
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@pserver12:~ # getfattr -dm -
>>
>
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
>
mnt/gluster/brick1/storage/i
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
So you started seeing this issue after rolling it back to 3.1.3?
On Wed, May 18, 2011 at 1:30 PM, Martin Schenker
wrote:
> We're running 3.1.3. we had a brief test of 3.2.0 and rolled back to 3.1.3
> by reinstalling the Debian package.
>
> 0 root@pserver12:~ # gluster volume info all
>
> Volume Name: storage0
> Type: Distributed-Replicate
> Status: Started
> Number of Bricks: 4 x 2 = 8
> Transport-type: rdma
> Bricks:
> Brick1: de-dc1-c1-pserver3:/mnt/gluster/brick0/storage
> Brick2: de-dc1-c1-pserver5:/mnt/gluster/brick0/storage
> Brick3: de-dc1-c1-pserver3:/mnt/gluster/brick1/storage
> Brick4: de-dc1-c1-pserver5:/mnt/gluster/brick1/storage
> Brick5: de-dc1-c1-pserver12:/mnt/gluster/brick0/storage
> Brick6: de-dc1-c1-pserver13:/mnt/gluster/brick0/storage
> Brick7: de-dc1-c1-pserver12:/mnt/gluster/brick1/storage
> Brick8: de-dc1-c1-pserver13:/mnt/gluster/brick1/storage
> Options Reconfigured:
> network.ping-timeout: 5
> nfs.disable: on
> performance.cache-size: 4096MB
>
> Best, Martin
>
> -Original Message-
> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
> Sent: Wednesday, May 18, 2011 9:43 PM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Client and server file "view", different
> results?! Client can't see the right file.
>
> Which version are you running? Can you also post output from volume info?
>
> Meanwhile, anyone from dev want to answer??
>
> On Wed, May 18, 2011 at 1:53 AM, Martin Schenker
> wrote:
>> Here is another occurrence:
>>
>> The file 20819 is shown twice, different timestamps and attributes. 0
>> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in
> sync.
>> So what's going on?
>>
>>
>> 0 root@de-dc1-c1-pserver13:~ # ls -al
>>
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/2081*
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>>
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/20819
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>>
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
>> dd-images/20819
>>
>> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 |
> xargs
>> -i ls -al {}
>> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
>> hdd-images/20819
>> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>>
>> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
> -al
>> {}
>> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@pserver5:~ # getfattr -dm -
>>
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
>> hdd-images/20819
>> trusted.afr.storage0-client-0=0sAgIA
>> trusted.afr.storage0-client-1=0s
>> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>>
>> 0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
> -al
>> {}
>> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
>>
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> 0 root@pserver12:~ # getfattr -dm -
>>
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
>> /hdd-images/20819
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>>
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
>> hdd-images/20819
>> trusted.afr.storage0-client-6=0s
>> trusted.afr.storage0-client-7=0s
>> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>> trusted.glusterfs.dht.linkto="storage0-replicate-0
>>
>> 0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 |
> xargs
>> -i ls -al {
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
We're running 3.1.3. we had a brief test of 3.2.0 and rolled back to 3.1.3
by reinstalling the Debian package.
0 root@pserver12:~ # gluster volume info all
Volume Name: storage0
Type: Distributed-Replicate
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: rdma
Bricks:
Brick1: de-dc1-c1-pserver3:/mnt/gluster/brick0/storage
Brick2: de-dc1-c1-pserver5:/mnt/gluster/brick0/storage
Brick3: de-dc1-c1-pserver3:/mnt/gluster/brick1/storage
Brick4: de-dc1-c1-pserver5:/mnt/gluster/brick1/storage
Brick5: de-dc1-c1-pserver12:/mnt/gluster/brick0/storage
Brick6: de-dc1-c1-pserver13:/mnt/gluster/brick0/storage
Brick7: de-dc1-c1-pserver12:/mnt/gluster/brick1/storage
Brick8: de-dc1-c1-pserver13:/mnt/gluster/brick1/storage
Options Reconfigured:
network.ping-timeout: 5
nfs.disable: on
performance.cache-size: 4096MB
Best, Martin
-Original Message-
From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
Sent: Wednesday, May 18, 2011 9:43 PM
To: Martin Schenker
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Client and server file "view", different
results?! Client can't see the right file.
Which version are you running? Can you also post output from volume info?
Meanwhile, anyone from dev want to answer??
On Wed, May 18, 2011 at 1:53 AM, Martin Schenker
wrote:
> Here is another occurrence:
>
> The file 20819 is shown twice, different timestamps and attributes. 0
> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in
sync.
> So what's going on?
>
>
> 0 root@de-dc1-c1-pserver13:~ # ls -al
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/2081*
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
>
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
>
> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 |
xargs
> -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
>
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
-al
> {}
> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver5:~ # getfattr -dm -
>
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
>
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-0=0sAgIA
> trusted.afr.storage0-client-1=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls
-al
> {}
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
>
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver12:~ # getfattr -dm -
>
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
>
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-6=0s
> trusted.afr.storage0-client-7=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
> trusted.glusterfs.dht.linkto="storage0-replicate-0
>
> 0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 |
xargs
> -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:39
>
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@de-dc1-c1-pserver13:~ # getfattr -dm -
>
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
>
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-6=0s
> trusted.afr.storage0-client-7=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
> trus
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Which version are you running? Can you also post output from volume info?
Meanwhile, anyone from dev want to answer??
On Wed, May 18, 2011 at 1:53 AM, Martin Schenker
wrote:
> Here is another occurrence:
>
> The file 20819 is shown twice, different timestamps and attributes. 0
> filesize on pserver3, outdated on pserver5, just 12&13 seems to be in sync.
> So what's going on?
>
>
> 0 root@de-dc1-c1-pserver13:~ # ls -al
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/2081*
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
> dd-images/20819
>
> 0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
> -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
> {}
> -rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver5:~ # getfattr -dm -
> /mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-0=0sAgIA
> trusted.afr.storage0-client-1=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
>
> 0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
> {}
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@pserver12:~ # getfattr -dm -
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-6=0s
> trusted.afr.storage0-client-7=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
> trusted.glusterfs.dht.linkto="storage0-replicate-0
>
> 0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
> -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:39
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> 0 root@de-dc1-c1-pserver13:~ # getfattr -dm -
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
> /hdd-images/20819
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
> hdd-images/20819
> trusted.afr.storage0-client-6=0s
> trusted.afr.storage0-client-7=0s
> trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
> trusted.glusterfs.dht.linkto="storage0-replicate-0
>
> Only entrance in log file on pserver5, no references in the other three
> logs/servers:
>
> 0 root@pserver5:~ # grep 20819
> /var/log/glusterfs/opt-profitbricks-storage.log
> [2011-05-17 20:37:30.52535] I [client-handshake.c:407:client3_1_reopen_cbk]
> 0-storage0-client-7: reopen on
> /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819 succeeded
> (remote-fd = 6)
> [2011-05-17 20:37:34.824934] I [afr-open.c:435:afr_openfd_sh]
> 0-storage0-replicate-3: data self-heal triggered. path:
> /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819, reason:
> Replicate up down flush, data lock is held
> [2011-05-17 20:37:34.825557] E
> [afr-self-heal-common.c:1214:sh_missing_entries_create]
> 0-storage0-replicate-3: no missing files -
> /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819.
> proceeding to metadata check
> [2011-05-17 21:08:59.241203] I
> [afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]
> 0-storage0-replicate-3: diff self-heal on
> /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819: 6 blocks
> of 409600 were different (0.00%)
> [2011-05-17 21:08:59.275873] I
> [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk]
> 0-storage0-replicate-3: background data self-heal completed on
> /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819
>
>
> __
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Here is another occurrence:
The file 20819 is shown twice, different timestamps and attributes. 0
filesize on pserver3, outdated on pserver5, just 12&13 seems to be in sync.
So what's going on?
0 root@de-dc1-c1-pserver13:~ # ls -al
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/2081*
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:44
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20819
0 root@de-dc1-c1-pserver3:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu vcb 0 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver3:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver5:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu vcb 53687091200 May 14 17:00
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver5:~ # getfattr -dm -
/mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick0/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-0=0sAgIA
trusted.afr.storage0-client-1=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
0 root@pserver12:~ # find /mnt/gluster/brick?/ -name 20819 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:41
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@pserver12:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
0 root@de-dc1-c1-pserver13:~ # find /mnt/gluster/brick?/ -name 20819 | xargs
-i ls -al {}
-rwxrwx--- 1 libvirt-qemu kvm 53687091200 May 18 08:39
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
0 root@de-dc1-c1-pserver13:~ # getfattr -dm -
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20819
getfattr: Removing leading '/' from absolute path names
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images/20819
trusted.afr.storage0-client-6=0s
trusted.afr.storage0-client-7=0s
trusted.gfid=0sa5/rvjUUQ3ibSf32O3izOw==
trusted.glusterfs.dht.linkto="storage0-replicate-0
Only entrance in log file on pserver5, no references in the other three
logs/servers:
0 root@pserver5:~ # grep 20819
/var/log/glusterfs/opt-profitbricks-storage.log
[2011-05-17 20:37:30.52535] I [client-handshake.c:407:client3_1_reopen_cbk]
0-storage0-client-7: reopen on
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819 succeeded
(remote-fd = 6)
[2011-05-17 20:37:34.824934] I [afr-open.c:435:afr_openfd_sh]
0-storage0-replicate-3: data self-heal triggered. path:
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819, reason:
Replicate up down flush, data lock is held
[2011-05-17 20:37:34.825557] E
[afr-self-heal-common.c:1214:sh_missing_entries_create]
0-storage0-replicate-3: no missing files -
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819.
proceeding to metadata check
[2011-05-17 21:08:59.241203] I
[afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]
0-storage0-replicate-3: diff self-heal on
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819: 6 blocks
of 409600 were different (0.00%)
[2011-05-17 21:08:59.275873] I
[afr-self-heal-common.c:1527:afr_self_heal_completion_cbk]
0-storage0-replicate-3: background data self-heal completed on
/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20819
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
On 05/17/2011 08:08 AM, Martin Schenker wrote: This is an inherited system, I guess it was set up by hand. I guess I can switch off these options, but the glusterd service will have to be restarted, right?!? Yes. I'm also getting current error messages like these on the peer pair 3&5: Pserver3 [2011-05-17 10:06:28.540355] E [rpc-clnt.c:199:call_bail] 0-storage0-client-2: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x805809xsent = 2011-05-17 09:36:18.393519. timeout = 1800 Hmmm ... Looks like others have seen this before. Error message suggests some sort of protocol error. Its in a code path in rpc/rpc-lib/src/rpc-clnt.c and the function is named "call_bail". This code looks like it is part of a timeout callback (I am guessing when it doesn't get a response in time, and the timer is hard coded to 10 seconds). There is a note there with a TODO about making that configurable. If the machine is under tremendous load, it is possible that a response is delayed more than 10 seconds, so that this portion of the code falls through to the timeout, rather than processing an rpc call). Pserver5 [2011-05-17 10:02:23.738887] E [dht-common.c:1873:dht_getxattr] 0-storage0-dht: layout is NULL [2011-05-17 10:02:23.738909] W [fuse-bridge.c:2499:fuse_xattr_cbk] 0-glusterfs-fuse: 489090: GETXATTR() /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da 6ef/hdd-images/21351 => -1 (No such file or directory) [2011-05-17 10:02:23.738954] W [fuse-bridge.c:660:fuse_setattr_cbk] 0-glusterfs-fuse: 489091: SETATTR() /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da 6ef/hdd-images/21351 => -1 (Invalid argument) Best, Martin -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Landman Sent: Tuesday, May 17, 2011 1:54 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file. On 05/17/2011 01:43 AM, Martin Schenker wrote: Yes, it is! Here's the volfile: cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol: volume storage0-client-0 type protocol/client option remote-host de-dc1-c1-pserver3 option remote-subvolume /mnt/gluster/brick0/storage option transport-type rdma option ping-timeout 5 end-volume Hmmm ... did you create these by hand or using the CLI? I noticed quick-read and stat-cache on. We recommend turning both of them off. We experienced many issues with them on (from gluster 3.x.y) -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
This is an inherited system, I guess it was set up by hand. I guess I can switch off these options, but the glusterd service will have to be restarted, right?!? I'm also getting current error messages like these on the peer pair 3&5: Pserver3 [2011-05-17 10:06:28.540355] E [rpc-clnt.c:199:call_bail] 0-storage0-client-2: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x805809xsent = 2011-05-17 09:36:18.393519. timeout = 1800 Pserver5 [2011-05-17 10:02:23.738887] E [dht-common.c:1873:dht_getxattr] 0-storage0-dht: layout is NULL [2011-05-17 10:02:23.738909] W [fuse-bridge.c:2499:fuse_xattr_cbk] 0-glusterfs-fuse: 489090: GETXATTR() /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da 6ef/hdd-images/21351 => -1 (No such file or directory) [2011-05-17 10:02:23.738954] W [fuse-bridge.c:660:fuse_setattr_cbk] 0-glusterfs-fuse: 489091: SETATTR() /images/2078/ebb83b05-3a83-9d18-ad8f-8542864da 6ef/hdd-images/21351 => -1 (Invalid argument) Best, Martin -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Landman Sent: Tuesday, May 17, 2011 1:54 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file. On 05/17/2011 01:43 AM, Martin Schenker wrote: > Yes, it is! > > Here's the volfile: > > cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol: > > volume storage0-client-0 > type protocol/client > option remote-host de-dc1-c1-pserver3 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume Hmmm ... did you create these by hand or using the CLI? I noticed quick-read and stat-cache on. We recommend turning both of them off. We experienced many issues with them on (from gluster 3.x.y) -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
On 05/17/2011 01:43 AM, Martin Schenker wrote: Yes, it is! Here's the volfile: cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol: volume storage0-client-0 type protocol/client option remote-host de-dc1-c1-pserver3 option remote-subvolume /mnt/gluster/brick0/storage option transport-type rdma option ping-timeout 5 end-volume Hmmm ... did you create these by hand or using the CLI? I noticed quick-read and stat-cache on. We recommend turning both of them off. We experienced many issues with them on (from gluster 3.x.y) -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
It version 3.1.3 (we tried 3.2.0 for about 10h and rolled back) Unfortunateltly the file view was "repaired" already by brutally copying manually from the correct /mnt (server) mountpoint to the /opt (client) mount which fixed the situation for now. We needed the files accessible ASAP. Best, Martin > -Original Message- > From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com] > Sent: Tuesday, May 17, 2011 10:41 AM > To: Martin Schenker > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Client and server file "view", > different results?! Client can't see the right file. > > > hi Martin, > Could you please gather the following outputs so that we > can debug as to what is happening: > 1) whats the version of the gluster. > 2) backend "ls -l" of files in question on all bricks that > file is replicated on. > 3) 'ls -l" o/p from mnt point for that file. > > Thanks > Pranith > - Original Message - > From: "Martin Schenker" > To: "Pranith Kumar. Karampuri" > Cc: gluster-users@gluster.org > Sent: Tuesday, May 17, 2011 11:13:32 AM > Subject: RE: [Gluster-users] Client and server file "view", > different results?! Client can't see the right file. > > Yes, it is! > > Here's the volfile: > > cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol: > > volume storage0-client-0 > type protocol/client > option remote-host de-dc1-c1-pserver3 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-1 > type protocol/client > option remote-host de-dc1-c1-pserver5 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-2 > type protocol/client > option remote-host de-dc1-c1-pserver3 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-3 > type protocol/client > option remote-host de-dc1-c1-pserver5 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-4 > type protocol/client > option remote-host de-dc1-c1-pserver12 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-5 > type protocol/client > option remote-host de-dc1-c1-pserver13 > option remote-subvolume /mnt/gluster/brick0/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-6 > type protocol/client > option remote-host de-dc1-c1-pserver12 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-client-7 > type protocol/client > option remote-host de-dc1-c1-pserver13 > option remote-subvolume /mnt/gluster/brick1/storage > option transport-type rdma > option ping-timeout 5 > end-volume > > volume storage0-replicate-0 > type cluster/replicate > subvolumes storage0-client-0 storage0-client-1 > end-volume > > volume storage0-replicate-1 > type cluster/replicate > subvolumes storage0-client-2 storage0-client-3 > end-volume > > volume storage0-replicate-2 > type cluster/replicate > subvolumes storage0-client-4 storage0-client-5 > end-volume > > volume storage0-replicate-3 > type cluster/replicate > subvolumes storage0-client-6 storage0-client-7 > end-volume > > volume storage0-dht > type cluster/distribute > subvolumes storage0-replicate-0 storage0-replicate-1 > storage0-replicate-2 storage0-replicate-3 end-volume > > volume storage0-write-behind > type performance/write-behind > subvolumes storage0-dht > end-volume > > volume storage0-read-ahead > type performance/read-ahead > subvolumes storage0-write-behind > end-volume > > volume storage0-io-cache > type performance/io-cache > option cache-size 4096MB > subvolumes storage0-read-ahead > end-volume > > volume storage0-quick-read > type performance/quick-read > option cache-size 4096MB > subvolumes storage0-io-cache > end-volume > > volume storage0-stat-prefetch > type performance/stat-prefetch > sub
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
hi Martin, Could you please gather the following outputs so that we can debug as to what is happening: 1) whats the version of the gluster. 2) backend "ls -l" of files in question on all bricks that file is replicated on. 3) 'ls -l" o/p from mnt point for that file. Thanks Pranith - Original Message - From: "Martin Schenker" To: "Pranith Kumar. Karampuri" Cc: gluster-users@gluster.org Sent: Tuesday, May 17, 2011 11:13:32 AM Subject: RE: [Gluster-users] Client and server file "view", different results?! Client can't see the right file. Yes, it is! Here's the volfile: cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol: volume storage0-client-0 type protocol/client option remote-host de-dc1-c1-pserver3 option remote-subvolume /mnt/gluster/brick0/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-1 type protocol/client option remote-host de-dc1-c1-pserver5 option remote-subvolume /mnt/gluster/brick0/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-2 type protocol/client option remote-host de-dc1-c1-pserver3 option remote-subvolume /mnt/gluster/brick1/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-3 type protocol/client option remote-host de-dc1-c1-pserver5 option remote-subvolume /mnt/gluster/brick1/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-4 type protocol/client option remote-host de-dc1-c1-pserver12 option remote-subvolume /mnt/gluster/brick0/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-5 type protocol/client option remote-host de-dc1-c1-pserver13 option remote-subvolume /mnt/gluster/brick0/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-6 type protocol/client option remote-host de-dc1-c1-pserver12 option remote-subvolume /mnt/gluster/brick1/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-client-7 type protocol/client option remote-host de-dc1-c1-pserver13 option remote-subvolume /mnt/gluster/brick1/storage option transport-type rdma option ping-timeout 5 end-volume volume storage0-replicate-0 type cluster/replicate subvolumes storage0-client-0 storage0-client-1 end-volume volume storage0-replicate-1 type cluster/replicate subvolumes storage0-client-2 storage0-client-3 end-volume volume storage0-replicate-2 type cluster/replicate subvolumes storage0-client-4 storage0-client-5 end-volume volume storage0-replicate-3 type cluster/replicate subvolumes storage0-client-6 storage0-client-7 end-volume volume storage0-dht type cluster/distribute subvolumes storage0-replicate-0 storage0-replicate-1 storage0-replicate-2 storage0-replicate-3 end-volume volume storage0-write-behind type performance/write-behind subvolumes storage0-dht end-volume volume storage0-read-ahead type performance/read-ahead subvolumes storage0-write-behind end-volume volume storage0-io-cache type performance/io-cache option cache-size 4096MB subvolumes storage0-read-ahead end-volume volume storage0-quick-read type performance/quick-read option cache-size 4096MB subvolumes storage0-io-cache end-volume volume storage0-stat-prefetch type performance/stat-prefetch subvolumes storage0-quick-read end-volume volume storage0 type debug/io-stats subvolumes storage0-stat-prefetch end-volume > -Original Message- > From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com] > Sent: Tuesday, May 17, 2011 7:16 AM > To: Martin Schenker > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Client and server file "view", > different results?! Client can't see the right file. > > > Martin, > Is this a distributed-replicate setup?. Could you > attach the vol-file of the client. > > Pranith > - Original Message - > From: "Martin Schenker" > To: gluster-users@gluster.org > Sent: Monday, May 16, 2011 2:49:29 PM > Subject: [Gluster-users] Client and server file "view", > different results?! Client can't see the right file. > > > Client and server file "view", different results?! Client > can't see the right file. > > Hi all! > > Here we have another mismatch between the client "view" and > the server mounts: > > From the server site everything seems well, the 20G file is > visible and the attributes seem to match: > > 0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr.&q
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Yes, it is!
Here's the volfile:
cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol:
volume storage0-client-0
type protocol/client
option remote-host de-dc1-c1-pserver3
option remote-subvolume /mnt/gluster/brick0/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-1
type protocol/client
option remote-host de-dc1-c1-pserver5
option remote-subvolume /mnt/gluster/brick0/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-2
type protocol/client
option remote-host de-dc1-c1-pserver3
option remote-subvolume /mnt/gluster/brick1/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-3
type protocol/client
option remote-host de-dc1-c1-pserver5
option remote-subvolume /mnt/gluster/brick1/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-4
type protocol/client
option remote-host de-dc1-c1-pserver12
option remote-subvolume /mnt/gluster/brick0/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-5
type protocol/client
option remote-host de-dc1-c1-pserver13
option remote-subvolume /mnt/gluster/brick0/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-6
type protocol/client
option remote-host de-dc1-c1-pserver12
option remote-subvolume /mnt/gluster/brick1/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-client-7
type protocol/client
option remote-host de-dc1-c1-pserver13
option remote-subvolume /mnt/gluster/brick1/storage
option transport-type rdma
option ping-timeout 5
end-volume
volume storage0-replicate-0
type cluster/replicate
subvolumes storage0-client-0 storage0-client-1
end-volume
volume storage0-replicate-1
type cluster/replicate
subvolumes storage0-client-2 storage0-client-3
end-volume
volume storage0-replicate-2
type cluster/replicate
subvolumes storage0-client-4 storage0-client-5
end-volume
volume storage0-replicate-3
type cluster/replicate
subvolumes storage0-client-6 storage0-client-7
end-volume
volume storage0-dht
type cluster/distribute
subvolumes storage0-replicate-0 storage0-replicate-1
storage0-replicate-2 storage0-replicate-3
end-volume
volume storage0-write-behind
type performance/write-behind
subvolumes storage0-dht
end-volume
volume storage0-read-ahead
type performance/read-ahead
subvolumes storage0-write-behind
end-volume
volume storage0-io-cache
type performance/io-cache
option cache-size 4096MB
subvolumes storage0-read-ahead
end-volume
volume storage0-quick-read
type performance/quick-read
option cache-size 4096MB
subvolumes storage0-io-cache
end-volume
volume storage0-stat-prefetch
type performance/stat-prefetch
subvolumes storage0-quick-read
end-volume
volume storage0
type debug/io-stats
subvolumes storage0-stat-prefetch
end-volume
> -Original Message-
> From: Pranith Kumar. Karampuri [mailto:prani...@gluster.com]
> Sent: Tuesday, May 17, 2011 7:16 AM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Client and server file "view",
> different results?! Client can't see the right file.
>
>
> Martin,
> Is this a distributed-replicate setup?. Could you
> attach the vol-file of the client.
>
> Pranith
> - Original Message -
> From: "Martin Schenker"
> To: gluster-users@gluster.org
> Sent: Monday, May 16, 2011 2:49:29 PM
> Subject: [Gluster-users] Client and server file "view",
> different results?! Client can't see the right file.
>
>
> Client and server file "view", different results?! Client
> can't see the right file.
>
> Hi all!
>
> Here we have another mismatch between the client "view" and
> the server mounts:
>
> From the server site everything seems well, the 20G file is
> visible and the attributes seem to match:
>
> 0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8
> f-8542864da6ef/hdd-images/
>
> # file:
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f
> -8542864da6ef/hdd-images//20964
> trusted.afr.storage0-client-2=0x
> trusted.afr.storage0-client-3=0x
>
> 0 root@pserver5:~ # find /mnt/gluster/ -name 20964 | xargs -i
> ls -al {}
> -rwxrwx--- 1 libvirt-qemu vcb 21474836480 May 13 11:21
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8
> f-8542864da6ef/hdd-images/20964
&g
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Martin,
Is this a distributed-replicate setup?. Could you attach the vol-file of
the client.
Pranith
- Original Message -
From: "Martin Schenker"
To: gluster-users@gluster.org
Sent: Monday, May 16, 2011 2:49:29 PM
Subject: [Gluster-users] Client and server file "view", different results?!
Client can't see the right file.
Client and server file "view", different results?! Client can't see the right
file.
Hi all!
Here we have another mismatch between the client "view" and the server mounts:
From the server site everything seems well, the 20G file is visible and the
attributes seem to match:
0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images//20964
trusted.afr.storage0-client-2=0x
trusted.afr.storage0-client-3=0x
0 root@pserver5:~ # find /mnt/gluster/ -name 20964 | xargs -i ls -al {}
-rwxrwx--- 1 libvirt-qemu vcb 21474836480 May 13 11:21
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20964
But the client view shows 2!! files with 0 byte size!! And these aren't any
link files created by Gluster… ( with the T on the end)
0 root@pserver5:~ # find /opt/profitbricks/storage/ -name 20964 | xargs -i ls
-al {}
-rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20964
-rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20964
I'm a bit stumped that we seem to have so many weird errors cropping up. Any
ideas? I've checked the ext4 filesystem on all boxes, no real problems. We run
a distributed cluster with 4 servers offering 2 bricks each.
Best, Martin
> -Original Message-
> From: Mohit Anchlia [ mailto:mohitanch...@gmail.com ]
> Sent: Monday, May 16, 2011 2:24 AM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Brick pair file mismatch,
> self-heal problems?
>
>
> Try this to trigger self heal:
>
> find -noleaf -print0 -name | xargs
> --null stat >/dev/null
>
>
>
> On Sun, May 15, 2011 at 11:20 AM, Martin Schenker
> wrote:
> > Can someone enlighten me what's going on here? We have a two peers,
> > the file 21313 is shown through the client mountpoint as
> "1Jan1970",
> > attribs on server pserver3 don't match but NO self-heal or
> repair can
> > be triggered through "ls -alR"?!?
> >
> > Checking the files through the server mounts show that two versions
> > are on the system. But the wrong one (as with the
> "1Jan1970") seems to
> > be the preferred one by the client?!?
> >
> > Do I need to use setattr or what in order to get the client
> to see the
> > RIGHT version?!? This is not the ONLY file displaying this
> problematic
> > behaviour!
> >
> > Thanks for any feedback.
> >
> > Martin
> >
> > pserver5:
> >
> > 0 root@pserver5:~ # ls -al
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > 4da6ef
> > /hdd-images
> >
> > -rwxrwx--- 1 libvirt-qemu vcb 483183820800 May 13 13:41 21313
> >
> > 0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > 4da6ef
> > /hdd-images/21313
> > getfattr: Removing leading '/' from absolute path names
> > # file:
> >
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f
> -8542864da6ef/
> > hdd-images/21313
> > trusted.afr.storage0-client-2=0x
> > trusted.afr.storage0-client-3=0x
> >
> > 0 root@pserver5:~ # ls -alR
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
> > a6ef/h
> > dd-images/21313
> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> 8542864da6ef/h
> > dd-images/21313
> >
> > pserver3:
> >
> > 0 root@pserver3:~ # ls -al
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > 4da6ef
> > /hdd-images
> >
> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970 21313
> >
> > 0 root@pserver3:~ # ls -alR
&g
Re: [Gluster-users] Client and server file "view", different results?! Client can't see the right file.
What happens when you read the file? Do you see right contents? These
look like linked files created in order to locate the files in the
right server. Did you recently upgrade or add/remove bricks?
Can you also look at the gfid on these files from server side?
Run
getfattr -dm -
On Mon, May 16, 2011 at 2:19 AM, Martin Schenker
wrote:
> Hi all!
>
> Here we have another mismatch between the client "view" and the server
> mounts:
>
> From the server site everything seems well, the 20G file is visible and the
> attributes seem to match:
>
> 0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/
>
> # file:
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images//20964
> trusted.afr.storage0-client-2=0x
> trusted.afr.storage0-client-3=0x
>
> 0 root@pserver5:~ # find /mnt/gluster/ -name 20964 | xargs -i ls -al
> {}
> -rwxrwx--- 1 libvirt-qemu vcb 21474836480 May 13 11:21
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20964
>
> But the client view shows 2!! files with 0 byte size!! And these aren't any
> link files created by Gluster… ( with the T on the end)
>
> 0 root@pserver5:~ # find /opt/profitbricks/storage/ -name 20964 |
> xargs -i ls -al {}
> -rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20964
>
> -rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/hdd-images/20964
>
> I'm a bit stumped that we seem to have so many weird errors cropping up. Any
> ideas? I've checked the ext4 filesystem on all boxes, no real problems. We
> run a distributed cluster with 4 servers offering 2 bricks each.
>
> Best, Martin
>
>
>> -Original Message-
>> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
>> Sent: Monday, May 16, 2011 2:24 AM
>> To: Martin Schenker
>> Cc: gluster-users@gluster.org
>> Subject: Re: [Gluster-users] Brick pair file mismatch,
>> self-heal problems?
>>
>>
>> Try this to trigger self heal:
>>
>> find -noleaf -print0 -name | xargs
>> --null stat >/dev/null
>>
>>
>>
>> On Sun, May 15, 2011 at 11:20 AM, Martin Schenker
>> wrote:
>> > Can someone enlighten me what's going on here? We have a two peers,
>> > the file 21313 is shown through the client mountpoint as
>> "1Jan1970",
>> > attribs on server pserver3 don't match but NO self-heal or
>> repair can
>> > be triggered through "ls -alR"?!?
>> >
>> > Checking the files through the server mounts show that two versions
>> > are on the system. But the wrong one (as with the
>> "1Jan1970") seems to
>> > be the preferred one by the client?!?
>> >
>> > Do I need to use setattr or what in order to get the client
>> to see the
>> > RIGHT version?!? This is not the ONLY file displaying this
>> problematic
>> > behaviour!
>> >
>> > Thanks for any feedback.
>> >
>> > Martin
>> >
>> > pserver5:
>> >
>> > 0 root@pserver5:~ # ls -al
>> >
>> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
>> > 4da6ef
>> > /hdd-images
>> >
>> > -rwxrwx--- 1 libvirt-qemu vcb 483183820800 May 13 13:41 21313
>> >
>> > 0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
>> >
>> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
>> > 4da6ef
>> > /hdd-images/21313
>> > getfattr: Removing leading '/' from absolute path names
>> > # file:
>> >
>> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f
>> -8542864da6ef/
>> > hdd-images/21313
>> > trusted.afr.storage0-client-2=0x
>> > trusted.afr.storage0-client-3=0x
>> >
>> > 0 root@pserver5:~ # ls -alR
>> >
>> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
>> > a6ef/h
>> > dd-images/21313
>> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
>> >
>> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
>> 8542864da6ef/h
>> > dd-images/21313
>> >
>> > pserver3:
>> >
>> > 0 root@pserver3:~ # ls -al
>> >
>> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
>> > 4da6ef
>> > /hdd-images
>> >
>> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970 21313
>> >
>> > 0 root@pserver3:~ # ls -alR
>> >
>> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
>> > a6ef/h
>> > dd-images/21313
>> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
>> >
>> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
>> 8542864da6ef/h
>> > dd-images/21313
>> >
>> > 0 root@pserver3:~ # getfattr -R -d -e hex -m "trusted.afr."
>> > /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-
>> > ad8f-8542864da6ef/hdd-images/21313
>> > getfattr: Removing leading '/' from absolute path names
>> > # file:
>> >
>> mnt/
[Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Hi all!
Here we have another mismatch between the client "view" and the server
mounts:
>From the server site everything seems well, the 20G file is visible and the
attributes seem to match:
0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/
# file:
mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/
hdd-images//20964
trusted.afr.storage0-client-2=0x
trusted.afr.storage0-client-3=0x
0 root@pserver5:~ # find /mnt/gluster/ -name 20964 | xargs -i ls -al
{}
-rwxrwx--- 1 libvirt-qemu vcb 21474836480 May 13 11:21
/mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef
/hdd-images/20964
But the client view shows 2!! files with 0 byte size!! And these aren't any
link files created by Gluster
( with the T on the end)
0 root@pserver5:~ # find /opt/profitbricks/storage/ -name 20964 |
xargs -i ls -al {}
-rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20964
-rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
/opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864da6ef/h
dd-images/20964
I'm a bit stumped that we seem to have so many weird errors cropping up. Any
ideas? I've checked the ext4 filesystem on all boxes, no real problems. We
run a distributed cluster with 4 servers offering 2 bricks each.
Best, Martin
> -Original Message-
> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
> Sent: Monday, May 16, 2011 2:24 AM
> To: Martin Schenker
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Brick pair file mismatch,
> self-heal problems?
>
>
> Try this to trigger self heal:
>
> find -noleaf -print0 -name | xargs
> --null stat >/dev/null
>
>
>
> On Sun, May 15, 2011 at 11:20 AM, Martin Schenker
> wrote:
> > Can someone enlighten me what's going on here? We have a two peers,
> > the file 21313 is shown through the client mountpoint as
> "1Jan1970",
> > attribs on server pserver3 don't match but NO self-heal or
> repair can
> > be triggered through "ls -alR"?!?
> >
> > Checking the files through the server mounts show that two versions
> > are on the system. But the wrong one (as with the
> "1Jan1970") seems to
> > be the preferred one by the client?!?
> >
> > Do I need to use setattr or what in order to get the client
> to see the
> > RIGHT version?!? This is not the ONLY file displaying this
> problematic
> > behaviour!
> >
> > Thanks for any feedback.
> >
> > Martin
> >
> > pserver5:
> >
> > 0 root@pserver5:~ # ls -al
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > 4da6ef
> > /hdd-images
> >
> > -rwxrwx--- 1 libvirt-qemu vcb 483183820800 May 13 13:41 21313
> >
> > 0 root@pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > 4da6ef
> > /hdd-images/21313
> > getfattr: Removing leading '/' from absolute path names
> > # file:
> >
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f
> -8542864da6ef/
> > hdd-images/21313
> > trusted.afr.storage0-client-2=0x
> > trusted.afr.storage0-client-3=0x
> >
> > 0 root@pserver5:~ # ls -alR
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
> > a6ef/h
> > dd-images/21313
> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> 8542864da6ef/h
> > dd-images/21313
> >
> > pserver3:
> >
> > 0 root@pserver3:~ # ls -al
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > 4da6ef
> > /hdd-images
> >
> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970 21313
> >
> > 0 root@pserver3:~ # ls -alR
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
> > a6ef/h
> > dd-images/21313
> > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> 8542864da6ef/h
> > dd-images/21313
> >
> > 0 root@pserver3:~ # getfattr -R -d -e hex -m "trusted.afr."
> > /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-
> > ad8f-8542864da6ef/hdd-images/21313
> > getfattr: Removing leading '/' from absolute path names
> > # file:
> >
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864
> > da6ef/
> > hdd-images/21313
> > trusted.afr.storage0-client-2=0x
> > trusted.afr.storage0-client-3=0x0b090900
> <- mismatch,
> > should be targeted for self-heal/repair? Why is there a
> difference in the
> > views?
> >
> >
> > From the volfile:
> >
> > volume storage0-client-2
> >type protocol/client
> >option remote-host de-dc1-c1-pserver3
>
