Re: [Gluster-users] Resync or how to force the replication

[gandalf istari]

 Thank you so much,

It seems to be working the directories are now created but still empty. I
suppose this will take a while to sync 44GB

The only change I have done was: leave out nas-02-data for your command.

whats the best way now to monitor the sync process ?



You couldn't force start the volume because the brick
"nas-02-data:/datastore" doesn't have the "trusted.glusterfs.volume-id" .

>From nas-02 node execute :

1. setfattr -n trusted.glusterfs.volume-id -v
0xfdff519085ef4cba9056a6bbbd8d6863 nas-02-data:/datastore

2. gluster volume start datastore1 force.


-Shwetha
On 11/27/2013 12:23 PM, gandalf istari wrote:

 Hi Shwetha,

 [root@nas-01 ~]# getfattr -d -e hex -m . /datastore

getfattr: Removing leading '/' from absolute path names

# file: datastore

trusted.afr.datastore1-client-0=0x

trusted.afr.datastore1-client-1=0x

trusted.gfid=0x0001

trusted.glusterfs.dht=0x0001

trusted.glusterfs.volume-id=0xfdff519085ef4cba9056a6bbbd8d6863



 [root@nas-02 ~]# getfattr -d -e hex -m . /datastore

getfattr: Removing leading '/' from absolute path names

# file: datastore


security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000

  I don't understand what happened .


 gr

Patrick

 Hi Gandalf,

can you run the following command on the brick path?

"getfattr -d -e hex -m . /datastore" on both "nas-01-data" and
"nas-02-data" nodes.

This will let us know whether there is "trusted.glusterfs.volume-id" set.

-Shwetha

On 11/26/2013 07:36 PM, gandalf istari wrote:

 hi thanks for the quick answer.

 I'm running glusterfs 3.4.1

 [root@nas-02 datastore]# gluster volume start datastore1 force

volume start: datastore1: failed: Failed to get extended attribute
trusted.glusterfs.volume-id for brick dir /datastore. Reason : No data
available

It seems that the .gluster directory is missing for some reason.


 volume replace-brick datastore1 nas-01-data:/datastore nas-02-data:/datastore
commit force

To rebuild/replace the missing brick ?

I'm quite new with glusterfs

Thanks






 On 26/11/13 12:47, gandalf istari wrote:

Hi have setup a two node replication glusterfs. After the initial
installation the "master" node was put into the datacenter and after two
week we moved the second one also to the datacenter.

 But the sync has not started yet.

 On the "master"

gluster> volume info all

Volume Name: datastore1

Type: Replicate

Volume ID: fdff5190-85ef-4cba-9056-a6bbbd8d6863

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nas-01-data:/datastore

Brick2: nas-02-data:/datastore

gluster> peer status

Number of Peers: 1


 Hostname: nas-02-data

Uuid: 71df9f86-a87b-481d-896c-c0d4ab679cfa

 State: Peer in Cluster (Connected)


 On the "slave"

gluster> peer status

Number of Peers: 1

Hostname: 192.168.70.6

Uuid: 97ef0154-ad7b-402a-b0cb-22be09134a3c

 State: Peer in Cluster (Connected)


 gluster> volume status all

Status of volume: datastore1

Gluster process   Port Online Pid

--

Brick nas-01-data:/datastore  49152 Y 2130

Brick nas-02-data:/datastore  N/A N N/A

NFS Server on localhost   2049 Y 8064

Self-heal Daemon on localhost  N/A Y 8073

NFS Server on 192.168.70.6  2049 Y 3379

Self-heal Daemon on 192.168.70.6  N/A Y 3384

Which version of glusterfs are you running?

volume status suggests that the second brick (nas-02-data:/datastore) is
not running.

Can you run "gluster volume start  force" in any of these two
nodes and try again?
Then you would also required to run `find . | xargs stat` on the mountpoint
of the volume. That should trigger the self heal.



 There are no active volume tasks


 I would like to run on the "slave" gluster volume sync nas-01-data
datastore1

BTW, There is no concept of "master" and "slave" in afr (replication).
However there is concept of "master volume" and "slave volume" in gluster
geo-replication.

 But then the virtual machines hosted will be unavailible is there another
way to start the replication ?


 Thanks






___
Gluster-users mailing
listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing
listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Unable to get any gluster information.

[Bobby Jacob]

Hi,

I'm running GlusterFS 3.3.2 on Centos 6.4. There seems to be some issue with 
one of my gluster servers. GFS001 and GFS002 are the 2 servers. Please see 
below results.

GFS001 > gluster volume info
ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Read-only file 
system)
ERROR: failed to open logfile /var/log/glusterfs/cli.log

GFS001 > gluster volume  sftpvol start force
ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Read-only file 
system)
ERROR: failed to open logfile /var/log/glusterfs/cli.log

GFS002 > gluster volume status
gluster volume status
Status of volume: sftpvol
Gluster process  Port   
  OnlinePid
--
Brick GFS001:/mnt/sftpbrick24009N   
  N/A
Brick GFS002/mnt/sftpbrick 24009Y   
  1352
NFS Server on localhost 38467   
 Y 1358
Self-heal Daemon on localhost N/A  Y
 1364
NFS Server on 172.16.95.16538467N   
  N/A
Self-heal Daemon on 172.16.95.165   N/A  N  
   N/A

What is the cause of this issue.? How can I resolve it.

Thanks & Regards,
Bobby Jacob

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Resync or how to force the replication

[shwetha]

You couldn't force start the volume because the brick 
"nas-02-data:/datastore" doesn't have the "trusted.glusterfs.volume-id" .


From nas-02 node execute :

1. setfattr -n trusted.glusterfs.volume-id -v 
0xfdff519085ef4cba9056a6bbbd8d6863 nas-02-data:/datastore


2. gluster volume start datastore1 force.


-Shwetha
On 11/27/2013 12:23 PM, gandalf istari wrote:

Hi Shwetha,

[root@nas-01 ~]# getfattr -d -e hex -m . /datastore

getfattr: Removing leading '/' from absolute path names

# file: datastore

trusted.afr.datastore1-client-0=0x

trusted.afr.datastore1-client-1=0x

trusted.gfid=0x0001

trusted.glusterfs.dht=0x0001

trusted.glusterfs.volume-id=0xfdff519085ef4cba9056a6bbbd8d6863



[root@nas-02 ~]# getfattr -d -e hex -m . /datastore

getfattr: Removing leading '/' from absolute path names

# file: datastore

security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000

I don't understand what happened .


gr

Patrick


Hi Gandalf,

can you run the following command on the brick path?

"getfattr -d -e hex -m . /datastore" on both "nas-01-data" and 
"nas-02-data" nodes.


This will let us know whether there is "trusted.glusterfs.volume-id" set.

-Shwetha

On 11/26/2013 07:36 PM, gandalf istari wrote:

hi thanks for the quick answer.

I'm running glusterfs 3.4.1

[root@nas-02 datastore]# gluster volume start datastore1 force

volume start: datastore1: failed: Failed to get extended attribute 
trusted.glusterfs.volume-id for brick dir /datastore. Reason : No 
data available


It seems that the .gluster directory is missing for some reason.


volume replace-brick datastore1 nas-01-data:/datastorenas-02-data:/datastore
commit force
To rebuild/replace the missing brick ?
I'm quite new with glusterfs
Thanks






On 26/11/13 12:47, gandalf istari wrote:
Hi have setup a two node replication glusterfs. After the initial 
installation the "master" node was put into the datacenter and after 
two week we moved the second one also to the datacenter.


But the sync has not started yet.

On the "master"

gluster> volume info all

Volume Name: datastore1

Type: Replicate

Volume ID: fdff5190-85ef-4cba-9056-a6bbbd8d6863

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nas-01-data:/datastore

Brick2: nas-02-data:/datastore

gluster> peer status

Number of Peers: 1


Hostname: nas-02-data

Uuid: 71df9f86-a87b-481d-896c-c0d4ab679cfa

State: Peer in Cluster (Connected)


On the "slave"

gluster> peer status

Number of Peers: 1

Hostname: 192.168.70.6

Uuid: 97ef0154-ad7b-402a-b0cb-22be09134a3c

State: Peer in Cluster (Connected)


gluster> volume status all

Status of volume: datastore1

Gluster processPortOnlinePid

--

Brick nas-01-data:/datastore49152Y2130

Brick nas-02-data:/datastoreN/ANN/A

NFS Server on localhost2049Y8064

Self-heal Daemon on localhostN/AY8073

NFS Server on 192.168.70.62049Y3379

Self-heal Daemon on 192.168.70.6N/AY3384


Which version of glusterfs are you running?

volume status suggests that the second brick (nas-02-data:/datastore) 
is not running.


Can you run "gluster volume start  force" in any of these 
two nodes and try again?
Then you would also required to run `find . | xargs stat` on the 
mountpoint of the volume. That should trigger the self heal.


There are no active volume tasks


I would like to run on the "slave" gluster volume sync nas-01-data 
datastore1


BTW, There is no concept of "master" and "slave" in afr 
(replication). However there is concept of "master volume" and "slave 
volume" in gluster geo-replication.


But then the virtual machines hosted will be unavailible is there 
another way to start the replication ?



Thanks






___
Gluster-users mailing list
Gluster-users@gluster.org  
http://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org  
http://supercolony.gluster.org/mailman/listinfo/gluster-users







___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Community Weekly Meeting

[James]

Sorry for being a bit confused. I'd like to participate, but I'm a bit
fuzzy about when exactly. Can someone confirm the time and timezone?

Cheers
James

On Fri, Nov 22, 2013 at 9:58 AM, Vijay Bellur  wrote:
> On 11/22/2013 07:44 PM, John Mark Walker wrote:
>>
>> Thank you for setting this up, Vijay!
>>
>> In the interests of getting west coast USA participation, I'm wondering
>> if this is too early (6am Pacific Time). Or we can trade off - alternate
>> between this time and 12 hours later on successive weeks.
>
>
> 6 AM Pacific Time is considered one of the better time slots for a global
> meeting. When folks in US spring forward, it would be 7 AM which might work
> better. Also, UTC + 2 translates to 11 PM in the far east. Irrespective of
> the slot we select, we will miss some part of the world :-/.
>
> We can even set up an online poll to determine what time works best for
> majority of us. I will send out recurring invites after we finalize the
> schedule.
>
>
>>
>> I'm looking to hear from others who would be interested in participating.
>
>
> +1
>
> Thanks,
> Vijay
>
>>
>>
>> 
>>
>>
>> The following is a new meeting request:
>>
>> Subject: Gluster Community Weekly Meeting
>> Organizer: "Vijay Bellur" 
>>
>> Location: #gluster-meeting on irc.freenode.net
>> Time: Wednesday, November 27, 2013, 7:30:00 PM - 8:30:00 PM GMT
>> +05:30 Chennai, Kolkata, Mumbai, New Delhi
>>
>> Invitees: gluster-de...@nongnu.org; gluster-users@gluster.org
>>
>>
>> *~*~*~*~*~*~*~*~*~*
>>
>> Greetings,
>>
>> We have had discussions around weekly IRC meetings for our community
>> in the past but we have not got to do that so far. Here is an
>> attempt to start the first of it at UTC + 2 on Wednesday next. Let
>> us discuss all aspects related to the Gluster community then!
>>
>> Etherpad for the meeting -
>> http://titanpad.com/gluster-community-meetings. Please feel free to
>> add your agenda items there.
>>
>> Cheers,
>> Vijay
>>
>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Community Weekly Meeting

[Vijay Bellur]

On 11/27/2013 12:25 PM, James wrote:

Sorry for being a bit confused. I'd like to participate, but I'm a bit
fuzzy about when exactly. Can someone confirm the time and timezone?


UTC + 2 today. In about 7 hours from now.

Cheers,
Vijay



Cheers
James

On Fri, Nov 22, 2013 at 9:58 AM, Vijay Bellur  wrote:

On 11/22/2013 07:44 PM, John Mark Walker wrote:


Thank you for setting this up, Vijay!

In the interests of getting west coast USA participation, I'm wondering
if this is too early (6am Pacific Time). Or we can trade off - alternate
between this time and 12 hours later on successive weeks.



6 AM Pacific Time is considered one of the better time slots for a global
meeting. When folks in US spring forward, it would be 7 AM which might work
better. Also, UTC + 2 translates to 11 PM in the far east. Irrespective of
the slot we select, we will miss some part of the world :-/.

We can even set up an online poll to determine what time works best for
majority of us. I will send out recurring invites after we finalize the
schedule.




I'm looking to hear from others who would be interested in participating.



+1

Thanks,
Vijay







 The following is a new meeting request:

 Subject: Gluster Community Weekly Meeting
 Organizer: "Vijay Bellur" 

 Location: #gluster-meeting on irc.freenode.net
 Time: Wednesday, November 27, 2013, 7:30:00 PM - 8:30:00 PM GMT
 +05:30 Chennai, Kolkata, Mumbai, New Delhi

 Invitees: gluster-de...@nongnu.org; gluster-users@gluster.org


 *~*~*~*~*~*~*~*~*~*

 Greetings,

 We have had discussions around weekly IRC meetings for our community
 in the past but we have not got to do that so far. Here is an
 attempt to start the first of it at UTC + 2 on Wednesday next. Let
 us discuss all aspects related to the Gluster community then!

 Etherpad for the meeting -
 http://titanpad.com/gluster-community-meetings. Please feel free to
 add your agenda items there.

 Cheers,
 Vijay




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Resync or how to force the replication

[gandalf istari]

 Hi Shwetha,

[root@nas-01 ~]# getfattr -d -e hex -m . /datastore

getfattr: Removing leading '/' from absolute path names

# file: datastore

trusted.afr.datastore1-client-0=0x

trusted.afr.datastore1-client-1=0x

trusted.gfid=0x0001

trusted.glusterfs.dht=0x0001

trusted.glusterfs.volume-id=0xfdff519085ef4cba9056a6bbbd8d6863



[root@nas-02 ~]# getfattr -d -e hex -m . /datastore

getfattr: Removing leading '/' from absolute path names

# file: datastore

security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000


I don't understand what happened .


gr

Patrick

Hi Gandalf,

can you run the following command on the brick path?

"getfattr -d -e hex -m . /datastore" on both "nas-01-data" and
"nas-02-data" nodes.

This will let us know whether there is "trusted.glusterfs.volume-id" set.

-Shwetha

On 11/26/2013 07:36 PM, gandalf istari wrote:

 hi thanks for the quick answer.

 I'm running glusterfs 3.4.1

 [root@nas-02 datastore]# gluster volume start datastore1 force

volume start: datastore1: failed: Failed to get extended attribute
trusted.glusterfs.volume-id for brick dir /datastore. Reason : No data
available

It seems that the .gluster directory is missing for some reason.


 volume replace-brick datastore1 nas-01-data:/datastore nas-02-data:/datastore
commit force

 To rebuild/replace the missing brick ?

I'm quite new with glusterfs

 Thanks






 On 26/11/13 12:47, gandalf istari wrote:

Hi have setup a two node replication glusterfs. After the initial
installation the "master" node was put into the datacenter and after two
week we moved the second one also to the datacenter.

 But the sync has not started yet.

 On the "master"

gluster> volume info all

Volume Name: datastore1

Type: Replicate

Volume ID: fdff5190-85ef-4cba-9056-a6bbbd8d6863

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nas-01-data:/datastore

Brick2: nas-02-data:/datastore

gluster> peer status

Number of Peers: 1


 Hostname: nas-02-data

Uuid: 71df9f86-a87b-481d-896c-c0d4ab679cfa

 State: Peer in Cluster (Connected)


 On the "slave"

gluster> peer status

Number of Peers: 1

Hostname: 192.168.70.6

Uuid: 97ef0154-ad7b-402a-b0cb-22be09134a3c

 State: Peer in Cluster (Connected)


 gluster> volume status all

Status of volume: datastore1

Gluster process   Port Online Pid

--

Brick nas-01-data:/datastore  49152 Y 2130

Brick nas-02-data:/datastore  N/A N N/A

NFS Server on localhost   2049 Y 8064

Self-heal Daemon on localhost  N/A Y 8073

NFS Server on 192.168.70.6  2049 Y 3379

Self-heal Daemon on 192.168.70.6  N/A Y 3384

Which version of glusterfs are you running?

volume status suggests that the second brick (nas-02-data:/datastore) is
not running.

Can you run "gluster volume start  force" in any of these two
nodes and try again?
Then you would also required to run `find . | xargs stat` on the mountpoint
of the volume. That should trigger the self heal.



 There are no active volume tasks


 I would like to run on the "slave" gluster volume sync nas-01-data
datastore1

BTW, There is no concept of "master" and "slave" in afr (replication).
However there is concept of "master volume" and "slave volume" in gluster
geo-replication.

 But then the virtual machines hosted will be unavailible is there another
way to start the replication ?


 Thanks






___
Gluster-users mailing
listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing
listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Resync or how to force the replication

[shwetha]

Hi Gandalf,

can you run the following command on the brick path?

"getfattr -d -e hex -m . /datastore" on both "nas-01-data" and 
"nas-02-data" nodes.


This will let us know whether there is "trusted.glusterfs.volume-id" set.

-Shwetha

On 11/26/2013 07:36 PM, gandalf istari wrote:

hi thanks for the quick answer.

I'm running glusterfs 3.4.1

[root@nas-02 datastore]# gluster volume start datastore1 force

volume start: datastore1: failed: Failed to get extended attribute 
trusted.glusterfs.volume-id for brick dir /datastore. Reason : No data 
available


It seems that the .gluster directory is missing for some reason.


volume replace-brick datastore1 nas-01-data:/datastorenas-02-data:/datastore
commit force
To rebuild/replace the missing brick ?
I'm quite new with glusterfs
Thanks






On 26/11/13 12:47, gandalf istari wrote:
Hi have setup a two node replication glusterfs. After the initial 
installation the "master" node was put into the datacenter and after 
two week we moved the second one also to the datacenter.


But the sync has not started yet.

On the "master"

gluster> volume info all

Volume Name: datastore1

Type: Replicate

Volume ID: fdff5190-85ef-4cba-9056-a6bbbd8d6863

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nas-01-data:/datastore

Brick2: nas-02-data:/datastore

gluster> peer status

Number of Peers: 1


Hostname: nas-02-data

Uuid: 71df9f86-a87b-481d-896c-c0d4ab679cfa

State: Peer in Cluster (Connected)


On the "slave"

gluster> peer status

Number of Peers: 1

Hostname: 192.168.70.6

Uuid: 97ef0154-ad7b-402a-b0cb-22be09134a3c

State: Peer in Cluster (Connected)


gluster> volume status all

Status of volume: datastore1

Gluster processPortOnlinePid

--

Brick nas-01-data:/datastore49152Y2130

Brick nas-02-data:/datastoreN/ANN/A

NFS Server on localhost2049Y8064

Self-heal Daemon on localhostN/AY8073

NFS Server on 192.168.70.62049Y3379

Self-heal Daemon on 192.168.70.6N/AY3384


Which version of glusterfs are you running?

volume status suggests that the second brick (nas-02-data:/datastore) 
is not running.


Can you run "gluster volume start  force" in any of these two 
nodes and try again?
Then you would also required to run `find . | xargs stat` on the 
mountpoint of the volume. That should trigger the self heal.


There are no active volume tasks


I would like to run on the "slave" gluster volume sync nas-01-data 
datastore1


BTW, There is no concept of "master" and "slave" in afr (replication). 
However there is concept of "master volume" and "slave volume" in 
gluster geo-replication.


But then the virtual machines hosted will be unavailible is there 
another way to start the replication ?



Thanks






___
Gluster-users mailing list
Gluster-users@gluster.org  
http://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] A question about comparing gluster to other storage

[Paul Robert Marino]

Well swift isn't a file system persay its just an object store which uses one of many possible underlying flies systems and offers a mechanism to do asynchronous rsync based replication if needed.I use swift on top of gluster very effectively it provides its own object store API's plus AWS compatible API and allows more traditional access to legacy apps via its native fuse client,  and NFS in addition it can be utilized with Samba and CTDB to provide clustered CIFS access if needed.I've also heard of people who use CEHP with Swift.So there are many options here.The one catch is if you are using a distribution based on RHEL including but not limited too CentOS or scientific Linux Gluster is definitely a wider deployed and easier install than CEHP. In fact there was a discussion about a few months back on the Redhat OpenStack list and it was pretty much decided much to the dismay of a CEHP fan who started the discussion that at least for now RDO and RHOS would only support Gluster and more traditional backend like local disks, SANS, etc. As a backend for Swift.-- Sent from my HP Pre3On Nov 26, 2013 17:26, Randy Breunling  wrote: I'm relatively new to the gluster community and wanted to know if there's anyone out there that can talk to me, or point me to comparative information on the following:
- glusterFS- SWIFT- CEPHI am interested in a solution that is object-based and will scale into single-digit petabytes.I'd like to know of experiences with these solutions that have to do with large-scale deployments.
If this is not the/a correct forum to discuss this...please let me know.Thanks...--Randy Breunlingrbreunl...@gmail.com 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Every requests only push to one server?

[Ta Ba Tuan]

Hi everyone,

This is my configuration:
#gluster volume create export replica 3 transport tcp 
gluster-data-01:/export/sda \

   gluster-data-02:/export/sda \
gluster-data-03:/export/sda \

   gluster-data-01:/export/sdb \
   gluster-data-02:/export/sdb \
   gluster-data-03:/export/sdb \

   gluster-data-01:/export/sdc \
   gluster-data-02:/export/sdc \
   gluster-data-03:/export/sdc

With this configurion, every requests will push gluster-data-01?
I worry that CPU and Memory on gluster-data-01 will overload.

Thanks!
--
TuanTaBa


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failed rebalance - lost files, inaccessible files, permission issues

[Shawn Heisey]

Here's what our own developer had to say about this:

On 11/8/2013 8:23 PM, Shawn Heisey wrote:

When I looked at the individual cases of lost or corrupted files, one
thing kept staring at me in the face until I recognized it:

[2013-11-02 03:56:36.472170] I [dht-rebalance.c:647:dht_migrate_file]
0-mdfs-dht: /REDACTED/mdfs/AKG/akgphotos/docs/000/002: attempting to
move from mdfs-replicate-2 to mdfs-replicate-12
[2013-11-02 03:56:36.472186] I [dht-rebalance.c:647:dht_migrate_file]
0-mdfs-dht: /REDACTED/mdfs/AKG/akgphotos/docs/000/002: attempting to
move from mdfs-replicate-2 to mdfs-replicate-12
[2013-11-02 03:56:36.480567] I [dht-rebalance.c:647:dht_migrate_file]
0-mdfs-dht: /REDACTED/mdfs/AKG/akgphotos/docs/000/002: attempting to
move from mdfs-replicate-2 to mdfs-replicate-12

Three simultaneous processes on the same file!  Of course that would
have undefined results, and be the cause of all our problems.  NFS may
not be related after all.

Tediously scrolling through the error log I found mostly errors where it
refused to copy files from a more empty brick to a fuller brick, which
makes perfect sense.  The wheels started falling off about 26 hours into
the rebalance.

[2013-10-29 23:13:17.193108] C
[client-handshake.c:126:rpc_client_ping_timer_expired] 0-mdfs-client-1:
server 10.116.0.22:24025 has not responded in the last 42 seconds,
disconnecting.
[2013-10-29 23:13:17.200616] E [rpc-clnt.c:373:saved_frames_unwind]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x36de60f808]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0)
[0x36de60f4c0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)
[0x36de60ef2e]))) 0-mdfs-client-1: forced unwinding frame type(GlusterFS
3.1) op(STAT(1)) called at 2013-10-29 23:12:20.641732 (xid=0x1763733x)
[2013-10-29 23:13:17.200659] W [client3_1-fops.c:474:client3_1_stat_cbk]
0-mdfs-client-1: remote operation failed: Transport endpoint is not
connected
[2013-10-29 23:13:17.200756] E [rpc-clnt.c:373:saved_frames_unwind]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x36de60f808]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0)
[0x36de60f4c0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)
[0x36de60ef2e]))) 0-mdfs-client-1: forced unwinding frame type(GlusterFS
Handshake) op(PING(3)) called at 2013-10-29 23:12:35.184481
(xid=0x1763734x)
[2013-10-29 23:13:17.200792] W [client-handshake.c:275:client_ping_cbk]
0-mdfs-client-1: timer must have expired
[2013-10-29 23:13:17.217477] I [client.c:2090:client_rpc_notify]
0-mdfs-client-1: disconnected
.
.
.
[2013-10-29 23:13:28.198386] I
[client-handshake.c:1636:select_server_supported_programs]
0-mdfs-client-1: Using Program GlusterFS 3.3.1, Num (1298437), Version
(330)
[2013-10-29 23:13:28.198803] I
[client-handshake.c:1433:client_setvolume_cbk] 0-mdfs-client-1:
Connected to 10.116.0.22:24025, attached to remote volume
'/bricks/d00v00/mdfs'.
[2013-10-29 23:13:28.198826] I
[client-handshake.c:1445:client_setvolume_cbk] 0-mdfs-client-1: Server
and Client lk-version numbers are not same, reopening the fds
[2013-10-29 23:13:28.205629] I
[client-handshake.c:1282:client_post_handshake] 0-mdfs-client-1: 42745
fds open - Delaying child_up until they are re-opened
[2013-10-29 23:13:28.205782] E [inode.c:1090:__inode_path]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client_post_handshake+0x110)
[0x7fb74bdf1c90]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(protocol_client_reopendir+0x107)
[0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a)
[0x36dde2ca1a]))) 0-: Assertion failed: 0
[2013-10-29 23:13:28.205852] W [inode.c:1091:__inode_path]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client_post_handshake+0x110)
[0x7fb74bdf1c90]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(protocol_client_reopendir+0x107)
[0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a)
[0x36dde2ca1a]))) 0-mdfs-client-1: invalid inode
[2013-10-29 23:13:28.205881] W
[client-handshake.c:1108:protocol_client_reopendir] 0-mdfs-client-1:
couldn't build path from inode ----
[2013-10-29 23:13:28.205944] E [inode.c:1090:__inode_path]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client_post_handshake+0x110)
[0x7fb74bdf1c90]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(protocol_client_reopendir+0x107)
[0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a)
[0x36dde2ca1a]))) 0-: Assertion failed: 0
[2013-10-29 23:13:28.205992] W [inode.c:1091:__inode_path]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client_post_handshake+0x110)
[0x7fb74bdf1c90]
(-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(protocol_client_reopendir+0x107)
[0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a)
[0x36dde2ca1a]))) 0-mdfs-client-1: invalid inode
[2013-10-29 23:13:28.206009] W
[client-handshake.c:1108:protocol_client_reopendir] 0-mdfs-client-1:
couldn't build path from inode -0

Re: [Gluster-users] A question about comparing gluster to other storage

[Jay Vyas]

- GlusterFS will be better if you want to do directory operations, and want
a "real" filesystem (i.e. where you can grep stuff, edit files, have
consistency gaurantees etc... and swift will be better for pure scale (less
need to worry about metadata == easier to scale on commodity hardware.

- Certainly both glusterFS and swift would scale to petabytes (object
stores are built for that sort of thing, as is gluster).


On Tue, Nov 26, 2013 at 5:25 PM, Randy Breunling wrote:

> I'm relatively new to the gluster community and wanted to know if there's
> anyone out there that can talk to me, or point me to comparative
> information on the following:
> - glusterFS
> - SWIFT
> - CEPH
>
> I am interested in a solution that is object-based and will scale into
> single-digit petabytes.
> I'd like to know of experiences with these solutions that have to do with
> large-scale deployments.
>
> If this is not the/a correct forum to discuss this...please let me know.
>
> Thanks...
>
> --Randy Breunling
>
> rbreunl...@gmail.com
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>



-- 
Jay Vyas
http://jayunit100.blogspot.com
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] A question about comparing gluster to other storage

[Randy Breunling]

I'm relatively new to the gluster community and wanted to know if there's
anyone out there that can talk to me, or point me to comparative
information on the following:
- glusterFS
- SWIFT
- CEPH

I am interested in a solution that is object-based and will scale into
single-digit petabytes.
I'd like to know of experiences with these solutions that have to do with
large-scale deployments.

If this is not the/a correct forum to discuss this...please let me know.

Thanks...

--Randy Breunling

rbreunl...@gmail.com
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Can't get gluster to mount on boot

[Marcus Bointon]

I've just been pointed at a possible solution to a problem I've mentioned on 
here several times over the last couple of years - that I've been totally 
unable to make gluster auto-mount on boot.

The closest I've got has been by using autofs to mount an NFS volume, however, 
its won't work if the gluster server is local because it spots that you're 
mounting from localhost and does a local bind mount instead, completely 
bypassing gluster and thus not working. The trick is that autofs.master 
supports a 'nobind' option that tells it specifically not to do that. 
Unfortunately that option was only introduced in autofs 5.0.6 and I'm currently 
stuck on 5.0.4 (on Ubuntu 10.04 LTS) so as yet I've not been able to try this, 
but hope springs eternal...

I just thought I'd mention it in case anyone else runs into the same problem.

In other news I have a couple of servers for sale that I thought might interest 
gluster users: http://info.smartmessages.net/servers-for-sale/

Marcus
-- 
Marcus Bointon
Synchromedia Limited: Creators of http://www.smartmessages.net/
UK 1CRM solutions
mar...@synchromedia.co.uk | http://www.synchromedia.co.uk/

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Resync or how to force the replication

[gandalf istari]

 hi thanks for the quick answer.

I'm running glusterfs 3.4.1

[root@nas-02 datastore]# gluster volume start datastore1 force

volume start: datastore1: failed: Failed to get extended attribute
trusted.glusterfs.volume-id for brick dir /datastore. Reason : No data
available

It seems that the .gluster directory is missing for some reason.


volume replace-brick datastore1 nas-01-data:/datastore nas-02-data:/datastore
commit force


To rebuild/replace the missing brick ?

I'm quite new with glusterfs


Thanks






On 26/11/13 12:47, gandalf istari wrote:

Hi have setup a two node replication glusterfs. After the initial
installation the "master" node was put into the datacenter and after two
week we moved the second one also to the datacenter.

 But the sync has not started yet.

 On the "master"

gluster> volume info all

Volume Name: datastore1

Type: Replicate

Volume ID: fdff5190-85ef-4cba-9056-a6bbbd8d6863

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nas-01-data:/datastore

Brick2: nas-02-data:/datastore

gluster> peer status

Number of Peers: 1


 Hostname: nas-02-data

Uuid: 71df9f86-a87b-481d-896c-c0d4ab679cfa

 State: Peer in Cluster (Connected)


 On the "slave"

gluster> peer status

Number of Peers: 1

Hostname: 192.168.70.6

Uuid: 97ef0154-ad7b-402a-b0cb-22be09134a3c

 State: Peer in Cluster (Connected)


 gluster> volume status all

Status of volume: datastore1

Gluster process   Port Online Pid

--

Brick nas-01-data:/datastore  49152 Y 2130

Brick nas-02-data:/datastore  N/A N N/A

NFS Server on localhost   2049 Y 8064

Self-heal Daemon on localhost  N/A Y 8073

NFS Server on 192.168.70.6  2049 Y 3379

Self-heal Daemon on 192.168.70.6  N/A Y 3384

Which version of glusterfs are you running?

volume status suggests that the second brick (nas-02-data:/datastore) is
not running.

Can you run "gluster volume start  force" in any of these two
nodes and try again?
Then you would also required to run `find . | xargs stat` on the mountpoint
of the volume. That should trigger the self heal.



 There are no active volume tasks


 I would like to run on the "slave" gluster volume sync nas-01-data
datastore1

BTW, There is no concept of "master" and "slave" in afr (replication).
However there is concept of "master volume" and "slave volume" in gluster
geo-replication.

 But then the virtual machines hosted will be unavailible is there another
way to start the replication ?


 Thanks






___
Gluster-users mailing
listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Resync or how to force the replication

[M S Vishwanath Bhat]

On 26/11/13 12:47, gandalf istari wrote:
Hi have setup a two node replication glusterfs. After the initial 
installation the "master" node was put into the datacenter and after 
two week we moved the second one also to the datacenter.


But the sync has not started yet.

On the "master"

gluster> volume info all

Volume Name: datastore1

Type: Replicate

Volume ID: fdff5190-85ef-4cba-9056-a6bbbd8d6863

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nas-01-data:/datastore

Brick2: nas-02-data:/datastore

gluster> peer status

Number of Peers: 1


Hostname: nas-02-data

Uuid: 71df9f86-a87b-481d-896c-c0d4ab679cfa

State: Peer in Cluster (Connected)


On the "slave"

gluster> peer status

Number of Peers: 1

Hostname: 192.168.70.6

Uuid: 97ef0154-ad7b-402a-b0cb-22be09134a3c

State: Peer in Cluster (Connected)


gluster> volume status all

Status of volume: datastore1

Gluster processPortOnlinePid

--

Brick nas-01-data:/datastore49152Y2130

Brick nas-02-data:/datastoreN/ANN/A

NFS Server on localhost2049Y8064

Self-heal Daemon on localhostN/AY8073

NFS Server on 192.168.70.62049Y3379

Self-heal Daemon on 192.168.70.6N/AY3384


Which version of glusterfs are you running?

volume status suggests that the second brick (nas-02-data:/datastore) is 
not running.


Can you run "gluster volume start  force" in any of these two 
nodes and try again?
Then you would also required to run `find . | xargs stat` on the 
mountpoint of the volume. That should trigger the self heal.


There are no active volume tasks


I would like to run on the "slave" gluster volume sync nas-01-data 
datastore1


BTW, There is no concept of "master" and "slave" in afr (replication). 
However there is concept of "master volume" and "slave volume" in 
gluster geo-replication.


But then the virtual machines hosted will be unavailible is there 
another way to start the replication ?



Thanks






___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Block size reported from FUSE-client

[Anand Avati]

You are seeing a side-effect of http://review.gluster.com/3631. Which
means: if your backend filesystem uses 4KB blocks, then the value reported
by gluster will be at worst 7 blocks smaller (4KB / 512 - 1).


On Tue, Nov 26, 2013 at 3:13 AM, Maik Kulbe
wrote:

> So st_blocks on FUSE mount is different from st_blocks on backend for the
>> same file?
>>
>
> Yes. Just a quick example: I create a file with 5 Bytes in size. I theory
> Gluster should report 8 x 512 Byte blocks, because the underlying XFS uses
> a 4K block size. Instead, it reports the minimum count of blocks the file
> size would fit in:
>
> client> echo test > /gluster/tmp/test.txt
>
> client> stat /gluster/tmp/test.txt
>  File: `/gluster/tmp/test.txt'
>  Size: 5Blocks: 1  IO Block: 131072 regular file
> Device: 14h/20d Inode: 12072747239032953097  Links: 1
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
> Access: 2013-11-26 12:09:22.960828776 +0100
> Modify: 2013-11-26 12:09:22.964828962 +0100
> Change: 2013-11-26 12:09:22.964828962 +0100
> Birth: -
>
> gluster> stat /bricks/0/tmp/test.txt
>  File: `/bricks/0/tmp/test.txt'
>  Size: 5Blocks: 8  IO Block: 4096   regular file
> Device: ca03h/51715dInode: 859069733   Links: 2
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
> Access: 2013-11-26 12:09:22.957683891 +0100
> Modify: 2013-11-26 12:09:22.961684089 +0100
> Change: 2013-11-26 12:09:22.961684089 +0100
> Birth: -
>
>
>
>> On Nov 25, 2013 8:50 AM, "Maik Kulbe" 
>> wrote:
>>
>> From man (2) stat:
>>
>> blksize_t st_blksize; /* blocksize for file system I/O */
>> blkcnt_t  st_blocks;  /* number of 512B blocks allocated */
>>
>> The 128K  you are seeing is "st_blksize" which is the recommended I/O
>> transfer size. The number of consumed blocks is always reported as 512
>> byte blocks. The actual block size with which storage allocation
>> happens
>> depends on the backend filesystem.
>>
>> This is what was confusing me. On the file systems one of our
>> programmers tested the latter one it was always showing the blocks
>> allocated. So if you had a 1k file and 4k block size it would report 8
>> 512-byte blocks, gluster just reports 2 blocks.
>>
>> Avati
>>
>> On Mon, Nov 25, 2013 at 7:18 AM, Maik Kulbe
>>  wrote:
>>
>> Hi,
>>
>> I've come to notice that the file system block size reported from stat
>> on a client is 128k, which is pretty high for the small files I use.
>> On
>> the other hand, I tested copying smaller files to the volume and it
>> seems those 128k are not the real block size - when I copy two 64k
>> files
>> to the volume `df` reports only a change after both files have been
>> copied.
>>
>> So my question would be what is the real block size for the Gluster
>> volume? The block size of the underlying xfs? Or something else? And
>> is
>> it possible to read the real block size? We wanted to use the block
>> size
>> reported by stat to calculate the real file size use on disk but it
>> seems that is not possible with Gluster..
>>
>> Thank you in advance,
>> Maik
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Block size reported from FUSE-client

[Maik Kulbe]

So st_blocks on FUSE mount is different from st_blocks on backend for the
same file?


Yes. Just a quick example: I create a file with 5 Bytes in size. I theory 
Gluster should report 8 x 512 Byte blocks, because the underlying XFS uses a 4K 
block size. Instead, it reports the minimum count of blocks the file size would 
fit in:

client> echo test > /gluster/tmp/test.txt

client> stat /gluster/tmp/test.txt
 File: `/gluster/tmp/test.txt'
 Size: 5Blocks: 1  IO Block: 131072 regular file
Device: 14h/20d Inode: 12072747239032953097  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2013-11-26 12:09:22.960828776 +0100
Modify: 2013-11-26 12:09:22.964828962 +0100
Change: 2013-11-26 12:09:22.964828962 +0100
Birth: -

gluster> stat /bricks/0/tmp/test.txt
 File: `/bricks/0/tmp/test.txt'
 Size: 5Blocks: 8  IO Block: 4096   regular file
Device: ca03h/51715dInode: 859069733   Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2013-11-26 12:09:22.957683891 +0100
Modify: 2013-11-26 12:09:22.961684089 +0100
Change: 2013-11-26 12:09:22.961684089 +0100
Birth: -



On Nov 25, 2013 8:50 AM, "Maik Kulbe" 
wrote:

From man (2) stat:

blksize_t st_blksize; /* blocksize for file system I/O */
blkcnt_t  st_blocks;  /* number of 512B blocks allocated */

The 128K  you are seeing is "st_blksize" which is the recommended I/O
transfer size. The number of consumed blocks is always reported as 512
byte blocks. The actual block size with which storage allocation
happens
depends on the backend filesystem.

This is what was confusing me. On the file systems one of our
programmers tested the latter one it was always showing the blocks
allocated. So if you had a 1k file and 4k block size it would report 8
512-byte blocks, gluster just reports 2 blocks.

Avati

On Mon, Nov 25, 2013 at 7:18 AM, Maik Kulbe
 wrote:

Hi,

I've come to notice that the file system block size reported from stat
on a client is 128k, which is pretty high for the small files I use.
On
the other hand, I tested copying smaller files to the volume and it
seems those 128k are not the real block size - when I copy two 64k
files
to the volume `df` reports only a change after both files have been
copied.

So my question would be what is the real block size for the Gluster
volume? The block size of the underlying xfs? Or something else? And
is
it possible to read the real block size? We wanted to use the block
size
reported by stat to calculate the real file size use on disk but it
seems that is not possible with Gluster..

Thank you in advance,
Maik
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Changelog translator (was: Geo Replication Hooks)

[Maik Kulbe]

Gluster now has the changelog translator (journaling mechanism) which records
changes made to the filesystem (on each brick).


In which version will that be included? 3.5? And is there any documentation on 
performance of journal Geo-Rep vs. old Geo-Rep?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication Hooks

[Fred van Zwieten]

Hey Venky,

That sounds...promising. But, I would like to do stuff *after* a file is
changed on the source but *before* the change is pushed through a geo-repl
link to a target. When multi fan-out replication is done, it should even be
possible to have some custom stuff happening on one target and some other
custom stuff on another target.\

Is that possible without touching the Geo-replication code? Then, I am
*very* interested indeed.

Fred

On Tue, Nov 26, 2013 at 6:42 AM, Venky Shankar wrote:

> Hey Fred,
>
> You could implement this without touching Geo-Replication code. Gluster
> now has the changelog translator (journaling mechanism) which records
> changes made to the filesystem (on each brick). Journals can be consumed
> using the changelog consumer library (libgfchangelog). Geo-Replication is
> just one such consumer of the changelogs (earlier the change detection
> mechanism used to be a filesystem crawl based on xtime).
>
> Thinking from hook-scripts POV, you would even invoke them from
> libgfchangelog -- which gets notified by the changelog translator after a
> certain time interval. The granularity here would be a bunch of files
> (GFIDs actually, not file names) instead of per file basis. The library has
> APIs to mark changelogs as processed, which would be used to invoke post
> replication hooks (but this is only on source, not on destination).
>
> Let me know what you think of the above approaches.
>
> Thanks,
> -venky
>
>
> On Tue, Nov 26, 2013 at 3:13 AM, Fred van Zwieten  > wrote:
>
>> Hi,
>>
>> I have created a proposal for the implementation of Geo Replication
>> Hooks. See here:
>> http://www.gluster.org/community/documentation/index.php/Features/Geo_Replication_Hooks
>>
>> Any comments, thoughts, etc would be great.
>>
>> Fred
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users