Re: [Gluster-users] Folder disappeared on the mount point , but exists on bricks.

On 10/20/2016 07:21 AM, yujin wrote: yes i check the trusted.gfid xattr for luo on all bricks: as below： -- getfattr -m . -d -e hex /data2/gluster/video/luo getfattr: Removing leading '/' from absolute path names # file: data2/gluster/video/luo

Re: [Gluster-users] locking problem on 3.8.5

I had a chance to look at the logs from both the nodes. I could see a repetitive instance of the following logs (in both the nodes): "Lock for Oracle_Legal_04 held by bb12d7e7-ded5-4d32-b294-8f5011f70afb" What that means is node with UUID is the culprit. However I doubt that the logs you shared

Re: [Gluster-users] periodic calls to gluster

On 18/10/2016 6:38 PM, Niels de Vos wrote: Nice! Thanks! I had reason to restart a node this morning, I've attached a log graph. And I get emails like the following: Subject: PROBLEM: Heal 200 Trigger: Heal 200 Trigger status: PROBLEM Trigger severity: High Trigger URL:

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 10:03 AM, Joe Julian wrote: Heh, well then a little less scary might be: find /var/log/glusterfs -type f Then if the list looks correct you can easily find /var/log/glusterfs -type f | xargs truncate --size=0 Thanks, I'll script that up. Will put together a test plan

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 10/19/2016 04:59 PM, Lindsay Mathieson wrote: On 20/10/2016 9:48 AM, Joe Julian wrote: Personally, with logrotate I use copytruncate. If I want to truncate the whole shebang manually: find -type f -exec truncate --size=0 {} \; Eeep! that looks scary, I just see me running that from

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 9:48 AM, Joe Julian wrote: Personally, with logrotate I use copytruncate. If I want to truncate the whole shebang manually: find -type f -exec truncate --size=0 {} \; Eeep! that looks scary, I just see me running that from root at 3am. -- Lindsay Mathieson

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 10/19/2016 04:46 PM, Lindsay Mathieson wrote: On 20/10/2016 9:30 AM, Joe Julian wrote: Joe, when you say truncated, just delete the logs before the test is started? That's one way of doing it, yes. :) nb: I've noticed that if you delete the logs while gluster is running they won't be

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 9:30 AM, Joe Julian wrote: Joe, when you say truncated, just delete the logs before the test is started? That's one way of doing it, yes. :) nb: I've noticed that if you delete the logs while gluster is running they won't be recreated, you have to restart the services. Is

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 9:31 AM, Joe Julian wrote: That makes me think, too, that it might be nice to see what kind of VM proxmox creates. I assume it uses libvirt so a dumpxml of an affected VM might be telling as well Unfortunately not, they use their own config file and build a cmd line from that,

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 10/19/2016 04:21 PM, Kevin Lemonnier wrote: If you have clean logs (truncated before you reproduced) everything from /var/log/glusterfs from the client and all the servers would be great. I'm assuming that proxmox doesn't redirect the client logs to some other place. No client logs with

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 10/19/2016 04:28 PM, Lindsay Mathieson wrote: On 20/10/2016 9:21 AM, Kevin Lemonnier wrote: No client logs with proxmox unfortunatly, you need to start qemu by hand to get those as far as I know. Ah, those logs! forgot about that. I'll replicate it again soo. Might take a couple of days

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 9:21 AM, Kevin Lemonnier wrote: No client logs with proxmox unfortunatly, you need to start qemu by hand to get those as far as I know. Ah, those logs! forgot about that. I'll replicate it again soo. Might take a couple of days and create a bug report. If you have clean

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

> > If you have clean logs (truncated before you reproduced) everything from > /var/log/glusterfs from the client and all the servers would be great. > I'm assuming that proxmox doesn't redirect the client logs to some other > place. > No client logs with proxmox unfortunatly, you need to

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 10/19/2016 04:04 PM, Lindsay Mathieson wrote: On 20/10/2016 8:43 AM, Joe Julian wrote: Is there a bug open with the client and server logs attached? I would take a stab at reproducing as well, but I want to make sure I'm comparing apples to apples. Which log names are those (sorry). I

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 8:43 AM, Joe Julian wrote: Is there a bug open with the client and server logs attached? I would take a stab at reproducing as well, but I want to make sure I'm comparing apples to apples. Which log names are those (sorry). I have all the logs from this mornings trial

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

Is there a bug open with the client and server logs attached? I would take a stab at reproducing as well, but I want to make sure I'm comparing apples to apples. On 10/19/2016 02:39 PM, Kevin Lemonnier wrote: Looked like it was going ok for a while, then blew up. The first windows vm which

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

> > Looked like it was going ok for a while, then blew up. The first windows > vm which was running diskmark died and won't boot. qemu-img check shows > the image hopelessly corrupted. 2nd VM has also crashed and is > unbootable, though qemuimg shows the qcow2 file as ok. > Ha, glad you

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 7:01 AM, Kevin Lemonnier wrote: Yes, you need to add a full replica set at once. I don't remember, but according to my history, looks like I've used this : gluster volume add-brick VMs host1:/brick host2:/brick host3:/brick force (I have the same without force just before that,

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

> > Ok, I'll give that a go > - what command do you use? > - I think from memory the extra bricks have to be added in blocks of three? > Yes, you need to add a full replica set at once. I don't remember, but according to my history, looks like I've used this : gluster volume add-brick VMs

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 20/10/2016 6:39 AM, Kevin Lemonnier wrote: Did you add a third brick in replication, if I understand it correctly ? Yup, took it from rep 2 to rep 3 That's not the problem I had, that does seem to be working fine. What broke was adding 3 new bricks to a replica 3, bumping it from 1 x 3 to

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

> > I had a go at reproducing it last night Kevin with 3.8.4 > > - new volume > > - Initial 2 bricks on two hosts > > - copied a windows VM on to it > > - Started some load (Crystal DiskMark in the VM) > > - Added a 3rd brick and node > For the size it's already more that what I have, most

Re: [Gluster-users] Healing Delays - possibly solved

On 2/10/2016 12:48 AM, Lindsay Mathieson wrote: This was raised earlier but I don't believe it was ever resolved and it is becoming a serious issue for me. I'm doing rolling upgrades on our three node cluster (Replica 3, Sharded, VM Workload). I update one node, reboot it, wait for

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 19/10/2016 11:08 PM, Kevin Lemonnier wrote: Yes, to be honest I wasn't even planning on rebalancing just yet, I was planning on letting it run a few days before, see if everything's fine, and then maybe rebalance. I tried the rebalance when everything came crashing down, hoping that might fix

Re: [Gluster-users] gluster volume not mounted on boot

On 20/10/2016 2:17 AM, Josep Manel Andrés wrote: Hi , I am trying to mount a volume during boot time, here are the logs from boot.log, everything seems ok: Can we see - your fstab entry for the gluster mount - the client log (/var/log/gluster/-.log) -- Lindsay Mathieson

[Gluster-users] gluster volume not mounted on boot

Hi , I am trying to mount a volume during boot time, here are the logs from boot.log, everything seems ok: [ OK ] Started GlusterFS, a clustered file-system server. [ OK ] Reached target Network is Online. Starting Login and scanning of iSCSI devices... Mounting

Re: [Gluster-users] Can't delete or move file off mounted brick

On 10/18/2016 11:31 AM, Tim wrote: But I cannot find any log relating to this on the gluster servers. Also I can cat the file no problem if I login to the bricks. What could be a reason of it not working on the mount point… Can you provide the output of `stat(1) ` and the output of

Re: [Gluster-users] Folder disappeared on the mount point , but exists on bricks.

On 10/19/2016 01:29 PM, yujin wrote: hi all ** I've just had an interestingissuewith our gluster storage: Folder disappeared on the mount point , but exists on bricks. e.g: mount point /videos_store subdirectory "video/luo" is exists on all bricks. when "ls

Re: [Gluster-users] trashcan file size limit

Thanks Jiffin, filed https://bugzilla.redhat.com/show_bug.cgi?id=1386766 In my limited knowledge of the original reasons for 1GB hardcode, either removing the limit altogether, or an additional “override" option parameter would be

[Gluster-users] Folder disappeared on the mount point , but exists on bricks.

hi all ** I've just had an interestingissuewith our gluster storage: Folder disappeared on the mount point , but exists on bricks. e.g: mount point /videos_store subdirectory "video/luo" is exists on all bricks. when "ls /videos_store/video/" the "luo" was

[Gluster-users] Folder disappeared on the mount point , but exists on bricks.

hi all ** I've just had an interestingissuewith our gluster storage: Folder disappeared on the mount point , but exists on bricks. e.g: mount point /videos_store subdirectory "video/luo" is exists on all bricks. when "ls /videos_store/video/" the "luo" was

[Gluster-users] Can't delete or move file off mounted brick

Hi gluster-users! I’ve got an issue currently with my 2x brick replicate volume. The volume is fine, and both bricks are replicating fine, but there is 1 file which refuses to be deleted or moved (and now can’t be viewed via a glusterfs mount) We just upgraded from |3.7.13-ubuntu1~trusty1| to

[Gluster-users] Weekly Community Meeting 19 Oct 2016 - Minutes

Hi all, Thank you to all the participants in today's community meeting. The next meeting is scheduled next week (October 26th) at #gluster-meeting on freenode. The minutes, logs and a summary for today's meeting can be found below. Minutes:

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

> > As Kevin said, the problem appeared before rebalancing if I understood > correctly. > Yes, to be honest I wasn't even planning on rebalancing just yet, I was planning on letting it run a few days before, see if everything's fine, and then maybe rebalance. I tried the rebalance when

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

On 19/10/16 14:59, Gandalf Corvotempesta wrote: Il 19 ott 2016 14:32, "Xavier Hernandez" > ha scritto: I had a similar issue while moving machines from an old gluster volume to a new volume with sharding enabled and I added new bricks to

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

Il 19 ott 2016 14:32, "Xavier Hernandez" ha scritto: > I had a similar issue while moving machines from an old gluster volume to a new volume with sharding enabled and I added new bricks to it. Maybe related to rebalance after adding bricks on a sharded volume? maybe that

Re: [Gluster-users] rot-13 Translator query

Xavi/Ravi, Thanks for the inputs. After disabling all the performance xlators on the client side and attaching to brick daemons before actually mounting the gluster volume I was able to get in to posix_readv function. Thanks and Regards, Ram -Original Message-

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

> are you using Proxmox, right ? > Yes, indeed. > I think it's important because Proxmox uses gfapi to connect each VM to > the disk, not FUSE. Maybe this is important to find the cause. I believe so yes, and I should add (I believe I mentionned it) that I am using GlusterFS 3.7.12. It took a

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

Hi Kevin, are you using Proxmox, right ? I think it's important because Proxmox uses gfapi to connect each VM to the disk, not FUSE. Maybe this is important to find the cause. I had a similar issue while moving machines from an old gluster volume to a new volume with sharding enabled and I

Re: [Gluster-users] trashcan file size limit

Hi Jackie, On 18/10/16 23:48, Jackie Tung wrote: Hi all, Documentation says: https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Trash/ */gluster volume set features.trash-max-filesize /* This command can be used to filter files entering trash directory based on their size.

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

Agreed. I will run the same test on an actual vm setup one of these days and see if I manage to recreate the issue (after I have completed some of my long pending tasks). Meanwhile if any of you find a consistent simpler test case to hit the issue, feel free to reply on this thread. At least I had