Re: [Gluster-users] HELP on Geo-Replication Faulty rsync failed with ENOENT

[Venky Shankar]

I meant "master node"


On Sat, Mar 29, 2014 at 11:53 AM, Venky Shankar wrote:

> rsync is in /usr/bin of Slave VM. Why "rsync" failed with ENOENT (No file
>> or directory)?
>>
>
> what about the mater node?
>
>
>
>>
>> I basically follow the Gluster_FS_3.3.0_admin guide.
>> What did I miss? I have tried on Debian and CentOS, all failed.
>>
>> BTW, under CentOS 6.4, I cannot even stop the geo-replication.
>> I get 'geo-replication command failed'
>> But I can stop geo-replication  on a volume in Debian's GlusterFS.
>>
>> Looking for your helps and thanks in advance.
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] HELP on Geo-Replication Faulty rsync failed with ENOENT

[Venky Shankar]

>
> rsync is in /usr/bin of Slave VM. Why "rsync" failed with ENOENT (No file
> or directory)?
>

what about the mater node?



>
> I basically follow the Gluster_FS_3.3.0_admin guide.
> What did I miss? I have tried on Debian and CentOS, all failed.
>
> BTW, under CentOS 6.4, I cannot even stop the geo-replication.
> I get 'geo-replication command failed'
> But I can stop geo-replication  on a volume in Debian's GlusterFS.
>
> Looking for your helps and thanks in advance.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Nux!]

On 28.03.2014 12:07, Dave Christianson wrote:
I've come across individual posts from people who supposedly have done 
this
in CentOS6.5. Basically, all that is shown in the posts is the XML 
file
generated, no mention of *how* that file is generated. Virt-manager 
has no
provision for attaching directly to the gluster volume except as a 
mount.

Neither virt-manager nor virt-install recognize the gluster:// type.


I have used this functionality with Cloudstack on CentOS 6.5 with stock 
KVM, it works.

What you need is:
-drive 
file=gluster+tcp://storage.cloudstack.tld:24007/primary/d691ac19-4ec1-47c1-b765-55f804b78bec


Right now you need to fuse-mount the gluster volume to create the 
backing file, but you can start the vm using gfapi as above.
See this article from Nield de Vos who authored the native Glusterfs 
storage driver for Cloudstack:


http://www.gluster.org/2014/02/setting-up-a-test-environment-for-apache-cloudstack-and-gluster/

HTH
Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Dave Christianson]

Hi all,

I just installed a Fedora 20 test host, and after struggling with
dependencies I'm happy to report that everything works perfectly. I'm not
relying on the virt-manager gui at all, and from what I'm reading,
currently libvirt cannot start an image from gluster via libgfapi anyway(?).

For various reasons I don't wish to use Fedora in my production
environment, so until the SIG group works their magic, if there is a way to
compile the latest sources for qemu and libvirt on CentOS or Debian, I'll
be in business.

The performance difference between libgfapi and fuse is like night and day,
very impressive. I never want to have to go back to running VM's over fuse
again.

Thanks again to everyone,
David


On Fri, Mar 28, 2014 at 10:47 AM, Lalatendu Mohanty wrote:

>  On 03/28/2014 07:26 PM, SATHEESARAN wrote:
>
> On 03/28/2014 05:37 PM, Dave Christianson wrote:
>
>   I've come across individual posts from people who supposedly have done
> this in CentOS6.5. Basically, all that is shown in the posts is the XML
> file generated, no mention of *how* that file is generated. Virt-manager
> has no provision for attaching directly to the gluster volume except as a
> mount. Neither virt-manager nor virt-install recognize the gluster:// type.
>
> Yes, even I was frustrated at this. There seems no way to utilize the
> image file using libgfapi.
> As Harsha mentioned in his earlier thread, libvirt is yet to expose this.
> I have already raised a bug regarding this,
> https://bugzilla.redhat.com/show_bug.cgi?id=1017308
>
>  Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for
> qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm
> also doesn't recognize the gluster:// type.
>
> With RHEL 6.5 / Centos 6.5 you can use, "/usr/libexec/qemu-kvm"
> This was the command that I used to make use of VM Images using libgfapi
> in RHEL 6.5
> >> /usr/libexec/qemu-kvm  -drive file=gluster://
> 10.70.37.87/testvol/test.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads-device
> virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0
>
> The above worked for me.
> If I am not wrong, the libvirt in RHEL 7 supports glusterfs's libgfapi way
> of access mechanism.
>
>
> Sas,
>
> What about Fedora 20? Fedora 20 should have all the latest RPMs , so it
> should  have required patches in libvirt too.
>
> -Lala
>
>
>
>
>  Debian and Ubuntu supposedly have the newer versions of gluster and
> qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...
>
>  Red Hat seems content to do their own thing. Although the verisons of
> libvirt and qemu are older, libgfapi is supposed to have been backported.
> It's a shame that full functionality is not included. It's mindboggling
> seeing that Red Hat owns glusterfs, you would think full support for the
> backend would have been included in their product. If it is, as you say,
> that RH includes this functionality only to RHN subscribers and is not made
> available downstream to CentOS/SL, and unless I can find a repository with
> the latest full versions of qemu & libvirt, then CentOS simply will not
> work.
>
>
> On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana  > wrote:
>
>> Virt-manager / libvirt is yet to expose perhaps this functionality -
>> but as far as i remember libvirt should be doing this as a
>> pass-through for the URL's which have been passed  as
>> ":///"
>>
>> Does libvirt 'invoke' fuse when passed "gluster://" schema?
>>
>> On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
>>  wrote:
>> > Good Evening,
>> >
>> > I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5
>> (and
>> > by virtue CentOS and SL). However I am unable to figure out how to
>> actually
>> > make it work as described. Virt-manager still only seems to support
>> > glusterfs volumes via fuse.
>> >
>> > I can use qemu-img to create a disk image on
>> gluster:///.
>> > But virt-manager can only use it from a fuse mounted fileshare. There
>> seems
>> > to be no ability to attach in virt-manager directly to the image on
>> > glusterfs using libgfapi.
>> >
>> > All documents I've found describe the use of the command
>> > "qemu-system-x86_64," however that command does not exist in CentOS 6.5.
>> > That appears to be the only way to start the domain using libgfapi. So
>> > basically, I can create an image via libgfapi but cannot do anything
>> useful
>> > with it.
>> >
>> > Should I be able to do this? If so, what's the procedure? Or is
>> CentOS/RHEL
>> > 6.5 just not fully integrated? I really want to be able to use libgfapi
>> and
>> > avoid the performance penalty of fuse. Should I just grab & compile the
>> > latest verisons of libvirt and qemu-kvm?
>> >
>> > ___
>> > Gluster-users mailing list
>> > Gluster-users@gluster.org
>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> Religious confuse piety with mere ritual, t

[Gluster-users] HELP on Geo-Replication Faulty rsync failed with ENOENT

[Cary Tsai]

Setup 2 VMs and they are all CentOS 6.4.
All two are installed with GlusterFS 3.4.2 (server, client, and
geo-replication)
No Firewalls, all using 'root' account. All are in the same subnet.

After start the geo-replication, keep getting:
2014-03-26 20:38:26.401585] I [monitor(monitor):80:monitor] Monitor:

[2014-03-26 20:38:26.402067] I [monitor(monitor):81:monitor] Monitor:
starting gsyncd worker
[2014-03-26 20:38:26.442378] I [gsyncd:404:main_i] : syncing:
gluster://localhost:mirror -> ssh://10.1.10.52:/data/mirror
[2014-03-26 20:38:27.786715] I [master:60:gmaster_builder] : setting
up master for normal sync mode
[2014-03-26 20:38:28.808115] I [master:679:crawl] _GMaster: new master is
379cfc2c-257d-4be1-9719-6fe163197a0c
[2014-03-26 20:38:28.808331] I [master:683:crawl] _GMaster: primary master
with volume id 379cfc2c-257d-4be1-9719-6fe163197a0c ...
[2014-03-26 20:38:29.302051] E [syncdutils:174:log_raise_exception] :
execution of "rsync" failed with ENOENT (No such file or directory)
[2014-03-26 20:38:29.302360] I [syncdutils:148:finalize] : exiting.

rsync is in /usr/bin of Slave VM. Why "rsync" failed with ENOENT (No file
or directory)?

I basically follow the Gluster_FS_3.3.0_admin guide.
What did I miss? I have tried on Debian and CentOS, all failed.

BTW, under CentOS 6.4, I cannot even stop the geo-replication.
I get 'geo-replication command failed'
But I can stop geo-replication  on a volume in Debian's GlusterFS.

Looking for your helps and thanks in advance.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] outage post-mortem

[Nicolas Ochem]

Joe,
Thanks for your reply.
I grep'd the logs for the name of one of the files that had become
unreachable over NFS after resync (i/o error). It comes up in
.log and nfs.log on the node that had stayed online:
The relevant logs are here :
https://gist.github.com/nicolasochem/f9d24a2bf57b0d40bb7d

One important piece of information is that the node that was taken offline
had previously filled up the root filesystem because of a
memory/southbridge issue which filled /var/log/messages completely. Upon
restoration of the machine, gluster did not come up because one file in
/var/lib/gluster/peers was empty.
The issue is described there :
https://bugzilla.redhat.com/show_bug.cgi?id=858732

I removed the empty peer file, glusterd started, and then I started having
i/o errors described in my original mail.

The key log data is IMO : "background meta-data data missing-entry
self-heal failed on"

Based on this and the log, could it be that gluster failed to write to
/var/lib/gluster because of disk full, which caused issues ?


On Fri, Mar 28, 2014 at 8:13 AM, Joe Julian  wrote:

>
>
> On March 27, 2014 11:08:03 PM PDT, Nicolas Ochem 
> wrote:
> >Hi list,
> >I would like to describe an issue I had today with Gluster and ask for
> >opinion:
> >
> >I have a replicated mount with 2 replica. There is about 1TB of
> >production
> >data in there in around 100.000 files. They sit on 2x Supermicro
> >x9dr3-ln4f
> >machines with a RAID array of 18TB each, 64gb of ram, 2x Xeon CPUs, as
> >recommended in Red Hat hardware guidelines for storage server. They
> >have a
> >10gb link between each other. I am running gluster 3.4.2 on centos 6.5
> >
> >This storage is NFS-mounted to a lot of production servers. A very
> >little
> >part of this data is actually useful, the rest is legacy.
> >
> >Due to some unrelated issue with one of the supermicro server (faulty
> >memory), I had to take one of the nodes offline for 3 days.
> >
> >When I brought it back up, some files and directories ended up in
> >heal-failed state (but no split-brain). Unfortunately that were the
> >critical files that had been edited in the last 3 days. On the NFS
> >mounts,
> >attempts to read these files resulted in I/O error.
> >
> >I was able to fix a few of these files by manually removing them in
> >each
> >brick and then copying them to the mounted volume again. But I did not
> >know
> >what to do when full directories were unreachable because of "heal
> >failed".
> >
> >I later read that healing could take time and that heal-failed may be a
> >transient state (is that correct?
> >
> http://stackoverflow.com/questions/19257054/is-it-normal-to-get-a-lot-of-heal-failed-entries-in-a-gluster-mount
> ),
> >but at the time I thought that was beyond recovery, so I proceeded to
> >destroy the gluster volume. Then on one of the replicas I moved the
> >content
> >of the brick to another directory, created another volume with the same
> >name, then copied the content of the brick to the mounted volume. This
> >took
> >around 2 hours. Then I had to reboot all my NFS-mounted machines which
> >were
> >in "stale NFS file handle" state.
> >
> >Few questions :
> >- I realize that I cannot expect 1TB of data to heal instantly, but is
> >there any way for me to know if  the system would have recovered
> >eventually
> >despite being shown as "heal failed" ?
> >- if yes, what amount of files and filesize should I clean-up from my
> >volume to make this time go under 10 minutes ?
> >- would native gluster mounts instead of NFS have been of help here ?
> >- would any other course of action have resulted in faster recovery
> >time ?
> >- is there a way in such situation to make one replica have authority
> >about
> >the correct status of the filesystem  ?
> >
> >Thanks in advance for your replies.
> >
> >
> Although the self-heal daemon can take time to heal all the files,
> accessing a file that needs healed does trigger the heal to be performed
> immediately by the client (the nfs server is the client in this case).
>
> Like pretty much all errors in GlusterFS, you would have had to look in
> the logs to find why something as vague as "heal failed" happened.
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] deleted files directly in distribute brick

[Bruce Spidel]

Justin,

Thanks for the response. That is exactly what I needed. See I had deleted
the files but did not know which entries in .glusterfs to delete. The trick
is the link count = 1 and not linked to another file | grep -v ' -> ' that
gave me the complete list. There turned out to be 1k+ files and over 30T of
hidden storage.


Bruce
Joink LLC
812-231-7078


On Fri, Mar 28, 2014 at 3:05 PM, Justin Dossey  wrote:

> Since the files in .glusterfs are hard links, your data isn't actually
> gone-- you could restore the lost data in fact if you were able to
> determine the names of the files that were deleted based on the contents of
> the hard links in the .glusterfs directory.  Big "if" but we're talking
> about recovering data, so it's up to you.  If practicable, it's smart to
> keep a list of inode-filepath associations somewhere, but this probably
> doesn't help now.
>
> You can use find(1) to locate which of the .glusterfs files references one
> of the deleted files on the brick:
> find /path/to/.glusterfs -links 1 -ls
>
>
>
> On Thu, Mar 27, 2014 at 1:29 PM, Bruce Spidel 
> wrote:
>
>> In a moment of insanity I deleted a few hundred files in a gluster brick
>> (30TB of 133TB) which is part of a distribute volume. I want these files
>> deleted. but did not do it through the client. The volume is reporting a
>> size as if they are still there. The files in .glusterfs are still there
>> and I don't know which ones to delete.
>>
>> I think the data was linked from the primary directories and the
>> .glusterfs directories and when I deleted the files from the primary
>> directories the data did not delete because of the links in .glusterfs.
>>
>> Is it survivable to delete all of .glusterfs in one or all bricks and
>> hope that gluster will repair it?
>>
>>
>> Bruce
>> Joink LLC
>> 812-231-7078
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Justin Dossey
> CTO, PodOmatic
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] deleted files directly in distribute brick

[Justin Dossey]

Since the files in .glusterfs are hard links, your data isn't actually
gone-- you could restore the lost data in fact if you were able to
determine the names of the files that were deleted based on the contents of
the hard links in the .glusterfs directory.  Big "if" but we're talking
about recovering data, so it's up to you.  If practicable, it's smart to
keep a list of inode-filepath associations somewhere, but this probably
doesn't help now.

You can use find(1) to locate which of the .glusterfs files references one
of the deleted files on the brick:
find /path/to/.glusterfs -links 1 -ls



On Thu, Mar 27, 2014 at 1:29 PM, Bruce Spidel wrote:

> In a moment of insanity I deleted a few hundred files in a gluster brick
> (30TB of 133TB) which is part of a distribute volume. I want these files
> deleted. but did not do it through the client. The volume is reporting a
> size as if they are still there. The files in .glusterfs are still there
> and I don't know which ones to delete.
>
> I think the data was linked from the primary directories and the
> .glusterfs directories and when I deleted the files from the primary
> directories the data did not delete because of the links in .glusterfs.
>
> Is it survivable to delete all of .glusterfs in one or all bricks and hope
> that gluster will repair it?
>
>
> Bruce
> Joink LLC
> 812-231-7078
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>



-- 
Justin Dossey
CTO, PodOmatic
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] failed to mount for op-version

[Vijay Bellur]

On 03/28/2014 04:36 PM, Mingfan Lu wrote:

when I using client of gluster3.3 to mount a gluster3.4.2 volume, I got
a mount failed error for "0-glusterd: Client x.x.x.x:709 (1 -> 1)
doesn't support required op-version (2). Rejecting volfile request


Any comments?



Can you please provide your volume configuration (o/p of gluster volume 
info would do)?


Thanks,
Vijay

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Lalatendu Mohanty]

On 03/28/2014 07:26 PM, SATHEESARAN wrote:

On 03/28/2014 05:37 PM, Dave Christianson wrote:
I've come across individual posts from people who supposedly have 
done this in CentOS6.5. Basically, all that is shown in the posts is 
the XML file generated, no mention of *how* that file is generated. 
Virt-manager has no provision for attaching directly to the gluster 
volume except as a mount. Neither virt-manager nor virt-install 
recognize the gluster:// type.
Yes, even I was frustrated at this. There seems no way to utilize the 
image file using libgfapi.

As Harsha mentioned in his earlier thread, libvirt is yet to expose this.
I have already raised a bug regarding this, 
https://bugzilla.redhat.com/show_bug.cgi?id=1017308
Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for 
qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. 
Qemu-kvm also doesn't recognize the gluster:// type.

With RHEL 6.5 / Centos 6.5 you can use, "/usr/libexec/qemu-kvm"
This was the command that I used to make use of VM Images using 
libgfapi in RHEL 6.5
>> /usr/libexec/qemu-kvm  -drive 
file=gluster://10.70.37.87/testvol/test.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0


The above worked for me.
If I am not wrong, the libvirt in RHEL 7 supports glusterfs's libgfapi 
way of access mechanism.


Sas,

What about Fedora 20? Fedora 20 should have all the latest RPMs , so it 
should  have required patches in libvirt too.


-Lala





Debian and Ubuntu supposedly have the newer versions of gluster and 
qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...


Red Hat seems content to do their own thing. Although the verisons of 
libvirt and qemu are older, libgfapi is supposed to have been 
backported. It's a shame that full functionality is not included. 
It's mindboggling seeing that Red Hat owns glusterfs, you would think 
full support for the backend would have been included in their 
product. If it is, as you say, that RH includes this functionality 
only to RHN subscribers and is not made available downstream to 
CentOS/SL, and unless I can find a repository with the latest full 
versions of qemu & libvirt, then CentOS simply will not work.



On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana 
mailto:har...@harshavardhana.net>> wrote:


Virt-manager / libvirt is yet to expose perhaps this functionality -
but as far as i remember libvirt should be doing this as a
pass-through for the URL's which have been passed  as
":///"

Does libvirt 'invoke' fuse when passed "gluster://" schema?

On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
mailto:davidchristians...@gmail.com>> wrote:
> Good Evening,
>
> I have read that libgfapi has been backported to qemu-kvm in
RHEL 6.5 (and
> by virtue CentOS and SL). However I am unable to figure out how
to actually
> make it work as described. Virt-manager still only seems to support
> glusterfs volumes via fuse.
>
> I can use qemu-img to create a disk image on
gluster:///.
> But virt-manager can only use it from a fuse mounted fileshare.
There seems
> to be no ability to attach in virt-manager directly to the image on
> glusterfs using libgfapi.
>
> All documents I've found describe the use of the command
> "qemu-system-x86_64," however that command does not exist in
CentOS 6.5.
> That appears to be the only way to start the domain using
libgfapi. So
> basically, I can create an image via libgfapi but cannot do
anything useful
> with it.
>
> Should I be able to do this? If so, what's the procedure? Or is
CentOS/RHEL
> 6.5 just not fully integrated? I really want to be able to use
libgfapi and
> avoid the performance penalty of fuse. Should I just grab &
compile the
> latest verisons of libvirt and qemu-kvm?
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://supercolony.gluster.org/mailman/listinfo/gluster-users



--
Religious confuse piety with mere ritual, the virtuous confuse
regulation with outcomes




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] failed to mount for op-version

[Mingfan Lu]

For my cluster is newly installed, could I update the glusterd.info files
in each node to set the operating_veresion=1 ??
It seems work, the op_version of  volumes created are 1 now.


On Fri, Mar 28, 2014 at 7:12 PM, Carlos Capriotti <
capriotti.car...@gmail.com> wrote:

> Any chance you can install/update your client ?
>
>
> On Fri, Mar 28, 2014 at 12:06 PM, Mingfan Lu  wrote:
>
>> when I using client of gluster3.3 to mount a gluster3.4.2 volume, I got a
>> mount failed error for "0-glusterd: Client x.x.x.x:709 (1 -> 1) doesn't
>> support required op-version (2). Rejecting volfile request
>>
>>
>> Any comments?
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Justin Clift]

On 28/03/2014, at 12:07 PM, Dave Christianson wrote:
> I've come across individual posts from people who supposedly have done this 
> in CentOS6.5. Basically, all that is shown in the posts is the XML file 
> generated, no mention of *how* that file is generated. Virt-manager has no 
> provision for attaching directly to the gluster volume except as a mount. 
> Neither virt-manager nor virt-install recognize the gluster:// type.
> 
> Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for 
> qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm also 
> doesn't recognize the gluster:// type.
> 
> Debian and Ubuntu supposedly have the newer versions of gluster and 
> qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...
> 
> Red Hat seems content to do their own thing. Although the verisons of libvirt 
> and qemu are older, libgfapi is supposed to have been backported. It's a 
> shame that full functionality is not included. It's mindboggling seeing that 
> Red Hat owns glusterfs, you would think full support for the backend would 
> have been included in their product. If it is, as you say, that RH includes 
> this functionality only to RHN subscribers and is not made available 
> downstream to CentOS/SL, and unless I can find a repository with the latest 
> full versions of qemu & libvirt, then CentOS simply will not work.

As a thought, it might be helpful to ask on the libvirt-users mailing list:

  https://www.redhat.com/mailman/listinfo/libvirt-users
  http://libvirt.org/contact.html

Pretty sure the guys who did the actual integration work are on that mailing
list, so should be able to help. :)

Regards and best wishes,

Justin Clift

--
Open Source and Standards @ Red Hat

twitter.com/realjustinclift

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] outage post-mortem

[Joe Julian]

On March 27, 2014 11:08:03 PM PDT, Nicolas Ochem  
wrote:
>Hi list,
>I would like to describe an issue I had today with Gluster and ask for
>opinion:
>
>I have a replicated mount with 2 replica. There is about 1TB of
>production
>data in there in around 100.000 files. They sit on 2x Supermicro
>x9dr3-ln4f
>machines with a RAID array of 18TB each, 64gb of ram, 2x Xeon CPUs, as
>recommended in Red Hat hardware guidelines for storage server. They
>have a
>10gb link between each other. I am running gluster 3.4.2 on centos 6.5
>
>This storage is NFS-mounted to a lot of production servers. A very
>little
>part of this data is actually useful, the rest is legacy.
>
>Due to some unrelated issue with one of the supermicro server (faulty
>memory), I had to take one of the nodes offline for 3 days.
>
>When I brought it back up, some files and directories ended up in
>heal-failed state (but no split-brain). Unfortunately that were the
>critical files that had been edited in the last 3 days. On the NFS
>mounts,
>attempts to read these files resulted in I/O error.
>
>I was able to fix a few of these files by manually removing them in
>each
>brick and then copying them to the mounted volume again. But I did not
>know
>what to do when full directories were unreachable because of "heal
>failed".
>
>I later read that healing could take time and that heal-failed may be a
>transient state (is that correct?
>http://stackoverflow.com/questions/19257054/is-it-normal-to-get-a-lot-of-heal-failed-entries-in-a-gluster-mount),
>but at the time I thought that was beyond recovery, so I proceeded to
>destroy the gluster volume. Then on one of the replicas I moved the
>content
>of the brick to another directory, created another volume with the same
>name, then copied the content of the brick to the mounted volume. This
>took
>around 2 hours. Then I had to reboot all my NFS-mounted machines which
>were
>in "stale NFS file handle" state.
>
>Few questions :
>- I realize that I cannot expect 1TB of data to heal instantly, but is
>there any way for me to know if  the system would have recovered
>eventually
>despite being shown as "heal failed" ?
>- if yes, what amount of files and filesize should I clean-up from my
>volume to make this time go under 10 minutes ?
>- would native gluster mounts instead of NFS have been of help here ?
>- would any other course of action have resulted in faster recovery
>time ?
>- is there a way in such situation to make one replica have authority
>about
>the correct status of the filesystem  ?
>
>Thanks in advance for your replies.
>
>
Although the self-heal daemon can take time to heal all the files, accessing a 
file that needs healed does trigger the heal to be performed immediately by the 
client (the nfs server is the client in this case).  

Like pretty much all errors in GlusterFS, you would have had to look in the 
logs to find why something as vague as "heal failed" happened. 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Dave Christianson]

Fair enough, Kaleb. I shouldn't have let my frustration do the talking.

It's not a show-stopper for me - yet. At this stage I have enough freedom
with this project to keep tinkering and testing. Meanwhile I plan to follow
the SIG team's progress closely.

Thanks everyone for your valuable insight1
Dave


On Fri, Mar 28, 2014 at 10:15 AM, Kaleb S. KEITHLEY wrote:

> On 03/28/2014 08:07 AM, Dave Christianson wrote:
>
>>
>> Red Hat seems content to do their own thing. Although the versions of
>> libvirt and qemu are older, libgfapi is supposed to have been
>> backported. It's a shame that full functionality is not included. It's
>> mindboggling seeing that Red Hat owns glusterfs, you would think full
>> support for the backend would have been included in their product. If it
>> is, as you say, that RH includes this functionality only to RHN
>> subscribers and is not made available downstream to CentOS/SL, and
>> unless I can find a repository with the latest full versions of qemu &
>> libvirt, then CentOS simply will not work.
>>
>>
> I'm not sure that's a fair expectation.
>
> (As a side note, GlusterFS is a community project -- it's owned by the
> community. Red Hat has a product that it owns -- RHS or RHSS -- that is based
> on GlusterFS.)
>
> Core 'vanilla' RHEL is pretty conservative. Just because "... Red Hat owns
> GlusterFS..." (sic) doesn't mean we get to randomly update the libvirt that
> ships in RHEL. RHEL has its own QA cycle which gates when things like
> libvirt can be updated.
>
> As a result the newer libvirt has to be delivered in a separate channel.
> The fact that CentOS -- which is a clone of vanilla RHEL -- doesn't
> distribute some or all of the things that are in channels is not Red Hat's
> fault.
>
> If there are no RPMs of the newer libvirt available the fault lies with
> the community. The source for the newer libvirt is certainly available, as
> it always is -- it needs is someone in the community to package it for
> general consumption. Someone has done that for Ubuntu! And as John Mark has
> already indicated, that's something that the CentOS Storage SIG is intended
> to address.
>
> And finally, although I can't promise that it will be, it's entirely
> possible that the newer libvirt could be in RHEL 6.6, and then CentOS would
> have it too.
>
> --
>
> Kaleb
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Kaleb S. KEITHLEY]

On 03/28/2014 08:07 AM, Dave Christianson wrote:


Red Hat seems content to do their own thing. Although the versions of
libvirt and qemu are older, libgfapi is supposed to have been
backported. It's a shame that full functionality is not included. It's
mindboggling seeing that Red Hat owns glusterfs, you would think full
support for the backend would have been included in their product. If it
is, as you say, that RH includes this functionality only to RHN
subscribers and is not made available downstream to CentOS/SL, and
unless I can find a repository with the latest full versions of qemu &
libvirt, then CentOS simply will not work.



I'm not sure that's a fair expectation.

(As a side note, GlusterFS is a community project — it's owned by the 
community. Red Hat has a product that it owns — RHS or RHSS — that is 
based on GlusterFS.)


Core 'vanilla' RHEL is pretty conservative. Just because "... Red Hat 
owns GlusterFS..." (sic) doesn't mean we get to randomly update the 
libvirt that ships in RHEL. RHEL has its own QA cycle which gates when 
things like libvirt can be updated.


As a result the newer libvirt has to be delivered in a separate channel. 
The fact that CentOS — which is a clone of vanilla RHEL — doesn't 
distribute some or all of the things that are in channels is not Red 
Hat's fault.


If there are no RPMs of the newer libvirt available the fault lies with 
the community. The source for the newer libvirt is certainly available, 
as it always is — it needs is someone in the community to package it for 
general consumption. Someone has done that for Ubuntu! And as John Mark 
has already indicated, that's something that the CentOS Storage SIG is 
intended to address.


And finally, although I can't promise that it will be, it's entirely 
possible that the newer libvirt could be in RHEL 6.6, and then CentOS 
would have it too.


--

Kaleb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Lalatendu Mohanty]

On 03/28/2014 05:37 PM, Dave Christianson wrote:
I've come across individual posts from people who supposedly have done 
this in CentOS6.5. Basically, all that is shown in the posts is the 
XML file generated, no mention of *how* that file is generated. 
Virt-manager has no provision for attaching directly to the gluster 
volume except as a mount. Neither virt-manager nor virt-install 
recognize the gluster:// type.


Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for 
qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. 
Qemu-kvm also doesn't recognize the gluster:// type.


Debian and Ubuntu supposedly have the newer versions of gluster and 
qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...


Red Hat seems content to do their own thing. Although the verisons of 
libvirt and qemu are older, libgfapi is supposed to have been 
backported. It's a shame that full functionality is not included. It's 
mindboggling seeing that Red Hat owns glusterfs, you would think full 
support for the backend would have been included in their product. If 
it is, as you say, that RH includes this functionality only to RHN 
subscribers and is not made available downstream to CentOS/SL, and 
unless I can find a repository with the latest full versions of qemu & 
libvirt, then CentOS simply will not work.




I am pretty sure there would be some technical reason behind not having 
all support for libgfapi. May be somebody who has more knowledge about 
specifics should give some more context.


Regarding making functionality only to RHN subscribers and is not made 
available downstream to CentOS/SL,


 * It might not be related to source availability as libvirt is
   available under LGPL [2] , so nothing can be changed here about
   availability of source.
 * CentOS is built upon a particular version of RHEL which does not
   have the relevant package (which already been mentioned) [1]


To address these issues CentOS community is now working on special 
interest groups, where new and relevant packages can be provided which 
is is not present as part of core CentOS. Which is the exactly the case 
here. In the same line CentOS Storage SIG [3]  was proposed and already 
approved by CentOS board.


CentOS Storage SIG will provide all relevant packages for GlusterFS. The 
packages can have required upstream patches, so that community will not 
face an issue like this. The initiative has just started and we are 
working on it. Hopefully CentOS Storage SIG will solve a lot of problems 
like this and provide a easy way to deploy GlusterFS.


[1] 
http://wiki.centos.org/FAQ/General#head-d22edba1476618760e12d5d9529f354d2d46953a

[2] http://opensource.org/licenses/lgpl-license.html
[3] http://wiki.centos.org/SpecialInterestGroup/Storage/Proposal

Thanks,
Lala


On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana 
mailto:har...@harshavardhana.net>> wrote:


Virt-manager / libvirt is yet to expose perhaps this functionality -
but as far as i remember libvirt should be doing this as a
pass-through for the URL's which have been passed  as
":///"

Does libvirt 'invoke' fuse when passed "gluster://" schema?

On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
mailto:davidchristians...@gmail.com>> wrote:
> Good Evening,
>
> I have read that libgfapi has been backported to qemu-kvm in
RHEL 6.5 (and
> by virtue CentOS and SL). However I am unable to figure out how
to actually
> make it work as described. Virt-manager still only seems to support
> glusterfs volumes via fuse.
>
> I can use qemu-img to create a disk image on
gluster:///.
> But virt-manager can only use it from a fuse mounted fileshare.
There seems
> to be no ability to attach in virt-manager directly to the image on
> glusterfs using libgfapi.
>
> All documents I've found describe the use of the command
> "qemu-system-x86_64," however that command does not exist in
CentOS 6.5.
> That appears to be the only way to start the domain using
libgfapi. So
> basically, I can create an image via libgfapi but cannot do
anything useful
> with it.
>
> Should I be able to do this? If so, what's the procedure? Or is
CentOS/RHEL
> 6.5 just not fully integrated? I really want to be able to use
libgfapi and
> avoid the performance penalty of fuse. Should I just grab &
compile the
> latest verisons of libvirt and qemu-kvm?
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://supercolony.gluster.org/mailman/listinfo/gluster-users



--
Religious confuse piety with mere ritual, the virtuous confuse
regulation with outcomes




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Dave Christianson]

"/usr/libexec/qemu-kvm"

Actually, that did not work for me either. I issued the command in the form
of: "/usr/libexec/qemu-kvm -drive file=*gluster*://
*/l*,if=virtio". I got errors regarding inability
to open the image, and glusterfs not being a valid protocol. However, I
could create the image via libgfapi using qemu-img, I could see the file on
the filesystem, and the permissions are correct.

Also, I do have "option rpc-auth-allow-insecure on" in
/etc/glusterfs/glusterd.vol. Is there something else that's missing?

Thanks,
Dave



On Fri, Mar 28, 2014 at 9:56 AM, SATHEESARAN  wrote:

>  On 03/28/2014 05:37 PM, Dave Christianson wrote:
>
>   I've come across individual posts from people who supposedly have done
> this in CentOS6.5. Basically, all that is shown in the posts is the XML
> file generated, no mention of *how* that file is generated. Virt-manager
> has no provision for attaching directly to the gluster volume except as a
> mount. Neither virt-manager nor virt-install recognize the gluster:// type.
>
> Yes, even I was frustrated at this. There seems no way to utilize the
> image file using libgfapi.
> As Harsha mentioned in his earlier thread, libvirt is yet to expose this.
> I have already raised a bug regarding this,
> https://bugzilla.redhat.com/show_bug.cgi?id=1017308
>
>  Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for
> qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm
> also doesn't recognize the gluster:// type.
>
> With RHEL 6.5 / Centos 6.5 you can use, "/usr/libexec/qemu-kvm"
> This was the command that I used to make use of VM Images using libgfapi
> in RHEL 6.5
> >> /usr/libexec/qemu-kvm  -drive file=gluster://
> 10.70.37.87/testvol/test.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads-device
> virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0
>
> The above worked for me.
> If I am not wrong, the libvirt in RHEL 7 supports glusterfs's libgfapi way
> of access mechanism.
>
> -- Satheesaran
>
>
>  Debian and Ubuntu supposedly have the newer versions of gluster and
> qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...
>
>  Red Hat seems content to do their own thing. Although the verisons of
> libvirt and qemu are older, libgfapi is supposed to have been backported.
> It's a shame that full functionality is not included. It's mindboggling
> seeing that Red Hat owns glusterfs, you would think full support for the
> backend would have been included in their product. If it is, as you say,
> that RH includes this functionality only to RHN subscribers and is not made
> available downstream to CentOS/SL, and unless I can find a repository with
> the latest full versions of qemu & libvirt, then CentOS simply will not
> work.
>
>
> On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana  > wrote:
>
>> Virt-manager / libvirt is yet to expose perhaps this functionality -
>> but as far as i remember libvirt should be doing this as a
>> pass-through for the URL's which have been passed  as
>> ":///"
>>
>> Does libvirt 'invoke' fuse when passed "gluster://" schema?
>>
>> On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
>>  wrote:
>> > Good Evening,
>> >
>> > I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5
>> (and
>> > by virtue CentOS and SL). However I am unable to figure out how to
>> actually
>> > make it work as described. Virt-manager still only seems to support
>> > glusterfs volumes via fuse.
>> >
>> > I can use qemu-img to create a disk image on
>> gluster:///.
>> > But virt-manager can only use it from a fuse mounted fileshare. There
>> seems
>> > to be no ability to attach in virt-manager directly to the image on
>> > glusterfs using libgfapi.
>> >
>> > All documents I've found describe the use of the command
>> > "qemu-system-x86_64," however that command does not exist in CentOS 6.5.
>> > That appears to be the only way to start the domain using libgfapi. So
>> > basically, I can create an image via libgfapi but cannot do anything
>> useful
>> > with it.
>> >
>> > Should I be able to do this? If so, what's the procedure? Or is
>> CentOS/RHEL
>> > 6.5 just not fully integrated? I really want to be able to use libgfapi
>> and
>> > avoid the performance penalty of fuse. Should I just grab & compile the
>> > latest verisons of libvirt and qemu-kvm?
>> >
>> > ___
>> > Gluster-users mailing list
>> > Gluster-users@gluster.org
>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> Religious confuse piety with mere ritual, the virtuous confuse
>> regulation with outcomes
>>
>
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[SATHEESARAN]

On 03/28/2014 05:37 PM, Dave Christianson wrote:
I've come across individual posts from people who supposedly have done 
this in CentOS6.5. Basically, all that is shown in the posts is the 
XML file generated, no mention of *how* that file is generated. 
Virt-manager has no provision for attaching directly to the gluster 
volume except as a mount. Neither virt-manager nor virt-install 
recognize the gluster:// type.
Yes, even I was frustrated at this. There seems no way to utilize the 
image file using libgfapi.

As Harsha mentioned in his earlier thread, libvirt is yet to expose this.
I have already raised a bug regarding this, 
https://bugzilla.redhat.com/show_bug.cgi?id=1017308
Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for 
qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. 
Qemu-kvm also doesn't recognize the gluster:// type.

With RHEL 6.5 / Centos 6.5 you can use, "/usr/libexec/qemu-kvm"
This was the command that I used to make use of VM Images using libgfapi 
in RHEL 6.5
>> /usr/libexec/qemu-kvm  -drive 
file=gluster://10.70.37.87/testvol/test.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0


The above worked for me.
If I am not wrong, the libvirt in RHEL 7 supports glusterfs's libgfapi 
way of access mechanism.


-- Satheesaran



Debian and Ubuntu supposedly have the newer versions of gluster and 
qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...


Red Hat seems content to do their own thing. Although the verisons of 
libvirt and qemu are older, libgfapi is supposed to have been 
backported. It's a shame that full functionality is not included. It's 
mindboggling seeing that Red Hat owns glusterfs, you would think full 
support for the backend would have been included in their product. If 
it is, as you say, that RH includes this functionality only to RHN 
subscribers and is not made available downstream to CentOS/SL, and 
unless I can find a repository with the latest full versions of qemu & 
libvirt, then CentOS simply will not work.



On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana 
mailto:har...@harshavardhana.net>> wrote:


Virt-manager / libvirt is yet to expose perhaps this functionality -
but as far as i remember libvirt should be doing this as a
pass-through for the URL's which have been passed  as
":///"

Does libvirt 'invoke' fuse when passed "gluster://" schema?

On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
mailto:davidchristians...@gmail.com>> wrote:
> Good Evening,
>
> I have read that libgfapi has been backported to qemu-kvm in
RHEL 6.5 (and
> by virtue CentOS and SL). However I am unable to figure out how
to actually
> make it work as described. Virt-manager still only seems to support
> glusterfs volumes via fuse.
>
> I can use qemu-img to create a disk image on
gluster:///.
> But virt-manager can only use it from a fuse mounted fileshare.
There seems
> to be no ability to attach in virt-manager directly to the image on
> glusterfs using libgfapi.
>
> All documents I've found describe the use of the command
> "qemu-system-x86_64," however that command does not exist in
CentOS 6.5.
> That appears to be the only way to start the domain using
libgfapi. So
> basically, I can create an image via libgfapi but cannot do
anything useful
> with it.
>
> Should I be able to do this? If so, what's the procedure? Or is
CentOS/RHEL
> 6.5 just not fully integrated? I really want to be able to use
libgfapi and
> avoid the performance penalty of fuse. Should I just grab &
compile the
> latest verisons of libvirt and qemu-kvm?
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://supercolony.gluster.org/mailman/listinfo/gluster-users



--
Religious confuse piety with mere ritual, the virtuous confuse
regulation with outcomes




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[John Mark Walker]

Hi, 

I share your frustration. Just as an FYI, the CentOS storage SIG is designed to 
address this exact issue. 

You can see the proposal here: 
http://wiki.centos.org/SpecialInterestGroup/Storage/Proposal 

We are moving rapidly on this. In fact, for those of you in the bay area, we 
will have storage SIG hacking as part of the Data Liberate hackathon: 

http://dataliberate.eventbrite.com/ 

I'm hopeful that we will have a full stack solution in a matter of weeks. 
Lalatendu, who is heading up the effort from our end, can give more details. 

Whether we can meet your timeline is another matter. However, if Debian/Ubuntu 
meet your needs better, as the impartial community guy it is my recommendation 
that you use the best tool for the job. 

-JM 

- Original Message -

> I've come across individual posts from people who supposedly have done this
> in CentOS6.5. Basically, all that is shown in the posts is the XML file
> generated, no mention of *how* that file is generated. Virt-manager has no
> provision for attaching directly to the gluster volume except as a mount.
> Neither virt-manager nor virt-install recognize the gluster:// type.

> Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for
> qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm also
> doesn't recognize the gluster:// type.

> Debian and Ubuntu supposedly have the newer versions of gluster and
> qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...

> Red Hat seems content to do their own thing. Although the verisons of libvirt
> and qemu are older, libgfapi is supposed to have been backported. It's a
> shame that full functionality is not included. It's mindboggling seeing that
> Red Hat owns glusterfs, you would think full support for the backend would
> have been included in their product. If it is, as you say, that RH includes
> this functionality only to RHN subscribers and is not made available
> downstream to CentOS/SL, and unless I can find a repository with the latest
> full versions of qemu & libvirt, then CentOS simply will not work.

> On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana < har...@harshavardhana.net >
> wrote:

> > Virt-manager / libvirt is yet to expose perhaps this functionality -
> 
> > but as far as i remember libvirt should be doing this as a
> 
> > pass-through for the URL's which have been passed as
> 
> > ":///"
> 

> > Does libvirt 'invoke' fuse when passed "gluster://" schema?
> 

> > On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
> 
> > < davidchristians...@gmail.com > wrote:
> 
> > > Good Evening,
> 
> > >
> 
> > > I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5
> > > (and
> 
> > > by virtue CentOS and SL). However I am unable to figure out how to
> > > actually
> 
> > > make it work as described. Virt-manager still only seems to support
> 
> > > glusterfs volumes via fuse.
> 
> > >
> 
> > > I can use qemu-img to create a disk image on gluster:///.
> 
> > > But virt-manager can only use it from a fuse mounted fileshare. There
> > > seems
> 
> > > to be no ability to attach in virt-manager directly to the image on
> 
> > > glusterfs using libgfapi.
> 
> > >
> 
> > > All documents I've found describe the use of the command
> 
> > > "qemu-system-x86_64," however that command does not exist in CentOS 6.5.
> 
> > > That appears to be the only way to start the domain using libgfapi. So
> 
> > > basically, I can create an image via libgfapi but cannot do anything
> > > useful
> 
> > > with it.
> 
> > >
> 
> > > Should I be able to do this? If so, what's the procedure? Or is
> > > CentOS/RHEL
> 
> > > 6.5 just not fully integrated? I really want to be able to use libgfapi
> > > and
> 
> > > avoid the performance penalty of fuse. Should I just grab & compile the
> 
> > > latest verisons of libvirt and qemu-kvm?
> 
> > >
> 
> > > ___
> 
> > > Gluster-users mailing list
> 
> > > Gluster-users@gluster.org
> 
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 

> > --
> 
> > Religious confuse piety with mere ritual, the virtuous confuse
> 
> > regulation with outcomes
> 

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] failed to mount for op-version

[John Mark Walker]

- Original Message -

> no, we provide a service to our customers, it is not convenientto ask them
> all (200+) to update clients.
> Could we downgrade the op-version to 1 for a newly installed 3.4 cluster that
> not upgraded from 3.3?

My understanding is that you can - that it was the driving reason for 
introducing op versions in the first place. Unfortunately, I do not know the 
best practices for that. Anyone care to help? 

-JM 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Dave Christianson]

I've come across individual posts from people who supposedly have done this
in CentOS6.5. Basically, all that is shown in the posts is the XML file
generated, no mention of *how* that file is generated. Virt-manager has no
provision for attaching directly to the gluster volume except as a mount.
Neither virt-manager nor virt-install recognize the gluster:// type.

Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for
qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm
also doesn't recognize the gluster:// type.

Debian and Ubuntu supposedly have the newer versions of gluster and
qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy...

Red Hat seems content to do their own thing. Although the verisons of
libvirt and qemu are older, libgfapi is supposed to have been backported.
It's a shame that full functionality is not included. It's mindboggling
seeing that Red Hat owns glusterfs, you would think full support for the
backend would have been included in their product. If it is, as you say,
that RH includes this functionality only to RHN subscribers and is not made
available downstream to CentOS/SL, and unless I can find a repository with
the latest full versions of qemu & libvirt, then CentOS simply will not
work.


On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana
wrote:

> Virt-manager / libvirt is yet to expose perhaps this functionality -
> but as far as i remember libvirt should be doing this as a
> pass-through for the URL's which have been passed  as
> ":///"
>
> Does libvirt 'invoke' fuse when passed "gluster://" schema?
>
> On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
>  wrote:
> > Good Evening,
> >
> > I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5
> (and
> > by virtue CentOS and SL). However I am unable to figure out how to
> actually
> > make it work as described. Virt-manager still only seems to support
> > glusterfs volumes via fuse.
> >
> > I can use qemu-img to create a disk image on gluster:///.
> > But virt-manager can only use it from a fuse mounted fileshare. There
> seems
> > to be no ability to attach in virt-manager directly to the image on
> > glusterfs using libgfapi.
> >
> > All documents I've found describe the use of the command
> > "qemu-system-x86_64," however that command does not exist in CentOS 6.5.
> > That appears to be the only way to start the domain using libgfapi. So
> > basically, I can create an image via libgfapi but cannot do anything
> useful
> > with it.
> >
> > Should I be able to do this? If so, what's the procedure? Or is
> CentOS/RHEL
> > 6.5 just not fully integrated? I really want to be able to use libgfapi
> and
> > avoid the performance penalty of fuse. Should I just grab & compile the
> > latest verisons of libvirt and qemu-kvm?
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Religious confuse piety with mere ritual, the virtuous confuse
> regulation with outcomes
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] failed to mount for op-version

[Mingfan Lu]

no, we provide a service to our customers, it is not convenientto ask them
all (200+) to update clients.
Could we downgrade the op-version to 1 for a newly installed 3.4 cluster
that not upgraded from 3.3?


On Fri, Mar 28, 2014 at 7:12 PM, Carlos Capriotti <
capriotti.car...@gmail.com> wrote:

> Any chance you can install/update your client ?
>
>
> On Fri, Mar 28, 2014 at 12:06 PM, Mingfan Lu  wrote:
>
>> when I using client of gluster3.3 to mount a gluster3.4.2 volume, I got a
>> mount failed error for "0-glusterd: Client x.x.x.x:709 (1 -> 1) doesn't
>> support required op-version (2). Rejecting volfile request
>>
>>
>> Any comments?
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Users] Glusterd en Glusterfsd services are not set to autostart on install

[Sahina Bose]

On 03/28/2014 04:02 PM, Joop wrote:
Setting up oVirt with a separate cluster for storage using gluster we 
discovered that the gluster services are not set to autostart for the 
default runlevels.
Host(s) were clean Centos-6.5 hosts with the correct repositories and 
then added from the webui. Hosts got all the packages and gluster was 
correctly started for that session but after a reboot gluster isn't 
running and the host was set to non-operational.


Is this a known problem and if not should I file a BZ but under which 
category?


Which version of gluster are you using?

[Adding gluster-users]



Regards,

Joop

___
Users mailing list
us...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] failed to mount for op-version

[Carlos Capriotti]

Any chance you can install/update your client ?


On Fri, Mar 28, 2014 at 12:06 PM, Mingfan Lu  wrote:

> when I using client of gluster3.3 to mount a gluster3.4.2 volume, I got a
> mount failed error for "0-glusterd: Client x.x.x.x:709 (1 -> 1) doesn't
> support required op-version (2). Rejecting volfile request
>
>
> Any comments?
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] failed to mount for op-version

[Mingfan Lu]

when I using client of gluster3.3 to mount a gluster3.4.2 volume, I got a
mount failed error for "0-glusterd: Client x.x.x.x:709 (1 -> 1) doesn't
support required op-version (2). Rejecting volfile request


Any comments?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] outage post-mortem

[Nicolas Ochem]

Hi list,
I would like to describe an issue I had today with Gluster and ask for
opinion:

I have a replicated mount with 2 replica. There is about 1TB of production
data in there in around 100.000 files. They sit on 2x Supermicro x9dr3-ln4f
machines with a RAID array of 18TB each, 64gb of ram, 2x Xeon CPUs, as
recommended in Red Hat hardware guidelines for storage server. They have a
10gb link between each other. I am running gluster 3.4.2 on centos 6.5

This storage is NFS-mounted to a lot of production servers. A very little
part of this data is actually useful, the rest is legacy.

Due to some unrelated issue with one of the supermicro server (faulty
memory), I had to take one of the nodes offline for 3 days.

When I brought it back up, some files and directories ended up in
heal-failed state (but no split-brain). Unfortunately that were the
critical files that had been edited in the last 3 days. On the NFS mounts,
attempts to read these files resulted in I/O error.

I was able to fix a few of these files by manually removing them in each
brick and then copying them to the mounted volume again. But I did not know
what to do when full directories were unreachable because of "heal failed".

I later read that healing could take time and that heal-failed may be a
transient state (is that correct?
http://stackoverflow.com/questions/19257054/is-it-normal-to-get-a-lot-of-heal-failed-entries-in-a-gluster-mount),
but at the time I thought that was beyond recovery, so I proceeded to
destroy the gluster volume. Then on one of the replicas I moved the content
of the brick to another directory, created another volume with the same
name, then copied the content of the brick to the mounted volume. This took
around 2 hours. Then I had to reboot all my NFS-mounted machines which were
in "stale NFS file handle" state.

Few questions :
- I realize that I cannot expect 1TB of data to heal instantly, but is
there any way for me to know if  the system would have recovered eventually
despite being shown as "heal failed" ?
- if yes, what amount of files and filesize should I clean-up from my
volume to make this time go under 10 minutes ?
- would native gluster mounts instead of NFS have been of help here ?
- would any other course of action have resulted in faster recovery time ?
- is there a way in such situation to make one replica have authority about
the correct status of the filesystem  ?

Thanks in advance for your replies.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?

[Harshavardhana]

Virt-manager / libvirt is yet to expose perhaps this functionality -
but as far as i remember libvirt should be doing this as a
pass-through for the URL's which have been passed  as
":///"

Does libvirt 'invoke' fuse when passed "gluster://" schema?

On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson
 wrote:
> Good Evening,
>
> I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5 (and
> by virtue CentOS and SL). However I am unable to figure out how to actually
> make it work as described. Virt-manager still only seems to support
> glusterfs volumes via fuse.
>
> I can use qemu-img to create a disk image on gluster:///.
> But virt-manager can only use it from a fuse mounted fileshare. There seems
> to be no ability to attach in virt-manager directly to the image on
> glusterfs using libgfapi.
>
> All documents I've found describe the use of the command
> "qemu-system-x86_64," however that command does not exist in CentOS 6.5.
> That appears to be the only way to start the domain using libgfapi. So
> basically, I can create an image via libgfapi but cannot do anything useful
> with it.
>
> Should I be able to do this? If so, what's the procedure? Or is CentOS/RHEL
> 6.5 just not fully integrated? I really want to be able to use libgfapi and
> avoid the performance penalty of fuse. Should I just grab & compile the
> latest verisons of libvirt and qemu-kvm?
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users



-- 
Religious confuse piety with mere ritual, the virtuous confuse
regulation with outcomes
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users