Re: [ceph-users] Install question

[Robert van Leeuwen]

> I am trying to install Ceph on a Red Hat Linux server that does not have 
> external access
> through which it can access the URL’s and download the files needed.   The 
> documentation is
> not clear (to me) on how to install the software under these circumstances.

>Should I be downloading the source files and creating my own installable 
>packages, or is
>there another easier way to do it?

As mentioned, there are RPM's which are ready to go on the Ceph site.
However, you will have to make sure you also have the dependencies of the RPM, 
and the dependencies of the dependencies...
You probably see where this is going, and that is why we have yum and 
repositories on Redhat ;)

If remotely possible I think it would be best to synchronise the Ceph and EPEL 
repository to a place that the server can reach.
I assume there is a machine reachable which, at least, serves the Redhat RPM's, 
maybe the EPEL and Ceph repo's can be added there?
If not, make sure you have a lot of time and patience to copy stuff around.

Cheers,
Robert van Leeuwen


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] upgrade from bobtail to dumpling

[Dominik Mostowiec]

hi,
It is possible to (safe) upgrade directly from bobtail (0.56.6) to
dumpling (latest)?
Is there any instruction?

-- 
Regards
Dominik
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-create-keys hung

[Abhay Sachan]

Hi Joao,
Thanks for replying. All of my monitors are up and running and connected to
each other. "ceph -s" is failing on the cluster with following error:

2013-10-07 10:12:25.099261 7fd1b948d700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-10-07 10:12:25.099271 7fd1b948d700  0 librados: client.admin
initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

And the logs on each monitor has lots of entries like this:
NODE 1:

2013-10-07 03:58:51.153847 7ff2864c6700  0 mon.jul@0(probing).data_health(0)
update_stats avail 76% total 42332700 used 7901820 avail 32280480
2013-10-07 03:59:51.154051 7ff2864c6700  0 mon.jul@0(probing).data_health(0)
update_stats avail 76% total 42332700 used 7901832 avail 32280468
2013-10-07 04:00:51.154256 7ff2864c6700  0 mon.jul@0(probing).data_health(0)
update_stats avail 76% total 42332700 used 7901828 avail 32280472

NODE 2:
2013-10-07 10:12:10.345491 7fa6145b0700  0 mon.dec@0(probing).data_health(0)
update_stats avail 75% total 42332700 used 8199004 avail 31983296
2013-10-07 10:13:10.345677 7fa6145b0700  0 mon.dec@0(probing).data_health(0)
update_stats avail 75% total 42332700 used 8199004 avail 31983296
2013-10-07 10:14:10.345921 7fa6145b0700  0 mon.dec@0(probing).data_health(0)
update_stats avail 75% total 42332700 used 8199024 avail 31983276

NODE 3:

2013-10-07 10:13:00.880250 7fcd6459e700  0 mon.julilo@0(probing).data_health(0)
update_stats avail 35% total 42332700 used 25105920 avail
150763802013-10-07 10:14:00.880470 7fcd6459e700  0
mon.julilo@0(probing).data_health(0)
update_stats avail 35% total 42332700 used 25105924 avail 15076376
2013-10-07 10:15:00.880668 7fcd6459e700  0 mon.julilo@0(probing).data_health(0)
update_stats avail 35% total 42332700 used 25105924 avail 15076376

If you need some other logs, then please tell me how to enable/fetch them.
I will upload them someplace.


Regards,
Abhay

On Thu, Oct 3, 2013 at 8:31 PM, Joao Eduardo Luis wrote:

> On 10/03/2013 02:44 PM, Abhay Sachan wrote:
>
>> Hi All,
>> I have tried setting up a ceph cluster with 3 nodes (3 monitors). I am
>> using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster
>> creation (using ceph-deploy as well as mkcephfs), ceph-creates-keys
>> doesn't return on any of the servers. Whereas, if I create a cluster
>> with only 1 node (1 monitor), key creation goes through. Has anybody
>> seen this problem or any ideas what I might be missing??
>>
>> Regards,
>> Abhay
>>
>
> Those symptoms tell me that your monitors are not forming quorum.
> 'ceph-create-keys' needs the monitors to first establish a quorum,
> otherwise it will hang waiting for that to happen.
>
> Please make sure all your monitors are running.  If so, try running 'ceph
> -s' on your cluster.  If that hangs as well, try accessing each monitor's
> admin socket to check what's happening [1].  If that too fails, try looking
> into the logs for something obviously wrong.  If you are not able to
> discern anything useful at that point, upload the logs to some place and
> point us to them -- we'll then be happy to take a look.
>
> Hope this helps.
>
>   -Joao
>
> --
> Joao Eduardo Luis
> Software Engineer | http://inktank.com | http://ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data loss after force umount !

[higkoohk]

Thanks everyone ,

I think `umount -l` is the error,we shouldn't just do this operation
without any other conjunction operation.

I will continue to do more extreme test. I shouldn't exec `umount -l` ,
and I need to stop anyone to exec `umount -l`.

Lots of thanks !

-- Forwarded message --
From: Michael Lowe 
Date: 2013/10/8
Subject: Re: [ceph-users] Data loss after force umount !
To: higkoohk 


It won't unmount until the processes with open files exit, umount -l is
usually used in conjunction with lsof and kill.  You probably didn't
actually get the file system unmounted.

Sent from my iPad

On Oct 7, 2013, at 9:40 PM, higkoohk  wrote:

Hi Michael ,

umount -l : Lazy  unmount.  Detach  the  filesystem  from  the filesystem
hierarchy now, and cleanup all references to the filesystem as soon as it
is not busy anymore.

Do you mean '-f' ?

umount  -f :  Force unmount (in case of an unreachable NFS system).


But what do I do when `umount -l` be exec ?



2013/10/8 Michael Lowe 
That doesn't force the unmount.

Sent from my iPad

2013/10/8 Yan, Zheng 

>
> 2013-10-8 上午9:00于 "higkoohk" 写道：
>
> >
> > Thanks everyone, the env like this :
> >
> > Linux 3.0.97-1.el6.elrepo.x86_64 CentOS 6.4
> >
> > ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
> >
> > /dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw)
> > /dev/sdb1 on /var/lib/ceph/osd/ceph-3 type xfs (rw)
> > /dev/sdc1 on /var/lib/ceph/osd/ceph-4 type xfs (rw)
> >
> > meta-data=/dev/sdb1  isize=2048   agcount=4, agsize=8895321
> blks
> >  =   sectsz=512   attr=2, projid32bit=0
> > data =   bsize=4096   blocks=35581281, imaxpct=25
> >  =   sunit=0  swidth=0 blks
> > naming   =version 2  bsize=4096   ascii-ci=0
> > log  =internal   bsize=4096   blocks=17373, version=2
> >  =   sectsz=512   sunit=0 blks, lazy-count=1
> > realtime =none   extsz=4096   blocks=0, rtextents=0
> >
> > /usr/libexec/qemu-kvm -name centos6-clone2 -S -machine
> rhel6.4.0,accel=kvm -m 1000 -smp 2,sockets=2,cores=1,threads=1 -uuid
> dd1a7093-bdea-4816-8a62-df61cb0c9bfa -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos6-clone2.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=rbd:rbd/centos6-clone2:auth_supported=none:mon_host=
> agent21.kisops.org\:6789,if=none,id=drive-virtio-disk0 -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5c:71:f1,bus=pci.0,addr=0x3
> -vnc 0.0.0.0:0 -vga cirrus -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
> > Use `umount -l` to force umount.
>
> `umount -l` is not force umount, it just detach the fs from the hierarchy.
> The fs is mounted internelly in the kernel if there still are references.
> I'm not surprise the fs errors If you use `umount -l` to umount the rbd in
> one guest, then mount the  rbd in another guest.
>
> Yan, zheng
>
> >
> > Anything else ?
> >
> >
> > 2013/10/8 Mark Nelson 
> >>
> >> Also, mkfs, mount, and kvm disk options?
> >>
> >> Mark
> >>
> >>
> >> On 10/07/2013 03:15 PM, Samuel Just wrote:
> >>>
> >>> Sounds like it's probably an issue with the fs on the rbd disk?  What
> >>> fs was the vm using on the rbd?
> >>> -Sam
> >>>
> >>> On Mon, Oct 7, 2013 at 8:11 AM, higkoohk  wrote:
> 
>  We use ceph as the storage of kvm .
> 
>  I found the VMs errors when force umount the ceph disk.
> 
>  Is it just right ? How to repair it ?
> 
>  Many thanks .
> 
>  --higkoo
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data loss after force umount !

[Yan, Zheng]

2013-10-8 上午9:00于 "higkoohk" 写道：
>
> Thanks everyone, the env like this :
>
> Linux 3.0.97-1.el6.elrepo.x86_64 CentOS 6.4
>
> ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
>
> /dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw)
> /dev/sdb1 on /var/lib/ceph/osd/ceph-3 type xfs (rw)
> /dev/sdc1 on /var/lib/ceph/osd/ceph-4 type xfs (rw)
>
> meta-data=/dev/sdb1  isize=2048   agcount=4, agsize=8895321
blks
>  =   sectsz=512   attr=2, projid32bit=0
> data =   bsize=4096   blocks=35581281, imaxpct=25
>  =   sunit=0  swidth=0 blks
> naming   =version 2  bsize=4096   ascii-ci=0
> log  =internal   bsize=4096   blocks=17373, version=2
>  =   sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none   extsz=4096   blocks=0, rtextents=0
>
> /usr/libexec/qemu-kvm -name centos6-clone2 -S -machine
rhel6.4.0,accel=kvm -m 1000 -smp 2,sockets=2,cores=1,threads=1 -uuid
dd1a7093-bdea-4816-8a62-df61cb0c9bfa -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos6-clone2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:rbd/centos6-clone2:auth_supported=none:mon_host=agent21.kisops.org\:6789,if=none,id=drive-virtio-disk0
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5c:71:f1,bus=pci.0,addr=0x3
-vnc 0.0.0.0:0 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
> Use `umount -l` to force umount.

`umount -l` is not force umount, it just detach the fs from the hierarchy.
The fs is mounted internelly in the kernel if there still are references.
I'm not surprise the fs errors If you use `umount -l` to umount the rbd in
one guest, then mount the  rbd in another guest.

Yan, zheng
>
> Anything else ?
>
>
> 2013/10/8 Mark Nelson 
>>
>> Also, mkfs, mount, and kvm disk options?
>>
>> Mark
>>
>>
>> On 10/07/2013 03:15 PM, Samuel Just wrote:
>>>
>>> Sounds like it's probably an issue with the fs on the rbd disk?  What
>>> fs was the vm using on the rbd?
>>> -Sam
>>>
>>> On Mon, Oct 7, 2013 at 8:11 AM, higkoohk  wrote:

 We use ceph as the storage of kvm .

 I found the VMs errors when force umount the ceph disk.

 Is it just right ? How to repair it ?

 Many thanks .

 --higkoo
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] About Ceph SSD and HDD strategy

[Jason Villalta]

I tried putting Flashcache on my spindle OSDs using an Intel SSL and it
works great.  This is getting me read and write SSD caching instead of just
write performance on the journal.  It should also allow me to protect the
OSD journal on the same drive as the OSD data and still get benefits of SSD
caching for writes.


On Mon, Oct 7, 2013 at 11:43 AM, Jason Villalta  wrote:

> I found this without much effort.
>
> http://www.sebastien-han.fr/blog/2012/11/15/make-your-rbd-fly-with-flashcache/
>
>
> On Mon, Oct 7, 2013 at 11:39 AM, Jason Villalta wrote:
>
>> I also would be interested in how bcache or flashcache would integrate.
>>
>>
>> On Mon, Oct 7, 2013 at 11:34 AM, Martin Catudal wrote:
>>
>>> Thank's Mike,
>>>  Kyle Bader suggest me also to use my large SSD (900 GB) as cache
>>> drive using "bcache" or "flashcache".
>>> Since I have already plan to use SSD for my journal, I would certainly
>>> use also SSD as cache drive in addition.
>>>
>>> I will have to read documentation about "bcache" and his integration
>>> with Ceph.
>>>
>>> Martin
>>>
>>> Martin Catudal
>>> Responsable TIC
>>> Ressources Metanor Inc
>>> Ligne directe: (819) 218-2708
>>>
>>> Le 2013-10-07 11:25, Mike Lowe a écrit :
>>> > Based on my experience I think you are grossly underestimating the
>>> expense and frequency of flushes issued from your vm's.  This will be
>>> especially bad if you aren't using the async flush from qemu >= 1.4.2 as
>>> the vm is suspended while qemu waits for the flush to finish.  I think your
>>> best course of action until the caching pool work is completed (I think I
>>> remember correctly that this is currently in development) is to either use
>>> the ssd's as large caches with bcache or to use them for journal devices.
>>>  I'm sure there are some other more informed opinions out there on the best
>>> use of ssd's in a ceph cluster and hopefully they will chime in.
>>> >
>>> > On Oct 6, 2013, at 9:23 PM, Martin Catudal 
>>> wrote:
>>> >
>>> >> Hi Guys,
>>> >>  I read all Ceph documentation more than twice. I'm now very
>>> >> comfortable with all the aspect of Ceph except for the strategy of
>>> using
>>> >> my SSD and HDD.
>>> >>
>>> >> Here is my reflexion
>>> >>
>>> >> I've two approach in my understanding about use fast SSD (900 GB) for
>>> my
>>> >> primary storage and huge but slower HDD (4 TB) for replicas.
>>> >>
>>> >> FIRST APPROACH
>>> >> 1. I can use PG with cache write enable as my primary storage that's
>>> >> goes on my SSD and let replicas goes on my 7200 RPM.
>>> >>   With the cache write enable, I will gain performance for my VM
>>> >> user machine in VDI environment since Ceph client will not have to
>>> wait
>>> >> for the replicas write confirmation on the slower HDD.
>>> >>
>>> >> SECOND APPROACH
>>> >> 2. Use pools hierarchies and let's have one pool for the SSD as
>>> primary
>>> >> and lets the replicas goes to a second pool name platter for HDD
>>> >> replication.
>>> >>  As explain in the Ceph documentation
>>> >>  rule ssd-primary {
>>> >>ruleset 4
>>> >>type replicated
>>> >>min_size 5
>>> >>max_size 10
>>> >>step take ssd
>>> >>step chooseleaf firstn 1 type host
>>> >>step emit
>>> >>step take platter
>>> >>step chooseleaf firstn -1 type host
>>> >>step emit
>>> >>}
>>> >>
>>> >> At this point, I could not figure out what approach could have the
>>> most
>>> >> advantage.
>>> >>
>>> >> Your point of view would definitely help me.
>>> >>
>>> >> Sincerely,
>>> >> Martin
>>> >>
>>> >> --
>>> >> Martin Catudal
>>> >> Responsable TIC
>>> >> Ressources Metanor Inc
>>> >> Ligne directe: (819) 218-2708
>>> >> ___
>>> >> ceph-users mailing list
>>> >> ceph-users@lists.ceph.com
>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>>
>> --
>> --
>> *Jason Villalta*
>> Co-founder
>> [image: Inline image 1]
>> 800.799.4407x1230 | www.RubixTechnology.com
>>
>
>
>
> --
> --
> *Jason Villalta*
> Co-founder
> [image: Inline image 1]
> 800.799.4407x1230 | www.RubixTechnology.com
>



-- 
-- 
*Jason Villalta*
Co-founder
[image: Inline image 1]
800.799.4407x1230 | www.RubixTechnology.com
<>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data loss after force umount !

[higkoohk]

Thanks everyone, the env like this :

Linux 3.0.97-1.el6.elrepo.x86_64 CentOS 6.4

ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)

/dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw)
/dev/sdb1 on /var/lib/ceph/osd/ceph-3 type xfs (rw)
/dev/sdc1 on /var/lib/ceph/osd/ceph-4 type xfs (rw)

meta-data=/dev/sdb1  isize=2048   agcount=4, agsize=8895321 blks
 =   sectsz=512   attr=2, projid32bit=0
data =   bsize=4096   blocks=35581281, imaxpct=25
 =   sunit=0  swidth=0 blks
naming   =version 2  bsize=4096   ascii-ci=0
log  =internal   bsize=4096   blocks=17373, version=2
 =   sectsz=512   sunit=0 blks, lazy-count=1
realtime =none   extsz=4096   blocks=0, rtextents=0

/usr/libexec/qemu-kvm -name centos6-clone2 -S -machine rhel6.4.0,accel=kvm
-m 1000 -smp 2,sockets=2,cores=1,threads=1 -uuid
dd1a7093-bdea-4816-8a62-df61cb0c9bfa -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos6-clone2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:rbd/centos6-clone2:auth_supported=none:mon_host=agent21.kisops.org\:6789,if=none,id=drive-virtio-disk0
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5c:71:f1,bus=pci.0,addr=0x3
-vnc 0.0.0.0:0 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Use `umount -l` to force umount.

Anything else ?


2013/10/8 Mark Nelson 

> Also, mkfs, mount, and kvm disk options?
>
> Mark
>
>
> On 10/07/2013 03:15 PM, Samuel Just wrote:
>
>> Sounds like it's probably an issue with the fs on the rbd disk?  What
>> fs was the vm using on the rbd?
>> -Sam
>>
>> On Mon, Oct 7, 2013 at 8:11 AM, higkoohk  wrote:
>>
>>> We use ceph as the storage of kvm .
>>>
>>> I found the VMs errors when force umount the ceph disk.
>>>
>>> Is it just right ? How to repair it ?
>>>
>>> Many thanks .
>>>
>>> --higkoo
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Client Timeout on Rados Gateway

[Gregory Farnum]

On Mon, Oct 7, 2013 at 2:40 PM, Gruher, Joseph R
 wrote:
> Could you clarify something for me... I have a cluster network (10.0.0.x) and 
> a public network (10.23.37.x).  All the Ceph machines have one interface on 
> each network and clients (when configured normally) would only be on the 
> public network.  My ceph.conf uses 10.0.0.x IPs for the monitors but as you 
> mention below this can cause a problem for the client reaching the monitor 
> since it is not on that network.  This could cause the rados command to fail? 
>  What is the solution to that problem?  It doesn't seem like ceph.conf should 
> use the public IPs for the monitor, don't we want those to be on the private 
> network?  And the client wouldn't normally have access to the private 
> network.  Is this really just an issue with accuss using rados, as swift or 
> rbd would not need to access the monitors?

The monitor ports need to be on what Ceph calls the "public" network.
The name is perhaps a little misleading, but the distinction is
between the cluster network — used only by the OSDs, to transfer data
between themselves — and the public network, which is used by
everybody. Monitors in particular are associated with an IP that
cannot change, so doing any kind of dual-broadcast would be finicky,
and they are the access point for every daemon or client which wants
to join the cluster.

Clients which are not using Ceph directly (as in the case of an S3
client, going through the RADOS Gateway) do not need to be on any of
the Ceph networks; they just need access to whatever port their
gateway is serving requests from.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Client Timeout on Rados Gateway

[Gruher, Joseph R]

Could you clarify something for me... I have a cluster network (10.0.0.x) and a 
public network (10.23.37.x).  All the Ceph machines have one interface on each 
network and clients (when configured normally) would only be on the public 
network.  My ceph.conf uses 10.0.0.x IPs for the monitors but as you mention 
below this can cause a problem for the client reaching the monitor since it is 
not on that network.  This could cause the rados command to fail?  What is the 
solution to that problem?  It doesn't seem like ceph.conf should use the public 
IPs for the monitor, don't we want those to be on the private network?  And the 
client wouldn't normally have access to the private network.  Is this really 
just an issue with accuss using rados, as swift or rbd would not need to access 
the monitors?



>-Original Message-
>From: Gregory Farnum [mailto:g...@inktank.com]
>Sent: Monday, October 07, 2013 1:27 PM
>To: Gruher, Joseph R
>Cc: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] Client Timeout on Rados Gateway
>
>The ping tests you're running are connecting to different interfaces
>(10.23.37.175) than those you specify in the "mon_hosts" option (10.0.0.2,
>10.0.0.3, 10.0.0.4). The client needs to be able to connect to the specified
>address; I'm guessing it's not routable from outside that network?
>
>The error you're getting once you put it inside the network is more
>interesting. What version of the Ceph packages do you have installed there,
>and what's installed on the monitors? (run "ceph-mon --version"
>on the monitor, and "rados --version" on the client, and it'll
>output.)
>-Greg
>Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>On Tue, Oct 1, 2013 at 12:45 PM, Gruher, Joseph R
> wrote:
>> Hello-
>>
>>
>>
>> I've set up a rados gateway but I'm having trouble accessing it from
>> clients.  I can access it using rados command line just fine from any
>> system in my ceph deployment, including my monitors and OSDs, the
>> gateway system, and even the admin system I used to run ceph-deploy.
>> However, when I set up a client outside the ceph nodes I get a timeout
>> error as shown at the bottom of the output pasted below.  I've turned
>> off authentication for the moment to simplify things.  Systems are
>> able to resolve names and reach each other via ping.  Any thoughts on what
>could be the issue here or how to debug?
>>
>>
>>
>> The failure:
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ rados df
>>
>> 2013-10-01 19:57:07.488970 7fd381db0780 monclient(hunting):
>> authenticate timed out after 30
>>
>> 2013-10-01 19:57:07.489174 7fd381db0780 librados: client.admin
>> authentication error (110) Connection timed out
>>
>> couldn't connect to cluster! error -110
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ sudo rados df
>>
>> 2013-10-01 19:57:44.461273 7fb6712d5780 monclient(hunting):
>> authenticate timed out after 30
>>
>> 2013-10-01 19:57:44.461440 7fb6712d5780 librados: client.admin
>> authentication error (110) Connection timed out
>>
>> couldn't connect to cluster! error -110
>>
>> ceph@cephclient01:/etc/ceph$
>>
>>
>>
>>
>>
>> Some details from the client:
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ pwd
>>
>> /etc/ceph
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ ls
>>
>> ceph.client.admin.keyring  ceph.conf  keyring.radosgw.gateway
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ cat ceph.conf
>>
>> [global]
>>
>> fsid = a45e6e54-70ef-4470-91db-2152965deec5
>>
>> mon_initial_members = cephtest02, cephtest03, cephtest04
>>
>> mon_host = 10.0.0.2,10.0.0.3,10.0.0.4
>>
>> osd_journal_size = 1024
>>
>> filestore_xattr_use_omap = true
>>
>> auth_cluster_required = none #cephx
>>
>> auth_service_required = none #cephx
>>
>> auth_client_required = none #cephx
>>
>>
>>
>> [client.radosgw.gateway]
>>
>> host = cephtest06
>>
>> keyring = /etc/ceph/keyring.radosgw.gateway
>>
>> rgw_socket_path = /tmp/radosgw.sock
>>
>> log_file = /var/log/ceph/radosgw.log
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ ping cephtest06
>>
>> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data.
>>
>> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1
>> ttl=64
>> time=0.216 ms
>>
>> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2
>> ttl=64
>> time=0.209 ms
>>
>> ^C
>>
>> --- cephtest06.jf.intel.com ping statistics ---
>>
>> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
>>
>> rtt min/avg/max/mdev = 0.209/0.212/0.216/0.015 ms
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ ping cephtest06.jf.intel.com
>>
>> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data.
>>
>> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1
>> ttl=64
>> time=0.223 ms
>>
>> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2
>> ttl=64
>> time=0.242 ms
>>
>> ^C
>>
>> --- cephtest06.jf.intel.com ping statistics ---
>>
>> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
>>
>> rtt min/avg/max/mdev = 0.223/0.232/0.242/0.017 ms
>

Re: [ceph-users] Client Timeout on Rados Gateway

[Gregory Farnum]

On Mon, Oct 7, 2013 at 1:35 PM, Gruher, Joseph R
 wrote:
> Thanks for the reply.  This eventually resolved itself when I upgraded the 
> client kernel from the Ubuntu Server 12.04.2 default to the 3.6.10 kernel.  
> Not sure if there is a good causal explanation there or if it might be a 
> coincidence.  I did see the kernel recommendations in the docs but I had 
> assumed those just applied to the Ceph machines and not clients - perhaps 
> that is a bad assumption.

The kernel should not matter for clients; no. The only other place I
could find that error string was a result of a version mismatch large
enough that it passed over an incompatible encoding change we hadn't
handled appropriately, so I was thinking that maybe your client was
using a very old repository. Glad to hear it seems to have worked
itself out!
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

>  Anyway, it works now, so I guess the next steps are to try moving the client 
> back to the public network and to re-enable authentication and see if it 
> works or if I still have an issue there.
>
> With regard to versions:
>
> ceph@cephtest06:/etc/ceph$ ceph-mon --version
> ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
>
> ceph@cephtest06:/etc/ceph$ uname -a
> Linux cephtest06 3.6.10-030610-generic #201212101650 SMP Mon Dec 10 21:51:40 
> UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> ceph@cephclient01:~/cos$ rados --version
> ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
>
> ceph@cephclient01:~/cos$ uname -a
> Linux cephclient01 3.6.10-030610-generic #201212101650 SMP Mon Dec 10 
> 21:51:40 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> Thanks,
> Joe
>
>>-Original Message-
>>From: Gregory Farnum [mailto:g...@inktank.com]
>>Sent: Monday, October 07, 2013 1:27 PM
>>To: Gruher, Joseph R
>>Cc: ceph-users@lists.ceph.com
>>Subject: Re: [ceph-users] Client Timeout on Rados Gateway
>>
>>The ping tests you're running are connecting to different interfaces
>>(10.23.37.175) than those you specify in the "mon_hosts" option (10.0.0.2,
>>10.0.0.3, 10.0.0.4). The client needs to be able to connect to the specified
>>address; I'm guessing it's not routable from outside that network?
>>
>>The error you're getting once you put it inside the network is more
>>interesting. What version of the Ceph packages do you have installed there,
>>and what's installed on the monitors? (run "ceph-mon --version"
>>on the monitor, and "rados --version" on the client, and it'll
>>output.)
>>-Greg
>>Software Engineer #42 @ http://inktank.com | http://ceph.com
>>
>>On Tue, Oct 1, 2013 at 12:45 PM, Gruher, Joseph R
>> wrote:
>>> Hello-
>>>
>>>
>>>
>>> I've set up a rados gateway but I'm having trouble accessing it from
>>> clients.  I can access it using rados command line just fine from any
>>> system in my ceph deployment, including my monitors and OSDs, the
>>> gateway system, and even the admin system I used to run ceph-deploy.
>>> However, when I set up a client outside the ceph nodes I get a timeout
>>> error as shown at the bottom of the output pasted below.  I've turned
>>> off authentication for the moment to simplify things.  Systems are
>>> able to resolve names and reach each other via ping.  Any thoughts on what
>>could be the issue here or how to debug?
>>>
>>>
>>>
>>> The failure:
>>>
>>>
>>>
>>> ceph@cephclient01:/etc/ceph$ rados df
>>>
>>> 2013-10-01 19:57:07.488970 7fd381db0780 monclient(hunting):
>>> authenticate timed out after 30
>>>
>>> 2013-10-01 19:57:07.489174 7fd381db0780 librados: client.admin
>>> authentication error (110) Connection timed out
>>>
>>> couldn't connect to cluster! error -110
>>>
>>>
>>>
>>>
>>>
>>> ceph@cephclient01:/etc/ceph$ sudo rados df
>>>
>>> 2013-10-01 19:57:44.461273 7fb6712d5780 monclient(hunting):
>>> authenticate timed out after 30
>>>
>>> 2013-10-01 19:57:44.461440 7fb6712d5780 librados: client.admin
>>> authentication error (110) Connection timed out
>>>
>>> couldn't connect to cluster! error -110
>>>
>>> ceph@cephclient01:/etc/ceph$
>>>
>>>
>>>
>>>
>>>
>>> Some details from the client:
>>>
>>>
>>>
>>> ceph@cephclient01:/etc/ceph$ pwd
>>>
>>> /etc/ceph
>>>
>>>
>>>
>>>
>>>
>>> ceph@cephclient01:/etc/ceph$ ls
>>>
>>> ceph.client.admin.keyring  ceph.conf  keyring.radosgw.gateway
>>>
>>>
>>>
>>>
>>>
>>> ceph@cephclient01:/etc/ceph$ cat ceph.conf
>>>
>>> [global]
>>>
>>> fsid = a45e6e54-70ef-4470-91db-2152965deec5
>>>
>>> mon_initial_members = cephtest02, cephtest03, cephtest04
>>>
>>> mon_host = 10.0.0.2,10.0.0.3,10.0.0.4
>>>
>>> osd_journal_size = 1024
>>>
>>> filestore_xattr_use_omap = true
>>>
>>> auth_cluster_required = none #cephx
>>>
>>> auth_service_required = none #cephx
>>>
>>> auth_client_required = none #cephx
>>>
>>>
>>>
>>> [client.radosgw.gateway]
>>>
>>> host = cephtest06
>>>
>>> keyring = /etc/ceph/keyring.radosgw.gateway
>>>
>>> rgw_socket_path = /tmp/radosgw.sock
>>>
>>> log_file = /var/log/ceph/radosgw.log
>>>
>>>
>>>
>>>
>>>
>>> ceph@cephclient01:/etc/c

Re: [ceph-users] share bucket with other users

[Samuel Just]

You can do this with s3 acls.
-Sam

On Wed, Oct 2, 2013 at 9:32 AM, Jefferson Alcantara
 wrote:
> I need share buckets created by one user with outher users without share the
> same access_key or secret_key , for example I have user jmoura with bucket
> name Jeff and I need share this bucket with user frocha and show the
> informations on bucket Jeff  , a
> anybody know  how I do that ?
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Client Timeout on Rados Gateway

[Gruher, Joseph R]

Thanks for the reply.  This eventually resolved itself when I upgraded the 
client kernel from the Ubuntu Server 12.04.2 default to the 3.6.10 kernel.  Not 
sure if there is a good causal explanation there or if it might be a 
coincidence.  I did see the kernel recommendations in the docs but I had 
assumed those just applied to the Ceph machines and not clients - perhaps that 
is a bad assumption.  Anyway, it works now, so I guess the next steps are to 
try moving the client back to the public network and to re-enable 
authentication and see if it works or if I still have an issue there.

With regard to versions:

ceph@cephtest06:/etc/ceph$ ceph-mon --version
ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)

ceph@cephtest06:/etc/ceph$ uname -a
Linux cephtest06 3.6.10-030610-generic #201212101650 SMP Mon Dec 10 21:51:40 
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

ceph@cephclient01:~/cos$ rados --version
ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)

ceph@cephclient01:~/cos$ uname -a
Linux cephclient01 3.6.10-030610-generic #201212101650 SMP Mon Dec 10 21:51:40 
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Thanks,
Joe

>-Original Message-
>From: Gregory Farnum [mailto:g...@inktank.com]
>Sent: Monday, October 07, 2013 1:27 PM
>To: Gruher, Joseph R
>Cc: ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] Client Timeout on Rados Gateway
>
>The ping tests you're running are connecting to different interfaces
>(10.23.37.175) than those you specify in the "mon_hosts" option (10.0.0.2,
>10.0.0.3, 10.0.0.4). The client needs to be able to connect to the specified
>address; I'm guessing it's not routable from outside that network?
>
>The error you're getting once you put it inside the network is more
>interesting. What version of the Ceph packages do you have installed there,
>and what's installed on the monitors? (run "ceph-mon --version"
>on the monitor, and "rados --version" on the client, and it'll
>output.)
>-Greg
>Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>On Tue, Oct 1, 2013 at 12:45 PM, Gruher, Joseph R
> wrote:
>> Hello-
>>
>>
>>
>> I've set up a rados gateway but I'm having trouble accessing it from
>> clients.  I can access it using rados command line just fine from any
>> system in my ceph deployment, including my monitors and OSDs, the
>> gateway system, and even the admin system I used to run ceph-deploy.
>> However, when I set up a client outside the ceph nodes I get a timeout
>> error as shown at the bottom of the output pasted below.  I've turned
>> off authentication for the moment to simplify things.  Systems are
>> able to resolve names and reach each other via ping.  Any thoughts on what
>could be the issue here or how to debug?
>>
>>
>>
>> The failure:
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ rados df
>>
>> 2013-10-01 19:57:07.488970 7fd381db0780 monclient(hunting):
>> authenticate timed out after 30
>>
>> 2013-10-01 19:57:07.489174 7fd381db0780 librados: client.admin
>> authentication error (110) Connection timed out
>>
>> couldn't connect to cluster! error -110
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ sudo rados df
>>
>> 2013-10-01 19:57:44.461273 7fb6712d5780 monclient(hunting):
>> authenticate timed out after 30
>>
>> 2013-10-01 19:57:44.461440 7fb6712d5780 librados: client.admin
>> authentication error (110) Connection timed out
>>
>> couldn't connect to cluster! error -110
>>
>> ceph@cephclient01:/etc/ceph$
>>
>>
>>
>>
>>
>> Some details from the client:
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ pwd
>>
>> /etc/ceph
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ ls
>>
>> ceph.client.admin.keyring  ceph.conf  keyring.radosgw.gateway
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ cat ceph.conf
>>
>> [global]
>>
>> fsid = a45e6e54-70ef-4470-91db-2152965deec5
>>
>> mon_initial_members = cephtest02, cephtest03, cephtest04
>>
>> mon_host = 10.0.0.2,10.0.0.3,10.0.0.4
>>
>> osd_journal_size = 1024
>>
>> filestore_xattr_use_omap = true
>>
>> auth_cluster_required = none #cephx
>>
>> auth_service_required = none #cephx
>>
>> auth_client_required = none #cephx
>>
>>
>>
>> [client.radosgw.gateway]
>>
>> host = cephtest06
>>
>> keyring = /etc/ceph/keyring.radosgw.gateway
>>
>> rgw_socket_path = /tmp/radosgw.sock
>>
>> log_file = /var/log/ceph/radosgw.log
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ ping cephtest06
>>
>> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data.
>>
>> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1
>> ttl=64
>> time=0.216 ms
>>
>> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2
>> ttl=64
>> time=0.209 ms
>>
>> ^C
>>
>> --- cephtest06.jf.intel.com ping statistics ---
>>
>> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
>>
>> rtt min/avg/max/mdev = 0.209/0.212/0.216/0.015 ms
>>
>>
>>
>>
>>
>> ceph@cephclient01:/etc/ceph$ ping cephtest06.jf.intel.com
>>
>> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data.
>>
>> 64 bytes from cephtest06.jf.in

Re: [ceph-users] Data loss after force umount !

[Mark Nelson]

Also, mkfs, mount, and kvm disk options?

Mark

On 10/07/2013 03:15 PM, Samuel Just wrote:

Sounds like it's probably an issue with the fs on the rbd disk?  What
fs was the vm using on the rbd?
-Sam

On Mon, Oct 7, 2013 at 8:11 AM, higkoohk  wrote:

We use ceph as the storage of kvm .

I found the VMs errors when force umount the ceph disk.

Is it just right ? How to repair it ?

Many thanks .

--higkoo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Client Timeout on Rados Gateway

[Gregory Farnum]

The ping tests you're running are connecting to different interfaces
(10.23.37.175) than those you specify in the "mon_hosts" option
(10.0.0.2, 10.0.0.3, 10.0.0.4). The client needs to be able to connect
to the specified address; I'm guessing it's not routable from outside
that network?

The error you're getting once you put it inside the network is more
interesting. What version of the Ceph packages do you have installed
there, and what's installed on the monitors? (run "ceph-mon --version"
on the monitor, and "rados --version" on the client, and it'll
output.)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

On Tue, Oct 1, 2013 at 12:45 PM, Gruher, Joseph R
 wrote:
> Hello-
>
>
>
> I’ve set up a rados gateway but I’m having trouble accessing it from
> clients.  I can access it using rados command line just fine from any system
> in my ceph deployment, including my monitors and OSDs, the gateway system,
> and even the admin system I used to run ceph-deploy.  However, when I set up
> a client outside the ceph nodes I get a timeout error as shown at the bottom
> of the output pasted below.  I’ve turned off authentication for the moment
> to simplify things.  Systems are able to resolve names and reach each other
> via ping.  Any thoughts on what could be the issue here or how to debug?
>
>
>
> The failure:
>
>
>
> ceph@cephclient01:/etc/ceph$ rados df
>
> 2013-10-01 19:57:07.488970 7fd381db0780 monclient(hunting): authenticate
> timed out after 30
>
> 2013-10-01 19:57:07.489174 7fd381db0780 librados: client.admin
> authentication error (110) Connection timed out
>
> couldn't connect to cluster! error -110
>
>
>
>
>
> ceph@cephclient01:/etc/ceph$ sudo rados df
>
> 2013-10-01 19:57:44.461273 7fb6712d5780 monclient(hunting): authenticate
> timed out after 30
>
> 2013-10-01 19:57:44.461440 7fb6712d5780 librados: client.admin
> authentication error (110) Connection timed out
>
> couldn't connect to cluster! error -110
>
> ceph@cephclient01:/etc/ceph$
>
>
>
>
>
> Some details from the client:
>
>
>
> ceph@cephclient01:/etc/ceph$ pwd
>
> /etc/ceph
>
>
>
>
>
> ceph@cephclient01:/etc/ceph$ ls
>
> ceph.client.admin.keyring  ceph.conf  keyring.radosgw.gateway
>
>
>
>
>
> ceph@cephclient01:/etc/ceph$ cat ceph.conf
>
> [global]
>
> fsid = a45e6e54-70ef-4470-91db-2152965deec5
>
> mon_initial_members = cephtest02, cephtest03, cephtest04
>
> mon_host = 10.0.0.2,10.0.0.3,10.0.0.4
>
> osd_journal_size = 1024
>
> filestore_xattr_use_omap = true
>
> auth_cluster_required = none #cephx
>
> auth_service_required = none #cephx
>
> auth_client_required = none #cephx
>
>
>
> [client.radosgw.gateway]
>
> host = cephtest06
>
> keyring = /etc/ceph/keyring.radosgw.gateway
>
> rgw_socket_path = /tmp/radosgw.sock
>
> log_file = /var/log/ceph/radosgw.log
>
>
>
>
>
> ceph@cephclient01:/etc/ceph$ ping cephtest06
>
> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data.
>
> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1 ttl=64
> time=0.216 ms
>
> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2 ttl=64
> time=0.209 ms
>
> ^C
>
> --- cephtest06.jf.intel.com ping statistics ---
>
> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
>
> rtt min/avg/max/mdev = 0.209/0.212/0.216/0.015 ms
>
>
>
>
>
> ceph@cephclient01:/etc/ceph$ ping cephtest06.jf.intel.com
>
> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data.
>
> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1 ttl=64
> time=0.223 ms
>
> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2 ttl=64
> time=0.242 ms
>
> ^C
>
> --- cephtest06.jf.intel.com ping statistics ---
>
> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
>
> rtt min/avg/max/mdev = 0.223/0.232/0.242/0.017 ms
>
>
>
>
>
> I did try putting the client on the 10.0.0.x network to see if that would
> affect behavior but that just seemed to introduce a new problem:
>
>
>
> ceph@cephclient01:/etc/ceph$ rados df
>
> 2013-10-01 21:37:29.439410 7f60d2a43700 failed to decode message of type 59
> v1: buffer::end_of_buffer
>
> 2013-10-01 21:37:29.439583 7f60d4a47700 monclient: hunting for new mon
>
>
>
> ceph@cephclient01:/etc/ceph$ ceph -m 10.0.0.2 -s
>
> 2013-10-01 21:37:42.341480 7f61eacd5700 monclient: hunting for new mon
>
> 2013-10-01 21:37:45.341024 7f61eacd5700 monclient: hunting for new mon
>
> 2013-10-01 21:37:45.343274 7f61eacd5700 monclient: hunting for new mon
>
>
>
> ceph@cephclient01:/etc/ceph$ ceph health
>
> 2013-10-01 21:39:52.833560 mon <- [health]
>
> 2013-10-01 21:39:52.834671 mon.0 -> 'unparseable JSON health' (-22)
>
> ceph@cephclient01:/etc/ceph$
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data loss after force umount !

[Samuel Just]

Sounds like it's probably an issue with the fs on the rbd disk?  What
fs was the vm using on the rbd?
-Sam

On Mon, Oct 7, 2013 at 8:11 AM, higkoohk  wrote:
> We use ceph as the storage of kvm .
>
> I found the VMs errors when force umount the ceph disk.
>
> Is it just right ? How to repair it ?
>
> Many thanks .
>
> --higkoo
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: FW: ceph-deploy again

[John Wilkins]

The original documentation was written with a script called mkcephfs
in mind. Then, we began including some documentation for Chef and
Crowbar.  We actually only had developer documentation for doing
things manually. We're working on providing manual steps now. While
it's not in the deployment section, there are some docs that may help
you understand what's going on when you deploy manually. First you
have to bootstrap a monitor. So we have some developer documentation
on that here:

http://ceph.com/docs/master/dev/mon-bootstrap/

There are also some docs for manually adding monitors and OSDs to an
existing cluster here:

http://ceph.com/docs/master/rados/operations/add-or-rm-mons/
http://ceph.com/docs/master/rados/operations/add-or-rm-osds/

There are also some documents for creating and modifying CRUSH maps here:

http://ceph.com/docs/master/rados/operations/crush-map/

There's a high level discussion of why we use the maps here:

http://ceph.com/docs/master/architecture/#scalability-and-high-availability

These procedural docs need some updating too, and we should have a
manual deploy section up soon.





John

On Sun, Oct 6, 2013 at 8:45 PM, Harvey Skinner  wrote:
> hi John, thank you for the reply on this.
>
> From further experimentation, it is the ceph-deploy v1.2.6 package for
> Cuttlefish that is broken or incorrectly defined/packaged.  I cleaned
> up the broken packages and tried again multiple times with same
> results.  You can install ceph-deploy from the ceph repo for
> Cuttlefish still, but as mentioned previously it is only v1.0-1 and
> full of known issues.
>
> For fun I reset my repo links for:
> deb http://ceph.com/debian-dumpling/ raring main
> deb http://ceph.com/packages/ceph-extras/debian/ raring main
>
> to check the dumpling version of ceph-deploy ... and it installed
> without problem.  The differece being that the ceph-deploy package in
> the ceph-extras has a specific python-pushy package called
> 'python-pushy_0.5.3-1raring.ceph_amd64.deb', apparently put together
> by ceph folks for deployment with ceph-deploy v1.2.6 w/ Dumpling &
> Raring.The problem is the same was not done for Cuttlefish &
> Raring.   So trying to install ceph-deploy for use with Cuttlefish(?)
> or at least on Raring is a waste of time.
>
> So even though I had not planned to, I am going with Dumpling release.
>   But I have ran into additional problems with ceph-deploy v1.2.6 and
> cannot use 'ceph-deploy install' to install ceph on my ceph nodes.
> It tries to install the ceph release key as one of the first steps and
> hangs.   From further experimentation, it is hanging because the
> command to retrieve the release key and add it is missing a 'sudo'
> command in it for the 'apt-key add' part of the command:
>
> Running command: wget -q -O-
> 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' |
> apt-key add –
>
> When you reissue the above command with a 'sudo' preceding the
> 'apt-key add -' it works fine.   So I ended up installing ceph bits on
> all my ceph nodes by hand.
>
> I could now create a 'new' cluster with ceph-deploy and thought I
> would try using a cluster name different than the default of 'ceph' to
> see how that works, I saw an e-mail saying you can have multiple
> clusters on same h/w and wanted to try it.  But it seems that
> ceph-deploy doesn't like working with clusters that aren't using the
> defalt cluster name of 'ceph'.  When I tried to add MONs it couldn't
> find 'ceph.conf' because it was named '.conf' and
> errored out.
>
> SO there are still issues with ceph-deploy that cause a lot of
> investigation and workaround time & effort and still have to accept
> its limitations.   The bad part of this is all the current online
> documentation is being changed to reference use of ceph-deploy only
> and any documentation of how to do tasks without ceph-deploy (to get
> around ceph-deploy limitations) is missing.
>
> Harvey
>
> On Thu, Oct 3, 2013 at 10:17 AM, John Wilkins  
> wrote:
>> I've run into this before too. I think with broken packages, you have to
>> uninstall the previous version and do apt-get autoremove as well. Sometimes
>> you have to manually uninstall whatever it lists as the broken packages and
>> then do autoremove. Then, reinstall.
>>
>>
>> On Thu, Oct 3, 2013 at 12:20 AM, Harvey Skinner  wrote:
>>>
>>> I am also having problems getting the latest version of ceph-deploy to
>>> install on Raring.
>>> I was able to install the updated ceph-deploy about two months ago for
>>> Ubuntu 12.04 and Cuttlefish using the following two lines in the
>>> '/etc/apt/sources.list.d/ceph.list'  apt sources file on my 'Admin' node:
>>>
>>>deb http://ceph.com/debian-cuttlefish/ precise main
>>>deb http://ceph.com/packages/ceph-extras/debian/ precise main
>>>
>>> This worked to download the latest version of ceph-deploy, I believe it
>>> was 1.2-0 or something like that.
>>>
>>>  I have now re-imaged my servers to Ubuntu 13.04 and going through th

Re: [ceph-users] FW: Install question

[Abhay Sachan]

Hi Alistair,
You can download the dumpling release rpms from this location: http://ceph
.com/rpm-dumpling/rhel6/x86_64/

And cuttlefish from here: http://ceph.com/rpm-cuttlefish/rhel6/x86_64/

You can download ceph-deploy from here: http://ceph.com/rpm-dumpling/rhel6/
noarch/

But from my personal experience, cuttlefish didn't work with RHEL 6.4, at
least for me.

For all other dependencies(there are many :|), you need to download them
manually(i used pkgs.org) and install them.

Let me know if you face some other problems in installation.

Thanks,
Abhay


On Mon, Oct 7, 2013 at 9:35 PM,  wrote:

> ** **
>
> Hi,
>
> ** **
>
> I am trying to install Ceph on a Red Hat Linux server that does not have
> external access through which it can access the URL’s and download the
> files needed.   The documentation is not clear (to me) on how to install
> the software under these circumstances.
>
> ** **
>
> Should I be downloading the source files and creating my own installable
> packages, or is there another easier way to do it?
>
> ** **
>
> Thanks
>
> Alistair
>
> ___
>
> This message is for information purposes only, it is not a recommendation,
> advice, offer or solicitation to buy or sell a product or service nor an
> official confirmation of any transaction. It is directed at persons who are
> professionals and is not intended for retail customer use. Intended for
> recipient only. This message is subject to the terms at:
> www.barclays.com/emaildisclaimer.
>
> For important disclosures, please see:
> www.barclays.com/salesandtradingdisclaimer regarding market commentary
> from Barclays Sales and/or Trading, who are active market participants; and
> in respect of Barclays Research, including disclosures relating to specific
> issuers, please see http://publicresearch.barclays.com.
>
> ___
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph with high disk densities?

[Mark Nelson]

On 10/07/2013 12:29 PM, Gruher, Joseph R wrote:



In our small test deployments (160 HDs and OSDs across 20 machines)
our performance is quickly bounded by CPU and memory overhead. These
are 2U machines with 2x 6-core Nehalem; and running 8 OSDs consumed
25% of the total CPU time. This was a cuttlefish deployment.


You might be interested in trying a more recent release.  We've implemented
the SSE4 CRC32c instruction for CPUs that support it, which dramatically
reduces CPU overhead during large sequential writes.  On a 4U box with 24
spinning disks and 8 SSDs (4 bay unused) this brought CPU usage down from
something like 80% to around 40% during large sequential writes if I'm
remembering correctly.  The choice of the underlying filesystem will also
affect CPU overhead.  BTRFS tends to be a bit more CPU intensive than say
EXT4.



Mark, what type of CPUs are in the 4U box (number of sockets, architecture, 
core count, frequency)?



That machine has 2 E5-2630L CPUs.  6 cores each at 2GHz.  Throughput was 
around 2.2GB/s from the client (so about 4.4GB/s going to all of the 
disks and SSDs when you count journal writes).


Mark
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph with high disk densities?

[Martin B Nielsen]

Hi Scott,

Just some observations from here.

We run 8 nodes, 2U units with 12x OSD each (4x 500GB ssd, 8x 4TB platter)
attached to 2x LSI 2308 cards. Each node uses an intel E5-2620 with 32G mem.

Granted, we only have like 25 VM (some fairly io-hungry, both iops and
throughput-wise though) on that cluster, but we hardly see any cpu-usage at
all. We have ~6k PG and according to munin our avg. cpu time is ~9% (that
is out of all cores, so 9% out of 1200% (6 cores, 6 HT)).

Sadly I didn't record cpu-usage while stresstesting or breaking it.

We're using cuttlefish and XFS. And again, this cluster is still pretty
underused, so the cpu-usage does not reflect a more active system.

Cheers,
Martin


On Mon, Oct 7, 2013 at 6:15 PM, Scott Devoid  wrote:

> I brought this up within the context of the RAID discussion, but it did
> not garner any responses. [1]
>
> In our small test deployments (160 HDs and OSDs across 20 machines) our
> performance is quickly bounded by CPU and memory overhead. These are 2U
> machines with 2x 6-core Nehalem; and running 8 OSDs consumed 25% of the
> total CPU time. This was a cuttlefish deployment.
>
> This seems like a rather high CPU overhead. Particularly when we are
> looking to hit density target of 10-15 4TB drives / U within 1.5 years.
> Does anyone have suggestions for hitting this requirement? Are there ways
> to reduce CPU and memory overhead per OSD?
>
> My one suggestion was to do some form of RAID to join multiple drives and
> present them to a single OSD. A 2 drive RAID-0 would halve the OSD overhead
> while doubling the failure rate and doubling the rebalance overhead. It is
> not clear to me if that is better or not.
>
> [1]
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-October/004833.html
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph with high disk densities?

[Gruher, Joseph R]

>> In our small test deployments (160 HDs and OSDs across 20 machines)
>> our performance is quickly bounded by CPU and memory overhead. These
>> are 2U machines with 2x 6-core Nehalem; and running 8 OSDs consumed
>> 25% of the total CPU time. This was a cuttlefish deployment.
>
>You might be interested in trying a more recent release.  We've implemented
>the SSE4 CRC32c instruction for CPUs that support it, which dramatically
>reduces CPU overhead during large sequential writes.  On a 4U box with 24
>spinning disks and 8 SSDs (4 bay unused) this brought CPU usage down from
>something like 80% to around 40% during large sequential writes if I'm
>remembering correctly.  The choice of the underlying filesystem will also
>affect CPU overhead.  BTRFS tends to be a bit more CPU intensive than say
>EXT4.
>

Mark, what type of CPUs are in the 4U box (number of sockets, architecture, 
core count, frequency)?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Deploy CEPH Cluster with SAN

[Mark Nelson]

On 10/07/2013 09:36 AM, Mr.Salvatore Rapisarda wrote:

Hi, i started to follow a project with OpenStack and I have some
questions about the storage and specifically the use of a storage SAN.
Actualy I have two SAN without cluster and replication feature. Can i
integrate the SAN with Ceph give these replication features / clusters
to storage (SAN)?


Hi Salvo,

Sort of... You might be able to deploy Ceph on top of the SAN, but it's 
certainly not really an ideal setup.  Presumably you'd want to create a 
crushmap such that your PGs always span SANs so you have some redundancy 
across them (assuming you care about that).  Not sure how the 
performance would be or the right way to export LUNs for OSDs though. 
It probably depends on the hardware being used and the SAN topology.


Sounds like kind of a fun experiment though.  Good luck! :)

Mark



Thanks

Salvo.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph with high disk densities?

[Corin Langosch]

Am 07.10.2013 18:23, schrieb Gregory Farnum:

There are a few tradeoffs you can make to reduce memory usage (I
believe the big one is maintaining a shorter PG log, which lets nodes
catch up without going through a full backfill), and there is also a
I wonder why this log has to be fully kept in memory and not stored on disk? Is 
there an easy way to calculate how long an OSD can be down with a given PG log 
size and iops? See http://tracker.ceph.com/issues/570 



Corin

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph with high disk densities?

[Mark Nelson]

Hi Scott,

On 10/07/2013 11:15 AM, Scott Devoid wrote:

I brought this up within the context of the RAID discussion, but it did
not garner any responses. [1]

In our small test deployments (160 HDs and OSDs across 20 machines) our
performance is quickly bounded by CPU and memory overhead. These are 2U
machines with 2x 6-core Nehalem; and running 8 OSDs consumed 25% of the
total CPU time. This was a cuttlefish deployment.


You might be interested in trying a more recent release.  We've 
implemented the SSE4 CRC32c instruction for CPUs that support it, which 
dramatically reduces CPU overhead during large sequential writes.  On a 
4U box with 24 spinning disks and 8 SSDs (4 bay unused) this brought CPU 
usage down from something like 80% to around 40% during large sequential 
writes if I'm remembering correctly.  The choice of the underlying 
filesystem will also affect CPU overhead.  BTRFS tends to be a bit more 
CPU intensive than say EXT4.




This seems like a rather high CPU overhead. Particularly when we are
looking to hit density target of 10-15 4TB drives / U within 1.5 years.
Does anyone have suggestions for hitting this requirement? Are there
ways to reduce CPU and memory overhead per OSD?


If nothing else, you can turn off crc32 calculations for the messenger 
in ceph.conf and on the client as a mount parameter if you are using 
cephfs.  That will help.  For small IO, we just started some work to 
look at whether or not we can reduce the amount of memory copying 
happening inside the OSDs which could potentially help here too, 
especially on ARM or other low power platforms.




My one suggestion was to do some form of RAID to join multiple drives
and present them to a single OSD. A 2 drive RAID-0 would halve the OSD
overhead while doubling the failure rate and doubling the rebalance
overhead. It is not clear to me if that is better or not.


If you have 60+ drives per node perhaps.  It kind of depends on how much 
throughput you can push over your network and what your disks and 
controllers are capable of.  Ceph seems to push controllers very hard, 
sometimes with both small random reads/writes and large sequential 
writes concurrently.  The fastest nodes we've tested have multiple 
controllers and skip expander backplanes entirely.


I suspect that in the future, the best platforms for Ceph on spinning 
disks will be extremely dense chassis' that house multiple nodes that 
each have a single CPU, a limited number of OSD disks per node (on a 
dedicated controller and no expander), and possibly some 2.5" bays for 
journals and system disks on an alternate controller.  10GbE would be 
enough to get reasonable performance out a node like this.  With faster 
storage or larger nodes, 40GbE or QDR+ IB might be more attractive.




[1]
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-October/004833.html


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph with high disk densities?

[Gregory Farnum]

On Mon, Oct 7, 2013 at 9:15 AM, Scott Devoid  wrote:
> I brought this up within the context of the RAID discussion, but it did not
> garner any responses. [1]
>
> In our small test deployments (160 HDs and OSDs across 20 machines) our
> performance is quickly bounded by CPU and memory overhead. These are 2U
> machines with 2x 6-core Nehalem; and running 8 OSDs consumed 25% of the
> total CPU time. This was a cuttlefish deployment.

That sounds about right. One of Ceph's design goals was to use the CPU
power which is generally available in storage boxes to make your
storage better — it is not targeted as a low-power way to aggregate
your spare compute server drives.
That said, we are pretty much always on the lookout for ways to reduce
CPU requirements so you may see this go down a respectable amount in
the future.

> This seems like a rather high CPU overhead. Particularly when we are looking
> to hit density target of 10-15 4TB drives / U within 1.5 years. Does anyone
> have suggestions for hitting this requirement? Are there ways to reduce CPU
> and memory overhead per OSD?

There are a few tradeoffs you can make to reduce memory usage (I
believe the big one is maintaining a shorter PG log, which lets nodes
catch up without going through a full backfill), and there is also a
relationship between cpu/memory usage and PG count — but of course the
cost of reducing PGs is having less even storage distributions.

> My one suggestion was to do some form of RAID to join multiple drives and
> present them to a single OSD. A 2 drive RAID-0 would halve the OSD overhead
> while doubling the failure rate and doubling the rebalance overhead. It is
> not clear to me if that is better or not.

I expect that some form of RAID will be necessary on the hyper-dense
systems that vendors are starting to come up with, yes. Nobody has
enough experience with a running system yet to know if that's a good
tradeoff to make.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph with high disk densities?

[Scott Devoid]

I brought this up within the context of the RAID discussion, but it did not
garner any responses. [1]

In our small test deployments (160 HDs and OSDs across 20 machines) our
performance is quickly bounded by CPU and memory overhead. These are 2U
machines with 2x 6-core Nehalem; and running 8 OSDs consumed 25% of the
total CPU time. This was a cuttlefish deployment.

This seems like a rather high CPU overhead. Particularly when we are
looking to hit density target of 10-15 4TB drives / U within 1.5 years.
Does anyone have suggestions for hitting this requirement? Are there ways
to reduce CPU and memory overhead per OSD?

My one suggestion was to do some form of RAID to join multiple drives and
present them to a single OSD. A 2 drive RAID-0 would halve the OSD overhead
while doubling the failure rate and doubling the rebalance overhead. It is
not clear to me if that is better or not.

[1]
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-October/004833.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] FW: Install question

[alistair.whittle]

Hi,

I am trying to install Ceph on a Red Hat Linux server that does not have 
external access through which it can access the URL's and download the files 
needed.   The documentation is not clear (to me) on how to install the software 
under these circumstances.

Should I be downloading the source files and creating my own installable 
packages, or is there another easier way to do it?

Thanks
Alistair

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] About Ceph SSD and HDD strategy

[Jason Villalta]

I found this without much effort.
http://www.sebastien-han.fr/blog/2012/11/15/make-your-rbd-fly-with-flashcache/


On Mon, Oct 7, 2013 at 11:39 AM, Jason Villalta  wrote:

> I also would be interested in how bcache or flashcache would integrate.
>
>
> On Mon, Oct 7, 2013 at 11:34 AM, Martin Catudal wrote:
>
>> Thank's Mike,
>>  Kyle Bader suggest me also to use my large SSD (900 GB) as cache
>> drive using "bcache" or "flashcache".
>> Since I have already plan to use SSD for my journal, I would certainly
>> use also SSD as cache drive in addition.
>>
>> I will have to read documentation about "bcache" and his integration
>> with Ceph.
>>
>> Martin
>>
>> Martin Catudal
>> Responsable TIC
>> Ressources Metanor Inc
>> Ligne directe: (819) 218-2708
>>
>> Le 2013-10-07 11:25, Mike Lowe a écrit :
>> > Based on my experience I think you are grossly underestimating the
>> expense and frequency of flushes issued from your vm's.  This will be
>> especially bad if you aren't using the async flush from qemu >= 1.4.2 as
>> the vm is suspended while qemu waits for the flush to finish.  I think your
>> best course of action until the caching pool work is completed (I think I
>> remember correctly that this is currently in development) is to either use
>> the ssd's as large caches with bcache or to use them for journal devices.
>>  I'm sure there are some other more informed opinions out there on the best
>> use of ssd's in a ceph cluster and hopefully they will chime in.
>> >
>> > On Oct 6, 2013, at 9:23 PM, Martin Catudal  wrote:
>> >
>> >> Hi Guys,
>> >>  I read all Ceph documentation more than twice. I'm now very
>> >> comfortable with all the aspect of Ceph except for the strategy of
>> using
>> >> my SSD and HDD.
>> >>
>> >> Here is my reflexion
>> >>
>> >> I've two approach in my understanding about use fast SSD (900 GB) for
>> my
>> >> primary storage and huge but slower HDD (4 TB) for replicas.
>> >>
>> >> FIRST APPROACH
>> >> 1. I can use PG with cache write enable as my primary storage that's
>> >> goes on my SSD and let replicas goes on my 7200 RPM.
>> >>   With the cache write enable, I will gain performance for my VM
>> >> user machine in VDI environment since Ceph client will not have to wait
>> >> for the replicas write confirmation on the slower HDD.
>> >>
>> >> SECOND APPROACH
>> >> 2. Use pools hierarchies and let's have one pool for the SSD as primary
>> >> and lets the replicas goes to a second pool name platter for HDD
>> >> replication.
>> >>  As explain in the Ceph documentation
>> >>  rule ssd-primary {
>> >>ruleset 4
>> >>type replicated
>> >>min_size 5
>> >>max_size 10
>> >>step take ssd
>> >>step chooseleaf firstn 1 type host
>> >>step emit
>> >>step take platter
>> >>step chooseleaf firstn -1 type host
>> >>step emit
>> >>}
>> >>
>> >> At this point, I could not figure out what approach could have the most
>> >> advantage.
>> >>
>> >> Your point of view would definitely help me.
>> >>
>> >> Sincerely,
>> >> Martin
>> >>
>> >> --
>> >> Martin Catudal
>> >> Responsable TIC
>> >> Ressources Metanor Inc
>> >> Ligne directe: (819) 218-2708
>> >> ___
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
> --
> --
> *Jason Villalta*
> Co-founder
> [image: Inline image 1]
> 800.799.4407x1230 | www.RubixTechnology.com
>



-- 
-- 
*Jason Villalta*
Co-founder
[image: Inline image 1]
800.799.4407x1230 | www.RubixTechnology.com
<>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] About Ceph SSD and HDD strategy

[Jason Villalta]

I also would be interested in how bcache or flashcache would integrate.


On Mon, Oct 7, 2013 at 11:34 AM, Martin Catudal  wrote:

> Thank's Mike,
>  Kyle Bader suggest me also to use my large SSD (900 GB) as cache
> drive using "bcache" or "flashcache".
> Since I have already plan to use SSD for my journal, I would certainly
> use also SSD as cache drive in addition.
>
> I will have to read documentation about "bcache" and his integration
> with Ceph.
>
> Martin
>
> Martin Catudal
> Responsable TIC
> Ressources Metanor Inc
> Ligne directe: (819) 218-2708
>
> Le 2013-10-07 11:25, Mike Lowe a écrit :
> > Based on my experience I think you are grossly underestimating the
> expense and frequency of flushes issued from your vm's.  This will be
> especially bad if you aren't using the async flush from qemu >= 1.4.2 as
> the vm is suspended while qemu waits for the flush to finish.  I think your
> best course of action until the caching pool work is completed (I think I
> remember correctly that this is currently in development) is to either use
> the ssd's as large caches with bcache or to use them for journal devices.
>  I'm sure there are some other more informed opinions out there on the best
> use of ssd's in a ceph cluster and hopefully they will chime in.
> >
> > On Oct 6, 2013, at 9:23 PM, Martin Catudal  wrote:
> >
> >> Hi Guys,
> >>  I read all Ceph documentation more than twice. I'm now very
> >> comfortable with all the aspect of Ceph except for the strategy of using
> >> my SSD and HDD.
> >>
> >> Here is my reflexion
> >>
> >> I've two approach in my understanding about use fast SSD (900 GB) for my
> >> primary storage and huge but slower HDD (4 TB) for replicas.
> >>
> >> FIRST APPROACH
> >> 1. I can use PG with cache write enable as my primary storage that's
> >> goes on my SSD and let replicas goes on my 7200 RPM.
> >>   With the cache write enable, I will gain performance for my VM
> >> user machine in VDI environment since Ceph client will not have to wait
> >> for the replicas write confirmation on the slower HDD.
> >>
> >> SECOND APPROACH
> >> 2. Use pools hierarchies and let's have one pool for the SSD as primary
> >> and lets the replicas goes to a second pool name platter for HDD
> >> replication.
> >>  As explain in the Ceph documentation
> >>  rule ssd-primary {
> >>ruleset 4
> >>type replicated
> >>min_size 5
> >>max_size 10
> >>step take ssd
> >>step chooseleaf firstn 1 type host
> >>step emit
> >>step take platter
> >>step chooseleaf firstn -1 type host
> >>step emit
> >>}
> >>
> >> At this point, I could not figure out what approach could have the most
> >> advantage.
> >>
> >> Your point of view would definitely help me.
> >>
> >> Sincerely,
> >> Martin
> >>
> >> --
> >> Martin Catudal
> >> Responsable TIC
> >> Ressources Metanor Inc
> >> Ligne directe: (819) 218-2708
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
-- 
*Jason Villalta*
Co-founder
[image: Inline image 1]
800.799.4407x1230 | www.RubixTechnology.com
<>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] About Ceph SSD and HDD strategy

[Martin Catudal]

Thank's Mike,
 Kyle Bader suggest me also to use my large SSD (900 GB) as cache 
drive using "bcache" or "flashcache".
Since I have already plan to use SSD for my journal, I would certainly 
use also SSD as cache drive in addition.

I will have to read documentation about "bcache" and his integration 
with Ceph.

Martin

Martin Catudal
Responsable TIC
Ressources Metanor Inc
Ligne directe: (819) 218-2708

Le 2013-10-07 11:25, Mike Lowe a écrit :
> Based on my experience I think you are grossly underestimating the expense 
> and frequency of flushes issued from your vm's.  This will be especially bad 
> if you aren't using the async flush from qemu >= 1.4.2 as the vm is suspended 
> while qemu waits for the flush to finish.  I think your best course of action 
> until the caching pool work is completed (I think I remember correctly that 
> this is currently in development) is to either use the ssd's as large caches 
> with bcache or to use them for journal devices.  I'm sure there are some 
> other more informed opinions out there on the best use of ssd's in a ceph 
> cluster and hopefully they will chime in.
>
> On Oct 6, 2013, at 9:23 PM, Martin Catudal  wrote:
>
>> Hi Guys,
>>  I read all Ceph documentation more than twice. I'm now very
>> comfortable with all the aspect of Ceph except for the strategy of using
>> my SSD and HDD.
>>
>> Here is my reflexion
>>
>> I've two approach in my understanding about use fast SSD (900 GB) for my
>> primary storage and huge but slower HDD (4 TB) for replicas.
>>
>> FIRST APPROACH
>> 1. I can use PG with cache write enable as my primary storage that's
>> goes on my SSD and let replicas goes on my 7200 RPM.
>>   With the cache write enable, I will gain performance for my VM
>> user machine in VDI environment since Ceph client will not have to wait
>> for the replicas write confirmation on the slower HDD.
>>
>> SECOND APPROACH
>> 2. Use pools hierarchies and let's have one pool for the SSD as primary
>> and lets the replicas goes to a second pool name platter for HDD
>> replication.
>>  As explain in the Ceph documentation
>>  rule ssd-primary {
>>ruleset 4
>>type replicated
>>min_size 5
>>max_size 10
>>step take ssd
>>step chooseleaf firstn 1 type host
>>step emit
>>step take platter
>>step chooseleaf firstn -1 type host
>>step emit
>>}
>>
>> At this point, I could not figure out what approach could have the most
>> advantage.
>>
>> Your point of view would definitely help me.
>>
>> Sincerely,
>> Martin
>>
>> -- 
>> Martin Catudal
>> Responsable TIC
>> Ressources Metanor Inc
>> Ligne directe: (819) 218-2708
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] About Ceph SSD and HDD strategy

[Mike Lowe]

Based on my experience I think you are grossly underestimating the expense and 
frequency of flushes issued from your vm's.  This will be especially bad if you 
aren't using the async flush from qemu >= 1.4.2 as the vm is suspended while 
qemu waits for the flush to finish.  I think your best course of action until 
the caching pool work is completed (I think I remember correctly that this is 
currently in development) is to either use the ssd's as large caches with 
bcache or to use them for journal devices.  I'm sure there are some other more 
informed opinions out there on the best use of ssd's in a ceph cluster and 
hopefully they will chime in.

On Oct 6, 2013, at 9:23 PM, Martin Catudal  wrote:

> Hi Guys,
> I read all Ceph documentation more than twice. I'm now very 
> comfortable with all the aspect of Ceph except for the strategy of using 
> my SSD and HDD.
> 
> Here is my reflexion
> 
> I've two approach in my understanding about use fast SSD (900 GB) for my 
> primary storage and huge but slower HDD (4 TB) for replicas.
> 
> FIRST APPROACH
> 1. I can use PG with cache write enable as my primary storage that's 
> goes on my SSD and let replicas goes on my 7200 RPM.
>  With the cache write enable, I will gain performance for my VM 
> user machine in VDI environment since Ceph client will not have to wait 
> for the replicas write confirmation on the slower HDD.
> 
> SECOND APPROACH
> 2. Use pools hierarchies and let's have one pool for the SSD as primary 
> and lets the replicas goes to a second pool name platter for HDD 
> replication.
> As explain in the Ceph documentation
> rule ssd-primary {
>   ruleset 4
>   type replicated
>   min_size 5
>   max_size 10
>   step take ssd
>   step chooseleaf firstn 1 type host
>   step emit
>   step take platter
>   step chooseleaf firstn -1 type host
>   step emit
>   }
> 
> At this point, I could not figure out what approach could have the most 
> advantage.
> 
> Your point of view would definitely help me.
> 
> Sincerely,
> Martin
> 
> -- 
> Martin Catudal
> Responsable TIC
> Ressources Metanor Inc
> Ligne directe: (819) 218-2708
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Data loss after force umount !

[higkoohk]

We use ceph as the storage of kvm .

I found the VMs errors when force umount the ceph disk.

Is it just right ? How to repair it ?

Many thanks .

--higkoo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Deploy CEPH Cluster with SAN

[Mr.Salvatore Rapisarda]

Hi, i started to follow a project with OpenStack and I have some 
questions about the storage and specifically the use of a storage SAN. 
Actualy I have two SAN without cluster and replication feature. Can i 
integrate the SAN with Ceph give these replication features / clusters 
to storage (SAN)?


Thanks

Salvo.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Rados gw upload problems

[Warren Wang]

I forgot to reply, but this did indeed work. Thanks Darren. 

--
Warren

On Oct 4, 2013, at 8:22 AM, Darren Birkett  wrote:

> Hi Warren,
> 
> Try using the ceph specific fastcgi module as detailed here:
> 
> http://ceph.com/docs/next/radosgw/manual-install/
> 
> And see if that helps.
> 
> There was a similar discussion on the list previously:
> 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-March/000360.html
> 
> 
> Thanks
> Darren
> 
> 
> 
> On 4 October 2013 05:12, Warren Wang  wrote:
>> Hi all, I'm having a problem uploading through the Rados GW.  I'm getting 
>> the following error, and searches haven't lead me to a solution.
>> 
>> [Fri Oct 04 04:05:11 2013] [error] [client xxx.xxx.xxx.xxx] chunked 
>> Transfer-Encoding forbidden: /swift/v1/wwang-container/test
>> 
>> FastCGI version:
>> ii  libapache2-mod-fastcgi   2.4.7~0910052141-1  
>>amd64Apache 2 FastCGI module for long-running CGI scripts
>> 
>> Auth works properly through keystone.  Getting hung up on this final part.
>> 
>> Thanks for any help,
>> Warren
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw keystone authtoken caching still not working with 0.67.4

[Darren Birkett]

Hi All,

In our prior tests with 0.67.3, keystone authtoken caching was broken
causing dreadful performance - see
http://www.spinics.net/lists/ceph-users/msg04531.html

We upgraded to release 0.67.4 as we wanted to test the apparent fix to
authtoken caching that was included in the release notes.  Unfortunately,
we still seem to be seeing the same issues - keystone is hit for every
single operation, causing a huge bottleneck in performance.

Aside from upgrading packages and restarting all daemons, are there some
options we are missing from the below to get things to work properly:

:/etc/ceph/ceph.conf
[client.radosgw.gateway]

rgw_keystone_url = http://x.x.x.x:35357
rgw_keystone_admin_token = xxx
rgw_keystone_accepted_roles = admin, Member
rgw_keystone_token_cache_size = 1



Thanks
Darren
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph access using curl

[Snider, Tim]

The –debug command worked as described. Can anyone give me a synposis of how 
the authentication token is generated?
   Token generated: 
AUTH_rgwtk0b007261646f733a7377696674046eff2c9ac6a5041b00545248a7893b900677683adaaca1095128b6edf8fc378d7d49d8"

The first part looks like a header: 'AUTH_rgwtk', (rados gateway token)??  
length == 11  and a prefix == rados:swift
AUTH_rgwtk0b0072 61 64 6f 73 3a 73 77 69 66 74
Length = 0xb  r  a  d  o  s  :  s  w  i  f  t

What is the token body -- How is the token body encoded? It's not a straight 
match of any keys shown by the radowgw-admin command?
046eff2c 9ac6a504 1b00545 248a7893 b9006776 83adaaca 1095128 b6edf8fc 378d7d49 
d8

root@controller21:~/ssbench-0.2.16# radosgw-admin user info --uid=rados
2013-10-07 05:55:34.804639 7ff1c3f6c780  0 WARNING: cannot read region map
{ "user_id": "rados",
  "display_name": "rados",
  "email": "n...@none.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
{ "id": "rados:swift",
  "permissions": "full-control"},
{ "id": "rados:swift1",
  "permissions": "full-control"}],
  "keys": [
{ "user": "rados",
  "access_key": "R5F0D2UCSK3618DJ829A",
  "secret_key": "PJR1rvV2+Xrzlwo+AZZKXextsDl45EaLljzopgjD"}],
  "swift_keys": [
{ "user": "rados:swift",
  "secret_key": "77iJvemrxWvYk47HW7pxsL+eHdA53AtLl2T0OyuG"},
{ "user": "rados:swift1",
  "secret_key": "l9Xlg66JvbNvMmZAj91AeQByEiP8R8sBahCJeqAG"}],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": []}

swift --debug -V 1.0 -A 
http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth -U rados:swift -K 
"77iJvemrxWvYk47HW7pxsL+eHdA53AtLl2T0OyuG"  list

DEBUG:swiftclient:REQ: curl -i 
http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth -X GET
DEBUG:swiftclient:RESP STATUS: 204
DEBUG:swiftclient:REQ: curl -i 
http://ictp-R2C4-Controller21.ict.englab.netapp.com/swift/v1?format=json -X GET 
-H "X-Auth-Token: 
AUTH_rgwtk0b007261646f733a7377696674046eff2c9ac6a5041b00545248a7893b900677683adaaca1095128b6edf8fc378d7d49d8"

DEBUG:swiftclient:RESP STATUS: 200

DEBUG:swiftclient:RESP BODY: 
[{"name":"ssbench_00","count":832,"bytes":85196800},...{"name":"xxx","count":1,"bytes":604}]
DEBUG:swiftclient:REQ: curl -i 
http://ictp-R2C4-Controller21.ict.englab.netapp.com/swift/v1?format=json&marker=xxx
 -X GET -H "X-Auth-Token: 
AUTH_rgwtk0b007261646f733a7377696674046eff2c9ac6a5041b00545248a7893b900677683adaaca1095128b6edf8fc378d7d49d8"
DEBUG:swiftclient:RESP STATUS: 200
DEBUG:swiftclient:RESP BODY: []

AUTH_rgwtk0b007261646f733a7377696674
  r a d o s : s w i f t
046eff2c 9ac6a504 1b00545 248a7893 b9006776 83adaaca 1095128 b6edf8fc 378d7d49 
d8

From: Darren Birkett [mailto:darren.birk...@gmail.com] 
Sent: Saturday, October 05, 2013 1:00 AM
To: Snider, Tim; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph access using curl

Try using passing '--debug' to the swift command. It should output the 
equivalent curl command for you to use.

- Darren
"Snider, Tim"  wrote:
I'm having pilot error with getting the path correct using curl.
Bucket listing using "radosgw-admin bucket list" works as does the swift API.
Can someone point out my (obvious) error?
 
Bucket list works:
root@controller21:/home/ceph/my-cluster#  radosgw-admin bucket list  
2013-10-04 11:28:13.144065 7fe5f26fe780  0 WARNING: cannot read region map
[
    "xxx"]
 
Swift API to Ceph works
    swift -V 1.0 -A 
http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth -U rados:swift -K 
"xxx"  stat
   Account: v1
    Containers: 1
   Objects: 1
 Bytes: 604
  Vary: Accept-Encoding
    Server: Apache/2.2.22 (Ubuntu)
    X-Account-Bytes-Used-Actual: 4096
    Content-Type: text/plain; charset=utf-8
    root@controller21:/home/ceph/my-cluster# swift -V 1.0 -A 
http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth -U rados:swift -K 
"77iJvemrxWvYk47HW7pxsL+eHdA53AtLl2T0OyuG"  list
    xxx
 
However I can’t get the correct http path. Variations of auth/v1.0, auth/v1 … 
don’t work either:
    root@controller21:/home/ceph/my-cluster# curl -v -H 'X-Storage-User: 
rados:swift' -H 'X-Storage-Pass: xxx'  
http://ictp-R2C4-Controller21.ict.englab.netapp.com/auth
    * About to connect() to ictp-R2C4-Controller21.ict.englab.netapp.com 
port 80 (#0)
    *   Trying 10.113.193.189... connected
    > GET /auth HTTP/1.1
    > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 
OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
    > Host: ictp-R2C4-Controller21.ict.englab.netapp.com
    > Accept: */*
    > X-Storage-User: rados:swift
    > X-Storage-Pass: 77iJvemrxWvYk47HW7pxsL+eHdA53AtLl2T0OyuG
    >
    < HTTP/1.1 403 Forbidden
    < Date: Fri, 04 Oct 2013 18:30:01 GMT
    < Server:

[ceph-users] High memory consumption of qemu/librdb with enabled cache

[Иван Миронов]

Hello!

I observe very high memory consumption on client with write-intensive load
with qemu 1.6.0 + librbd 0.67.3.

For benchmarking purposes I'm trying to simultaneously run 15 VMs with 3
GiB of RAM on the one host. Each VM uses RBD image cloned from protected
snapshot of "master image". After boot of each VM, "rpm -ihv" with a bunch
of really large RPMs (~8 GiB of unpacked small files) is automatically
started. Here is part of libvirt's XML of one of these VMs:


  
  



  
  
  


Some time after start I can see unexpected growth of memory consumption of
qemu-kvm processes:

 5565 qemu  20   0 9091m 7.3g  10m S  2.6  7.7   4:41.31 qemu-kvm
 5416 qemu  20   0 8059m 6.4g  10m S 27.8  6.8   4:40.93 qemu-kvm
 5490 qemu  20   0 6723m 5.3g  10m S 26.2  5.6   4:30.51 qemu-kvm
 5591 qemu  20   0 6475m 5.1g  10m S 39.1  5.3   4:35.68 qemu-kvm
 5390 qemu  20   0 6227m 4.9g  10m S  2.0  5.1   4:26.42 qemu-kvm
 5615 qemu  20   0 6203m 4.8g  10m S 27.5  5.1   4:34.56 qemu-kvm
 5692 qemu  20   0 6171m 4.8g  10m S 17.5  5.1   4:28.95 qemu-kvm
 5666 qemu  20   0 6163m 4.8g  10m S  2.0  5.1   4:29.66 qemu-kvm
 5740 qemu  20   0 6139m 4.8g  10m S 23.2  5.1   4:39.22 qemu-kvm
 5716 qemu  20   0 5899m 4.6g  10m S 20.2  4.8   4:30.84 qemu-kvm
 5539 qemu  20   0 5827m 4.5g  10m S  1.7  4.8   4:27.02 qemu-kvm
 5515 qemu  20   0 5651m 4.4g  10m S  4.6  4.7   4:25.20 qemu-kvm
 5640 qemu  20   0 5603m 4.3g  10m S  6.6  4.6   4:28.90 qemu-kvm
 5442 qemu  20   0 5373m 4.1g  10m S  2.3  4.4   4:28.45 qemu-kvm
 5466 qemu  20   0 5387m 4.1g  10m S 41.7  4.3   4:41.00 qemu-kvm

It could grow up further:

 5565 qemu 20 0 22.6g 18g 2772 S 2.6 20.0 6:07.40 qemu-kvm



And then free some part of memory at some point:

 5565 qemu 20 0 8011m 6.0g 2796 S 2.3 6.3 6:23.10 qemu-kvm




I tried to reduce cache size to defaults, as suggested on #ceph (replace
"rbd_cache=1:rbd_cache_max_dirty=134217728:rbd_cache_size=268435456:rbd_cache_max_dirty_age=20"
with just "rbd_cache=1"), but it didn't help much:

15297 qemu 20 0 7747m 6.1g 10m S 1.0 6.4 4:47.26 qemu-kvm

Then I tried to disable cache (remove "cache='writeback'" and change
rbd_cache to 0), and memory consumption became normal:

19590 qemu  20   0 4251m 3.0g  10m S  9.2  3.2   3:33.42 qemu-kvm
19526 qemu  20   0 4251m 3.0g  10m S  8.6  3.1   3:22.01 qemu-kvm
19399 qemu  20   0 4251m 3.0g  10m S  9.6  3.1   3:15.01 qemu-kvm
19612 qemu  20   0 4251m 3.0g  10m S  3.0  3.1   4:12.41 qemu-kvm
19568 qemu  20   0 4251m 3.0g  10m S  3.0  3.1   3:32.04 qemu-kvm
19632 qemu  20   0 4251m 3.0g  10m S  7.3  3.1   3:47.57 qemu-kvm
19419 qemu  20   0 4251m 3.0g  10m S  8.9  3.1   3:20.40 qemu-kvm
19484 qemu  20   0 4251m 3.0g  10m S  7.6  3.1   3:30.56 qemu-kvm
19676 qemu  20   0 4251m 3.0g  10m S  4.0  3.1   3:48.99 qemu-kvm
19654 qemu  20   0 4251m 3.0g  10m S  7.3  3.1   3:49.83 qemu-kvm
19464 qemu  20   0 4251m 3.0g  10m S  8.9  3.1   3:45.45 qemu-kvm
19441 qemu  20   0 4251m 3.0g  10m S  7.3  3.1   3:20.58 qemu-kvm
19377 qemu  20   0 4251m 3.0g  10m S  7.9  3.1   3:16.99 qemu-kvm
19548 qemu  20   0 4251m 3.0g  10m S  9.9  3.1   3:33.59 qemu-kvm
19506 qemu  20   0 4251m 3.0g  10m S  7.6  3.1   3:16.94 qemu-kvm

I also tried to drop all caches inside one of the VMs and see how memory
usage of qemu-kvm will change:

killall -s STOP rpm
sync
echo 3 >/proc/sys/vm/drop_caches

But it didn't made any difference outside of VM (except CPU usage because
of SIGSTOP).

Maybe it's a bug in librbd or qemu/rbd?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com