Re: [ceph-users] mds "laggy"

[Noah Watkins]

You may need to be root to look at the logs in /var/log/ceph. Turning up 
logging is helpful, too. Is the bug reproducible? It'd be great if you could 
get a core dump file for the crashed MDS process.

-Noah

On Apr 24, 2013, at 9:53 PM, Varun Chandramouli  wrote:

> Ceph version was a 0.58 build i cloned from github master branch 
> (0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c), The mds daemon 
> had crashed when it became laggy, I restarted it and the MR code continued to 
> execute. I am unable to see any mds logs in /var/log though. Should I be 
> enabling logging or something?
> 
> 
> Regards
> Varun
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds "laggy"

[Varun Chandramouli]

Ceph version was a 0.58 build i cloned from github master branch
(0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c), The mds
daemon had crashed when it became laggy, I restarted it and the MR code
continued to execute. I am unable to see any mds logs in /var/log though.
Should I be enabling logging or something?


Regards
Varun
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph error: active+clean+scrubbing+deep

[MinhTien MinhTien]

Hi David Zafman

I use Ceph 0.56.4

I don't use command "ceph osd set nodeep-scrub"

This command only use in version 0.60 ?

Thanks.

TienBm
Skype: tien.bm0805

On Thu, Apr 25, 2013 at 8:12 AM, David Zafman wrote:

>
> I'm not sure what the point of running with replication set to 1, but a
> new feature adds ceph commands to turn off scrubbing:
>
> Check ceph --help to see if you have a version that has this.
>
>   ceph osd set 
>   ceph osd unset 
>
> You might want to turn off both kinds of scrubbing.
>
> ceph osd set noscrub
> ceph osd set nodeep-scrub
>
>
> David Zafman
> Senior Developer
> http://www.inktank.com
>
> On Apr 16, 2013, at 12:30 AM, kakito  wrote:
>
> > Hi Martin B Nielsen,
> >
> > Thank you for your quick answer :)
> >
> > I am running with replication set to 1. Because my server used RAID 6,
> > divided into 4 partitons, earch partiton is 1 OSD, format ext4. I have 2
> > server ==> 8 OSD.
> >
> > Do you have any advice? ^^
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Bui Minh Tien
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Journal Information

[Mike Dawson]

Mandell,

Not sure if you can start with a partition to see which OSD it belongs 
to, but you can start with the OSDs to see what journal partition 
belongs to it:


ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep 
osd_journal | grep -v size


- Mike

On 4/24/2013 9:05 PM, Mandell Degerness wrote:

Given a partition, is there a command which can be run to validate if
the partition is used as a journal of an OSD and, if so, what OSD it
belongs to?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph error: active+clean+scrubbing+deep

[David Zafman]

I'm not sure what the point of running with replication set to 1, but a new 
feature adds ceph commands to turn off scrubbing:

Check ceph --help to see if you have a version that has this.

  ceph osd set 
  ceph osd unset 

You might want to turn off both kinds of scrubbing.

ceph osd set noscrub
ceph osd set nodeep-scrub


David Zafman
Senior Developer
http://www.inktank.com

On Apr 16, 2013, at 12:30 AM, kakito  wrote:

> Hi Martin B Nielsen,
> 
> Thank you for your quick answer :)
> 
> I am running with replication set to 1. Because my server used RAID 6, 
> divided into 4 partitons, earch partiton is 1 OSD, format ext4. I have 2 
> server ==> 8 OSD.
> 
> Do you have any advice? ^^
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Journal Information

[Mandell Degerness]

Given a partition, is there a command which can be run to validate if
the partition is used as a journal of an OSD and, if so, what OSD it
belongs to?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Libceph - socket error on read / write.

[MinhTien MinhTien]

Dear all

I use ceph 0.56.4 - CentOS 6.3 up kernel 3.8.8-1.el6.elrepo.x86_64.

I use multiple versions of the kernel, but I still encountering status:

libceph: osd3 172.30.33.2:6810 socket closed (con state OPEN)
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: osd3 172.30.33.2:6810 socket error on write
libceph: mon0 172.30.33.2:6789 socket closed (con state OPEN)
libceph: mon0 172.30.33.2:6789 session lost, hunting for new mon
libceph: mon0 172.30.33.2:6789 socket error on write
libceph: mds0 172.30.33.5:6800 socket closed (con state OPEN)
libceph: wrong peer, want 172.30.33.5:6800/6609, got 172.30.33.5:6800/
libceph: mds0 172.30.33.5:6800 socket error on read
libceph: wrong peer, want 172.30.33.5:6800/6609, got 172.30.33.5:6800/
libceph: mds0 172.30.33.5:6800 socket error on read
libceph: wrong peer, want 172.30.33.5:6800/6609, got 172.30.33.5:6800/
libceph: mds0 172.30.33.5:6800 socket error on read
libceph: wrong peer, want 172.30.33.5:6800/6609, got 172.30.33.5:6800/
libceph: mds0 172.30.33.5:6800 socket error on read


This has led to:

2013-04-25 07:42:50.309221 osd.3 [WRN] 16 slow requests, 5 included below;
oldest blocked for > 112.518922 secs
2013-04-25 07:42:50.309226 osd.3 [WRN] slow request 81.114924 seconds old,
received at 2013-04-25 07:41:29.194206: osd_op(client.16426.1:1342
1253435.0515 [write 0~4194304] 0.a3a0bc5c snapc 1=[]) currently
started
2013-04-25 07:42:50.309229 osd.3 [WRN] slow request 81.107046 seconds old,
received at 2013-04-25 07:41:29.202084: osd_op(client.16426.1:1349
1253435.051c [write 0~4194304 [1@-1]] 0.79e1bf44 snapc 1=[])
currently no flag points reached
2013-04-25 07:42:50.309234 osd.3 [WRN] slow request 81.099973 seconds old,
received at 2013-04-25 07:41:29.209157: osd_op(client.16426.1:1358
1253435.0525 [write 0~4194304] 0.d758b955 snapc 1=[]) currently
started
2013-04-25 07:42:50.309237 osd.3 [WRN] slow request 81.093918 seconds old,
received at 2013-04-25 07:41:29.215212: osd_op(client.16426.1:1364
1253435.052b [write 0~4194304] 0.66b34c17 snapc 1=[]) currently
started
2013-04-25 07:42:50.309241 osd.3 [WRN] slow request 81.088057 seconds old,
received at 2013-04-25 07:41:29.221073: osd_op(client.16426.1:1365
1253435.052c [write 0~4194304] 0.a5442446 snapc 1=[]) currently
started
2013-04-25 07:42:56.310042 osd.3 [WRN] 17 slow requests, 1 included below;
oldest blocked for > 118.519799 secs
2013-04-25 07:42:56.310047 osd.3 [WRN] slow request 80.232497 seconds old,
received at 2013-04-25 07:41:36.077510: osd_sub_op(mds.0.92:213 1.77
ea298b77/200.348a/head//1 [] v 1912'15556 snapset=0=[]:[] snapc=0=[])
v7 currently started
2013-04-25 07:42:49.523067 osd.6 [WRN] 2 slow requests, 1 included below;
oldest blocked for > 115.651487 secs
2013-04-25 07:42:49.523072 osd.6 [WRN] slow request 80.650518 seconds old,
received at 2013-04-25 07:41:28.872524: osd_op(mds.0.92:213 200.348a
[write 3153504~1418] 1.ea298b77) v4 currently waiting for subops from [3]
2013-04-25 07:43:16.313000 osd.3 [WRN] 18 slow requests, 1 included below;
oldest blocked for > 138.522756 secs
2013-04-25 07:43:16.313005 osd.3 [WRN] slow request 80.234601 seconds old,
received at 2013-04-25 07:41:56.078363: osd_sub_op(mds.0.92:216 1.77
ea298b77/200.348a/head//1 [] v 1912'15557 snapset=0=[]:[] snapc=0=[])
v7 currently started
2013-04-25 07:43:09.526520 osd.6 [WRN] 3 slow requests, 1 included below;
oldest blocked for > 135.654919 secs
2013-04-25 07:43:09.526525 osd.6 [WRN] slow request 80.653303 seconds old,
received at 2013-04-25 07:41:48.873171: osd_op(mds.0.92:216 200.348a
[write 3154922~1976] 1.ea298b77) v4 currently waiting for subops from [3]
2013-04-25 07:43:38.316232 osd.3 [WRN] 18 slow requests, 5 included below;
oldest blocked for > 160.525944 secs
2013-04-25 07:43:38.316237 osd.3 [WRN] slow request 160.525944 seconds old,
received at 2013-04-25 07:40:57.790208: osd_op(client.16426.1:1281
1253435.04d9 [write 0~4194304] 0.fc1b1d44 snapc 1=[]) currently
started
2013-04-25 07:43:38.316241 osd.3 [WRN] slow request 160.517433 seconds old,
received at 2013-04-25 07:40:57.798719: osd_op(client.16426.1:1282
1253435.04da [write 0~4194304] 0.b7baa9ea snapc 1=[]) currently
commit sent
2013-04-25 07:43:38.316245 osd.3 [WRN] slow request 160.510521 seconds old,
received at 2013-04-25 07:40:57.805631: osd_op(client.16426.1:1287
1253435.04df [write 0~4194304] 0.fe03215c snapc 1=[]) currently
started


How to fix it?

Thanks

-- 
Bui Minh Tien
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.c

[ceph-users] bad crc message in error logs

[James Harper]

I'm seeing a few messages like this on my OSD logfiles:

2013-04-25 00:00:08.174869 e3ca2b70  0 bad crc in data 1652929673 != exp 
2156854821
2013-04-25 00:00:08.179749 e3ca2b70  0 -- 192.168.200.191:6882/30908 >> 
192.168.200.197:0/3338580093 pipe(0xc70e1c0 sd=24 :6882 s=0 pgs=0 cs=0 
l=0).accept peer addr is really 192.168.200.197:0/3338580093 (socket is 
192.168.200.197:37804/0)

And also occasionally:

2013-04-25 01:40:08.006246 e3da8b70  0 auth: could not find secret_id=0
2013-04-25 01:40:08.006275 e3da8b70  0 cephx: verify_authorizer could not get 
service secret for service osd secret_id=0
2013-04-25 01:40:08.006286 e3da8b70  0 -- 192.168.200.190:6872/9757 >> 
192.168.200.197:0/3338580093 pipe(0xb2be540 sd=24 :6872 s=0 pgs=0 cs=0 
l=1).accept: got bad authorizer
2013-04-25 01:40:08.006467 e3da8b70  0 auth: could not find secret_id=0
2013-04-25 01:40:08.006482 e3da8b70  0 cephx: verify_authorizer could not get 
service secret for service osd secret_id=0
2013-04-25 01:40:08.006491 e3da8b70  0 -- 192.168.200.190:6872/9757 >> 
192.168.200.197:0/3338580093 pipe(0xb2be540 sd=24 :6872 s=0 pgs=0 cs=0 
l=1).accept: got bad authorizer
2013-04-25 01:40:08.803858 e3da8b70  0 -- 192.168.200.190:6872/9757 >> 
192.168.200.197:0/3338580093 pipe(0xb2be8c0 sd=24 :6872 s=0 pgs=0 cs=0 
l=0).accept peer addr is really 192.168.200.197:0/3338580093 (socket is 
192.168.200.197:43170/0)

This happens in both my OSD's. When the above is logged the client logs errors 
in the kernel logs like:

Apr 25 07:00:30 bitvs4 kernel: [415607.228653] libceph: osd72 
192.168.200.190:6872 socket closed (con state OPEN)
Apr 25 08:40:06 bitvs4 kernel: [421582.919271] libceph: osd82 
192.168.200.191:6882 socket closed (con state OPEN)
Apr 25 08:40:06 bitvs4 kernel: [421582.920608] libceph: osd82 
192.168.200.191:6882 socket error on read
Apr 25 09:01:07 bitvs4 kernel: [422843.611197] libceph: osd82 
192.168.200.191:6882 socket closed (con state OPEN)
Apr 25 10:00:36 bitvs4 kernel: [426412.718574] libceph: osd82 
192.168.200.191:6882 socket closed (con state OPEN)
Apr 25 10:00:36 bitvs4 kernel: [426412.719766] libceph: osd82 
192.168.200.191:6882 socket error on read

This seems to happen on a 20 minute boundary (eg 1:00, 1:20, or 1:40) but 
mostly the logging is independent in the OSD's - both OSD's tend not to log 
errors at the same time, but always on a 20 minute boundary.

I'm using ceph provided 0.60 packages for Debian.

Is this actually a problem? The client seems to perform just fine although I'm 
still tracking down some performance issues. 'ceph health' says everything is 
okay. The only evidence of a problem is in the logs. I can't tell if 'bad crc' 
refers to data received from the network or data read from disk.

Thanks

James

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs bandwidth issue

[Elso Andras]

Hi
I use the kernel module. I found only one mount parameter for
readahead: rsize. But didnt help.

> These settings are a bit silly. I think what you've got there is
> logically equivalent to having the stripe_unit and object_size both
> set to 512KB, but I'm not certain. 512KB is also a bit small for an FS
> object; why did you change it from the defaults at all? With streaming
> media you're probably better off with larger than smaller chunks...
>
I did not know that transfers are byte, stripe or object based. I changed
back to default sizes, but the incoming bandwidth is still high:
http://elbandi.net/ceph/ceph_bandwidth2.png

I tested some combinations, but all was unsuccessful. Sometimes the
difference was more than 10x (8G incoming, ~700Mbit outgoing).
Can i somehow log the kernel read request size and the osd response size?

Elbandi
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cuttlefish countdown

[Sage Weil]

Hi everyone-

We are down to a handful of urgent bugs (3!) and a cuttlefish release date 
that is less than a week away.  Thank you to everyone who has been 
involved in coding, testing, and stabilizing this release.  We are close!

If you would like to test the current release candidate, your efforts 
would be much appreciated!  For deb systems, you can do

 wget -q -O- 
'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc' | sudo 
apt-key add -
 echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release 
-sc)-x86_64-basic/ref/next $(lsb_release -sc) main | sudo tee 
/etc/apt/sources.list.d/ceph.list

For rpm users you can find packages at

 http://gitbuilder.ceph.com/ceph-rpm-centos6-x86_64-basic/ref/next/
 http://gitbuilder.ceph.com/ceph-rpm-fc17-x86_64-basic/ref/next/
 http://gitbuilder.ceph.com/ceph-rpm-fc18-x86_64-basic/ref/next/

A draft of the release notes is up at

 http://ceph.com/docs/master/release-notes/#v0-61

Let me know if I've missed anything!

sage

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD single process read performance

[Wido den Hollander]

On 04/24/2013 02:23 PM, Mark Nelson wrote:

On 04/24/2013 06:17 AM, Wido den Hollander wrote:

Hi,

I've been working with a Ceph 0.56.4 setup and I've been seeing some RBD
read performance issues with single processes / threads.

The setup is:
- 36 OSDs (2TB WD RE drives)
- 9 hosts (4 per OSD)
- 120GB Intel SSD as a journal per host
- 32GB Ram per host
- Quad Core Xeon CPU (E3-1220 V2 @ 3.10GHz)
- 2Gbit LACP link

The client (3.8.8 kernel) in this case is a single node connected with
20Gbit LACP to the same switches.

To sum it up, with "rados bench" I'm seeing about 918MB/sec read (LACP
doesn't balance well with one client) and 400MB/sec write.

Note: 2 RADOS bench processes with 64 threads each.

While doing those RADOS benches the disks nor the SSDs are really busy,
so it seems that can be tuned a bit further.

The problem is that when using either kernel RBD or librbd the read
speeds are a lot slower then a write in a single process:

dd if=/dev/zero of=/dev/rbd1 bs=4M count=1024: 290MB/sec
dd if=/dev/rbd1 of=/dev/null bs=4M count=1024: 65MB/sec

When running multiple writers I max out at somewhere around 400MB/sec,
the same as RADOS bench was telling me, but the reads go up to 300MB/sec
when running multiple readers.

Running multiple dd instances will still achieve about 60MB/sec per dd,
but it sums up to somewhere around 300MB/sec. (5 readers)

I changed the following settings:

osd op threads = 8
journal aio = true

The AIO journal showed a huge increase in write performance as expected,
but increasing the op threads didn't change that much. Going from 2
(default) to 4 gave me about 5MB/sec and going to 8 added another
3MB/sec.

Since I'm hitting the same RBD image over and over I'd expected these
blocks to be in the cache of that OSDs and have the read speeds reach
near line performance.

The big difference seems to be in the amount of threads. I noticed the
same with RADOS bench. With a smaller number of threads I wouldn't get
to the 918MB/sec and I had to spawn multiple processes to get there.

However, 65MB/sec write per RBD device doesn't seem like a lot.

I also tried with librbd, but that gives a similar read performance as
kernel RBD.

The end-goal is to run with librbd (OpenStack), but for now I just want
to crank up the read performance of a single process.

I found multiple threads regarding the read performance, one showed that
AMD systems where a problem with the hypertransport, but since these are
Intel systems that isn't the case.

Any suggestions? I'm not trying to touch any kernel settings (yet) since
the RADOS bench shows me a pretty high read performance.



Hi Wido,

I did some RBD testing with fio recently. This was 1 client node talking
to 1 server with 24 OSDs over 2 round-robin bonded 10GbE interfaces.  No
replication.  Multiple rados bench instances from the client node tops
out at like ~1.8GB/s writes and ~1.4GB/s reads.  I'm planning on doing a
more complete write up, but for now, here are some of the single volume
fio results.  The big thing here is that concurrency, even with a single
IO process, is needed to get good performance.  With more clients (even
just VMs on the same node), we can get throughput within about 80% of
the RADOS bench numbers.


In my case I'm using 3x replication, so when writing I have 36/3 the 
write performance available.


Issue is that I don't see a 100% util at all. But it's not about 
writing, the reading is just to slow.




4MB write performance using libaio:

1 volume, 1 process, and iodepth = 1

ceph 0.58, krbd:164MB/s
ceph 0.58, qemu/kvm, no cache:84MB/s
ceph 0.58, qemu/kvm, rbd cache:240MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:244MB/s


I tried with wip-bobtail-rbd-backports-req-order and with the recent 
patch for Qemu ( http://patchwork.ozlabs.org/patch/232489/ ) and get 
about 90MB/sec write, but again, it's about reads.




1 volume, 1 process, and iodepth = 16

ceph 0.58, krbd:711MB/s
ceph 0.58, qemu/kvm, no cache:899MB/s
ceph 0.58, qemu/kvm, rbd cache:227MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:680MB/s

4MB read performance using libaio:

1 volume, 1 process, and iodepth = 1

ceph 0.58, krbd:108MB/s
ceph 0.58, qemu/kvm, no cache:85MB/s
ceph 0.58, qemu/kvm, rbd cache:85MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:89MB/s

1 volume, 1 process, and iodepth = 16

ceph 0.58, krbd:516MB/s
ceph 0.58, qemu/kvm, no cache:839MB/s
ceph 0.58, qemu/kvm, rbd cache:823MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:830MB/s


With 4m size and an iodepth of 16 I'm maxing out a 90MB/sec inside a 
Qemu VM.


The whole reading seems sluggish. For example "man fio" took about 4 
seconds to show up. Even running apt-get update is rather slow.


The VM doesn't feel responsive at all, so trying to figure out where 
that comes from.




To get single request 

Re: [ceph-users] "Recommended" cache size on MDS

[Gregory Farnum]

On Wed, Apr 24, 2013 at 8:39 AM, Kevin Decherf  wrote:
> Hey world,
>
> I know that this question is tricky since it depends on the cluster size
> and objects profiles.
>
> For those who are using CephFS, what is your working cache size on your
> cluster? What problems have you encountered with this configuration?
>
> And for InkTank, do you have any recommendation on the ratio cache
> size/total cluster size? Or any better ratio than others observed
> in your labs?

It's not about total cluster size but about the number of active
files, and the size of your directories. :)

That said, the default 100,000 inodes is a very conservative number;
you can probably go at least an order of magnitude higher.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] "Recommended" cache size on MDS

[Kevin Decherf]

Hey world,

I know that this question is tricky since it depends on the cluster size
and objects profiles.

For those who are using CephFS, what is your working cache size on your
cluster? What problems have you encountered with this configuration?

And for InkTank, do you have any recommendation on the ratio cache
size/total cluster size? Or any better ratio than others observed
in your labs?

Thanks,
-- 
Kevin Decherf - @Kdecherf
GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F
http://kdecherf.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds "laggy"

[Noah Watkins]

Varun,

What version of Ceph are you running? Can you confirm that the MDS daemon 
(ceph-mds) is still running or has crashed when the MDS becomes 
laggy/unresponsive? If it has crashed checked the MDS log for a crash report. 
There were a couple Hadoop workloads that caused the MDS to misbehave for us as 
well..

-Noah

On Apr 24, 2013, at 12:56 AM, Varun Chandramouli  wrote:

> Hi All,
> 
> I am running the MapReduce wordcount code (on a ceph cluster consisting of 2 
> VMs) on a data set consisting of 5000 odd files (approx. 10gb size in total). 
> Periodically, the ceph health says that the mds is laggy/unresponsive, and I 
> get messages like the following:
> 
> 13/04/24 10:41:00 INFO mapred.JobClient:  map 11% reduce 3%
> 13/04/24 10:42:36 INFO mapred.JobClient:  map 12% reduce 3%
> 13/04/24 10:42:45 INFO mapred.JobClient:  map 12% reduce 4%
> 13/04/24 10:44:08 INFO mapred.JobClient:  map 13% reduce 4%
> 13/04/24 10:45:29 INFO mapred.JobClient:  map 14% reduce 4%
> 13/04/24 11:06:31 INFO mapred.JobClient: Task Id : 
> attempt_201304241023_0001_m_000706_0, Status : FAILED
> Task attempt_201304241023_0001_m_000706_0 failed to report status for 600 
> seconds. Killing!
> Task attempt_201304241023_0001_m_000706_0 failed to report status for 600 
> seconds. Killing!
> 
> I then have to manually restart the mds again, and the process continues 
> execution. Can someone please tell me the reason for this, and how to solve 
> it? Pasting my ceph.conf file below:
> 
> [global]
> auth client required = none
> auth cluster required = none
> auth service required = none
> 
> [osd]
> osd journal data = 1000
> filestore xattr use omap = true
> #   osd data = /var/lib/ceph/osd/ceph-$id
> 
> [mon.a]
> host = varunc4-virtual-machine
> mon addr = 10.72.148.209:6789
> #   mon data = /var/lib/ceph/mon/ceph-a
> 
> [mds.a]
> host = varunc4-virtual-machine
> #   mds data = /var/lib/ceph/mds/ceph-a
> 
> [osd.0]
> host = varunc4-virtual-machine
> 
> [osd.1]
> host = varunc5-virtual-machine
> 
> Regards
> Varun 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Best solution for shared FS on Ceph for webclusters

[Maik Kulbe]

On 04/24/2013 05:18 AM, Maik Kulbe wrote:
> Hi,
>
> I'm currently looking into several options on how to use ceph in a
small
> to mid size web cluster.
>
> I've ruled out CephFS as it is sadly not stable enaugh.
>
> Then I went with RBD and different approaches. OCFS2 on RBD did the
job
> well but had extrem performance issues when two processes where
> accessing files in the same folder(especially when doing writes).

Any idea if this was more due to OCFS2 or more due to Ceph?  I confess I
don't know much about how OCFS2 works.  Is it doing some kind of latency
sensitive operation when two files are being written per directory?



I can't say for sure but it seemed that the locking mechanisms of OCFS2
locked the whole folder. I tested concurrent writes in one folder and in
several folders and using multiple folders I could reach about 3K Ops
per second, with a single folder those rates dropped to under 2 (single!)
Ops per second, which is kind of strange but I wouldn't think that an rbd
volume could be responsible for that..


>
> At the moment I'm trying a solution that uses RBD with a normal FS
like
> EXT4 or ZFS and where two server export that block device via
NFS(with
> heartbeat for redundancy and failover) but that involves problems
with
> file system consistency.
>
> My question here is, what kind of software stack other users here
would
> suggest for this kind of workload?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Best solution for shared FS on Ceph for web clusters

[Mark Nelson]

On 04/24/2013 05:18 AM, Maik Kulbe wrote:

Hi,

I'm currently looking into several options on how to use ceph in a small
to mid size web cluster.

I've ruled out CephFS as it is sadly not stable enaugh.

Then I went with RBD and different approaches. OCFS2 on RBD did the job
well but had extrem performance issues when two processes where
accessing files in the same folder(especially when doing writes).


Any idea if this was more due to OCFS2 or more due to Ceph?  I confess I 
don't know much about how OCFS2 works.  Is it doing some kind of latency 
sensitive operation when two files are being written per directory?




At the moment I'm trying a solution that uses RBD with a normal FS like
EXT4 or ZFS and where two server export that block device via NFS(with
heartbeat for redundancy and failover) but that involves problems with
file system consistency.

My question here is, what kind of software stack other users here would
suggest for this kind of workload?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD single process read performance

[Mark Nelson]

On 04/24/2013 06:17 AM, Wido den Hollander wrote:

Hi,

I've been working with a Ceph 0.56.4 setup and I've been seeing some RBD
read performance issues with single processes / threads.

The setup is:
- 36 OSDs (2TB WD RE drives)
- 9 hosts (4 per OSD)
- 120GB Intel SSD as a journal per host
- 32GB Ram per host
- Quad Core Xeon CPU (E3-1220 V2 @ 3.10GHz)
- 2Gbit LACP link

The client (3.8.8 kernel) in this case is a single node connected with
20Gbit LACP to the same switches.

To sum it up, with "rados bench" I'm seeing about 918MB/sec read (LACP
doesn't balance well with one client) and 400MB/sec write.

Note: 2 RADOS bench processes with 64 threads each.

While doing those RADOS benches the disks nor the SSDs are really busy,
so it seems that can be tuned a bit further.

The problem is that when using either kernel RBD or librbd the read
speeds are a lot slower then a write in a single process:

dd if=/dev/zero of=/dev/rbd1 bs=4M count=1024: 290MB/sec
dd if=/dev/rbd1 of=/dev/null bs=4M count=1024: 65MB/sec

When running multiple writers I max out at somewhere around 400MB/sec,
the same as RADOS bench was telling me, but the reads go up to 300MB/sec
when running multiple readers.

Running multiple dd instances will still achieve about 60MB/sec per dd,
but it sums up to somewhere around 300MB/sec. (5 readers)

I changed the following settings:

osd op threads = 8
journal aio = true

The AIO journal showed a huge increase in write performance as expected,
but increasing the op threads didn't change that much. Going from 2
(default) to 4 gave me about 5MB/sec and going to 8 added another 3MB/sec.

Since I'm hitting the same RBD image over and over I'd expected these
blocks to be in the cache of that OSDs and have the read speeds reach
near line performance.

The big difference seems to be in the amount of threads. I noticed the
same with RADOS bench. With a smaller number of threads I wouldn't get
to the 918MB/sec and I had to spawn multiple processes to get there.

However, 65MB/sec write per RBD device doesn't seem like a lot.

I also tried with librbd, but that gives a similar read performance as
kernel RBD.

The end-goal is to run with librbd (OpenStack), but for now I just want
to crank up the read performance of a single process.

I found multiple threads regarding the read performance, one showed that
AMD systems where a problem with the hypertransport, but since these are
Intel systems that isn't the case.

Any suggestions? I'm not trying to touch any kernel settings (yet) since
the RADOS bench shows me a pretty high read performance.



Hi Wido,

I did some RBD testing with fio recently. This was 1 client node talking 
to 1 server with 24 OSDs over 2 round-robin bonded 10GbE interfaces.  No 
replication.  Multiple rados bench instances from the client node tops 
out at like ~1.8GB/s writes and ~1.4GB/s reads.  I'm planning on doing a 
more complete write up, but for now, here are some of the single volume 
fio results.  The big thing here is that concurrency, even with a single 
IO process, is needed to get good performance.  With more clients (even 
just VMs on the same node), we can get throughput within about 80% of 
the RADOS bench numbers.


4MB write performance using libaio:

1 volume, 1 process, and iodepth = 1

ceph 0.58, krbd:164MB/s
ceph 0.58, qemu/kvm, no cache:  84MB/s
ceph 0.58, qemu/kvm, rbd cache: 240MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:244MB/s

1 volume, 1 process, and iodepth = 16

ceph 0.58, krbd:711MB/s
ceph 0.58, qemu/kvm, no cache:  899MB/s
ceph 0.58, qemu/kvm, rbd cache: 227MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:680MB/s

4MB read performance using libaio:

1 volume, 1 process, and iodepth = 1

ceph 0.58, krbd:108MB/s
ceph 0.58, qemu/kvm, no cache:  85MB/s
ceph 0.58, qemu/kvm, rbd cache: 85MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:89MB/s

1 volume, 1 process, and iodepth = 16

ceph 0.58, krbd:516MB/s
ceph 0.58, qemu/kvm, no cache:  839MB/s
ceph 0.58, qemu/kvm, rbd cache: 823MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache:830MB/s

To get single request performance to scale farther, you'll have to 
diagnose if there are places that you can lower latency rather than hide 
it with concurrency.  That's not an easy task in a distributed system 
like Ceph.  There are probably opportunities for optimization, but I 
suspect it may take more than tweaking the ceph.conf file.


Mark
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD single process read performance

[Wido den Hollander]

Hi,

I've been working with a Ceph 0.56.4 setup and I've been seeing some RBD 
read performance issues with single processes / threads.


The setup is:
- 36 OSDs (2TB WD RE drives)
- 9 hosts (4 per OSD)
- 120GB Intel SSD as a journal per host
- 32GB Ram per host
- Quad Core Xeon CPU (E3-1220 V2 @ 3.10GHz)
- 2Gbit LACP link

The client (3.8.8 kernel) in this case is a single node connected with 
20Gbit LACP to the same switches.


To sum it up, with "rados bench" I'm seeing about 918MB/sec read (LACP 
doesn't balance well with one client) and 400MB/sec write.


Note: 2 RADOS bench processes with 64 threads each.

While doing those RADOS benches the disks nor the SSDs are really busy, 
so it seems that can be tuned a bit further.


The problem is that when using either kernel RBD or librbd the read 
speeds are a lot slower then a write in a single process:


dd if=/dev/zero of=/dev/rbd1 bs=4M count=1024: 290MB/sec
dd if=/dev/rbd1 of=/dev/null bs=4M count=1024: 65MB/sec

When running multiple writers I max out at somewhere around 400MB/sec, 
the same as RADOS bench was telling me, but the reads go up to 300MB/sec 
when running multiple readers.


Running multiple dd instances will still achieve about 60MB/sec per dd, 
but it sums up to somewhere around 300MB/sec. (5 readers)


I changed the following settings:

osd op threads = 8
journal aio = true

The AIO journal showed a huge increase in write performance as expected, 
but increasing the op threads didn't change that much. Going from 2 
(default) to 4 gave me about 5MB/sec and going to 8 added another 3MB/sec.


Since I'm hitting the same RBD image over and over I'd expected these 
blocks to be in the cache of that OSDs and have the read speeds reach 
near line performance.


The big difference seems to be in the amount of threads. I noticed the 
same with RADOS bench. With a smaller number of threads I wouldn't get 
to the 918MB/sec and I had to spawn multiple processes to get there.


However, 65MB/sec write per RBD device doesn't seem like a lot.

I also tried with librbd, but that gives a similar read performance as 
kernel RBD.


The end-goal is to run with librbd (OpenStack), but for now I just want 
to crank up the read performance of a single process.


I found multiple threads regarding the read performance, one showed that 
AMD systems where a problem with the hypertransport, but since these are 
Intel systems that isn't the case.


Any suggestions? I'm not trying to touch any kernel settings (yet) since 
the RADOS bench shows me a pretty high read performance.


--
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Best solution for shared FS on Ceph for web clusters

[Gandalf Corvotempesta]

2013/4/24 Maik Kulbe :
> At the moment I'm trying a solution that uses RBD with a normal FS like EXT4
> or ZFS and where two server export that block device via NFS(with heartbeat
> for redundancy and failover) but that involves problems with file system
> consistency.

If you don't need load balancing but just a failover NFS on top of RBD
should be ok as long you use only one NFS server per time.
If you do IP failover you will not have any FS consistency issue.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph 0.56.4 - OSD low request

[MinhTien MinhTien]

Dear all

I have 2 server:

earch server:

1 card raid: -- raid 6: 54TB,divided into 4 OSD (format ext4)
 -- raid 0: 248G, journal for 4 OSD (ext4).

My config file:

[global]
auth supported = cephx
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
keyring = /etc/ceph/keyring.admin
[mds]
keyring = /etc/ceph/keyring.$name
debug mds = 1
[mds.0]
host = Ceph-98
[mds.1]
host = Ceph-99

[osd]
osd data = /srv/ceph/osd$id
osd journal size = 106496
osd class dir = /usr/lib/rados-classes
keyring = /etc/ceph/keyring.$name
osd mkfs type = ext4
filestore xattr use omap = true
filestore fiemap = false
osd heartbeat interval = 12
osd heartbeat grace = 35
osd min down reports = 4
osd mon report interval min = 45
osd mon report interval max = 150
osd op complaint time = 80

filestore min sync interval = 1
filestore max sync interval = 30
osd scrub min interval = 120
debug osd = 20
debug ms = 1
debug filestore = 20
[osd.0]
host = Ceph-98
devs = /dev/sdb1
osd journal = /srv/ceph/ssd/journal-0
cluster addr = 172.30.48.98
public addr = 172.30.48.98
[osd.1]
host = Ceph-98
devs = /dev/sdb2
osd journal = /srv/ceph/ssd/journal-1
cluster addr = 172.30.48.98
public addr = 172.30.48.98
[osd.2]
host = Ceph-98
devs = /dev/sdb3
osd journal = /srv/ceph/ssd/journal-2
cluster addr = 172.30.48.98
public addr = 172.30.48.98

.
[osd.7]
host = Ceph-99
devs = /dev/sda4
osd journal = /srv/ceph/ssd/journal-7
cluster addr = 172.30.48.99
public addr = 172.30.48.99
[mon]
mon data = /srv/ceph/mon$id
mon osd down out interval = 3000
[mon.0]
host = Ceph-98
mon addr = 172.30.48.98:6789
[mon.1]
host = Ceph-99
mon addr = 172.30.48.99:6789

I used Centos 6.4 up kernel: 3.8.5-1.el6.elrepo.x86_64
Ceph 0.56.4

I get this error frequently:

2013-04-16 14:02:48.871577 osd.6 [WRN] 9 slow requests, 1 included below;
oldest blocked for > 161.356701 secs
2013-04-16 14:02:48.871581 osd.6 [WRN] slow request 160.410153 seconds old,
received at 2013-04-16 14:00:08.461397: osd_op(mds.0.31:353 200.211b
[write 3203393~2607] 1.90aa1669) v4 currently waiting for subops from [3]
2013-04-16 14:02:49.871761 osd.6 [WRN] 9 slow requests, 2 included below;
oldest blocked for > 162.356878 secs
2013-04-16 14:02:49.871766 osd.6 [WRN] slow request 160.798691 seconds old,
received at 2013-04-16 14:00:09.073036: osd_op(mds.0.31:354 200.211b
[write 3206000~3231] 1.90aa1669) v4 currently waiting for subops from [3]
2013-04-16 14:02:49.871780 osd.6 [WRN] slow request 160.083633 seconds old,
received at 2013-04-16 14:00:09.788094: osd_op(mds.0.31:356 200.211b
[write 3209231~3852] 1.90aa1669) v4 currently waiting for subops from [3]
2013-04-16 14:02:52.872229 osd.6 [WRN] 9 slow requests, 1 included below;
oldest blocked for > 165.357349 secs
2013-04-16 14:02:52.872233 osd.6 [WRN] slow request 160.357224 seconds old,
received at 2013-04-16 14:00:12.514974: osd_op(mds.0.31:357 200.211b
[write 3213083~4503] 1.90aa1669) v4 currently waiting for subops from [3]
2013-04-16 14:02:53.872484 osd.6 [WRN] 9 slow requests, 1 included below;
oldest blocked for > 166.357601 secs
2013-04-16 14:02:53.872489 osd.6 [WRN] slow request 160.099407 seconds old,
received at 2013-04-16 14:00:13.773043: osd_op(mds.0.31:359 200.211b
[write 3217586~4500] 1.90aa1669) v4 currently waiting for subops from [3]
2013-04-16 14:02:57.873113 osd.6 [WRN] 9 slow requests, 1 included below;
oldest blocked for > 170.358236 secs
2013-04-16 14:02:57.873117 osd.6 [WRN] slow request 160.357995 seconds old,
received at 2013-04-16 14:00:17.515090: osd_op(mds.0.31:361 200.211b
[write 3222086~4486] 1.90aa1669) v4 currently waiting for subops from [3]

and:

ceph -w:

2013-04-16 13:45:12.159336 mon.0 [INF] pgmap v280241: 640 pgs: 638
active+clean, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep; 14550
GB data, 14575 GB used, 89841 GB / 107 TB avail; 27B/s wr, 0op/s
2013-04-16 13:45:19.492099 mon.0 [INF] pgmap v280242: 640 pgs: 638
active+clean, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep; 14550
GB data, 14575 GB used, 89841 GB / 107 TB avail

dmesg:


Call Trace:
 [] schedule+0x29/0x70
 [] schedule_preempt_disabled+0xe/0x10
 [] __mutex_lock_slowpath+0xf6/0x170
 [] mutex_lock+0x2b/0x50
 [] ima_rdwr_violation_check+0x79/0x1b0
 [] ima_file_check+0x21/0x60
 [] do_last+0x45d/0x7c0
 [] path_openat+0xb3/0x480
 [] ? __alloc_pages_nodemask+0x2fb/0x320
 [] do_filp_open+0x49/0xa0
 [] ? __alloc_fd+0xdd/0x150
 [] do_sys_open+0x108/0x1f0
 [] sys_open+0x21/0x30
 [] system_call_fastpath+0x16/0x1b
libceph: osd3 172.30.48.98:6810 socket error on write
libceph: osd3 172.30.48.98:6810 socket error on write
libceph: osd6 down
libceph: osd6 up
libceph: osd3 172.30..48.98:6810 socket error on write
libceph: mon0 172.30..

[ceph-users] Best solution for shared FS on Ceph for web clusters

[Maik Kulbe]

Hi,

I'm currently looking into several options on how to use ceph in a small to mid 
size web cluster.

I've ruled out CephFS as it is sadly not stable enaugh.

Then I went with RBD and different approaches. OCFS2 on RBD did the job well 
but had extrem performance issues when two processes where accessing files in 
the same folder(especially when doing writes).

At the moment I'm trying a solution that uses RBD with a normal FS like EXT4 or 
ZFS and where two server export that block device via NFS(with heartbeat for 
redundancy and failover) but that involves problems with file system 
consistency.

My question here is, what kind of software stack other users here would suggest 
for this kind of workload?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd command error "librbd::ImageCtx: error finding header"

[Dennis Chen]

Hi guys,
The "librbd: Error listing snapshots: (95) Operation not supported" 
issue has been resolved, the root cause is when I deployed the OSD, I 
just copy /usr/local/bin/* from MON-MDS node to the OSD, not include the 
/usr/lib/rados-classes. So I re-transfer the /usr/lib/rados-classes 
folder of MON-MDS node to OSD's, it works after this:


root@~# rbd --image odm-kvm-img -p mypool info
rbd image 'odm-kvm-img':
size 10240 MB in 2560 objects
order 22 (4096 KB objects)
block_name_prefix: rb.0.1217.6b8b4567
format: 1

BRs,
Dennis

On 04/24/2013 11:39 AM, Dennis Chen wrote:

Hi,

A different error msg after your suggestion --

root@~# rbd -p mypool --image odm-kvm-img info
rbd: error opening image odm-kvm-img: (95) Operation not supported
2013-04-24 11:32:47.757778 7f49949f7780 -1 librbd: Error listing 
snapshots: (95) Operation not supported


I create the "odm-kvm-img" with this command:
root@~# rbd create odm-kvm-img --size 10240 --pool mypool

BRs,
Dennis

On 04/24/2013 11:28 AM, Chris Hoy Poy wrote:

Hi,

try

# rbd -p mypool --image odm-kvm-img info

cheers
//Chris

- Original Message -
From: "Dennis Chen" 
To: ceph-us...@ceph.com, "Dennis Chen" 
Sent: Wednesday, 24 April, 2013 11:24:30 AM
Subject: [ceph-users] rbd command error "librbd::ImageCtx: error 
findingheader"


Hi list,

I am using a ceph cluster (version 0.56.4) with all nodes (mon, mds,
osd...) deployed in the RHEL 6 distro, the client is based on Ubuntu 
12.10.

Now I am confused by a strange issue, seems the issue has been asked
before by google but no a clear answer for it. The specific details as
below--
in the client side, I want to create a rbd image, so I run the commands:

root@~# ceph osd pool create mypool 100 100
pool 'mypool' created

root@~# rbd ls -p mypool
odm-kvm-img

root@~# rbd --image odm-kvm-img info
rbd: error opening image 2013-04-24 10:43:42.800917 7fdb47d76780 -1
librbd::ImageCtx: error finding header: (2) No such file or
directoryodm-kvm-img:
(2) No such file or directory

So I tried those steps followed according the goolged:

root@~# rados ls -p mypool
odm-kvm-img.rbd
rbd_directory
root@~# rbd info odm-kvm-img.rbd
rbd: error opening image 2013-04-24 10:54:19.468770 7f8332dea780 -1
librbd::ImageCtx: error finding header: (2) No such file or directory
odm-kvm-img.rbd: (2) No such file or directory

odm-kvm-img.rbd is showed by 'rados ls' command and it's there, but why
I get an error when run the 'rbd info' command upon odm-kvm-img.rbd?
does anybody can be help about this?

BRs,
Dennis
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds "laggy"

[Varun Chandramouli]

Hi All,

I am running the MapReduce wordcount code (on a ceph cluster consisting of
2 VMs) on a data set consisting of 5000 odd files (approx. 10gb size in
total). Periodically, the ceph health says that the mds is
laggy/unresponsive, and I get messages like the following:

13/04/24 10:41:00 INFO mapred.JobClient:  map 11% reduce 3%
13/04/24 10:42:36 INFO mapred.JobClient:  map 12% reduce 3%
13/04/24 10:42:45 INFO mapred.JobClient:  map 12% reduce 4%
13/04/24 10:44:08 INFO mapred.JobClient:  map 13% reduce 4%
13/04/24 10:45:29 INFO mapred.JobClient:  map 14% reduce 4%
13/04/24 11:06:31 INFO mapred.JobClient: Task Id :
attempt_201304241023_0001_m_000706_0, Status : FAILED
Task attempt_201304241023_0001_m_000706_0 failed to report status for 600
seconds. Killing!
Task attempt_201304241023_0001_m_000706_0 failed to report status for 600
seconds. Killing!

I then have to manually restart the mds again, and the process continues
execution. Can someone please tell me the reason for this, and how to solve
it? Pasting my ceph.conf file below:

[global]
auth client required = none
auth cluster required = none
auth service required = none

[osd]
osd journal data = 1000
filestore xattr use omap = true
#   osd data = /var/lib/ceph/osd/ceph-$id

[mon.a]
host = varunc4-virtual-machine
mon addr = 10.72.148.209:6789
#   mon data = /var/lib/ceph/mon/ceph-a

[mds.a]
host = varunc4-virtual-machine
#   mds data = /var/lib/ceph/mds/ceph-a

[osd.0]
host = varunc4-virtual-machine

[osd.1]
host = varunc5-virtual-machine

Regards
Varun
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW and S3-compatible clients for PC and OSX

[Igor Laskovy]

Ok. I will try, thanks.
One further question - does needed manually start /etc/init.d/radosgw all
time when this host have been rebooted? Why it is not part of service ceph
-a start?


On Tue, Apr 23, 2013 at 11:05 PM, Lorieri  wrote:

> I've made some tests again with s3cmd
>
> you need to have a valid and accessible host_bucket key in the .s3cfg
> for example:
> host_bucket = %(bucket)s.myhostname.com
>
> if you dont have it, it does not allow you to use lowercase buckets
> I believe it checks if the bucket name is a valid dns name, etc
>
> for osx:
>
> brew install s3cmd
>
> []s
> -lorieri
>
>
>
>
> On Tue, Apr 23, 2013 at 4:00 PM, Igor Laskovy wrote:
>
>> So, I totally lost in this, but I did it, and now CrossFTP report:
>> [R1] Connect to osd01.ceph.labspace.studiogrizzly.com
>> [R1] Current path: /
>> [R1] Current path: /
>> [R1] LIST /
>> [R1] Request Error [
>> 
>> 404 Not Found
>> 
>> Not Found
>> The requested URL / was not found on this server.
>> 
>> ].
>>
>> On Tue, Apr 23, 2013 at 9:39 PM, Yehuda Sadeh  wrote:
>> > On Tue, Apr 23, 2013 at 11:33 AM, Igor Laskovy 
>> wrote:
>> >> In /etc/apache2/httpd.conf I have :
>> >> ServerName osd01.ceph.labspace.studiogrizzly.com
>> >>
>> >> In /etc/apache2/sites-available/rgw.conf :
>> >
>> >
>> >
>> >> FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock
>> >>
>> >> 
>> >> ServerName osd01.ceph.labspace.studiogrizzly.com
>> >> ServerAdmin igor.lask...@gmail.com
>> >> DocumentRoot /var/www
>> >> 
>> >
>> > remove this line ^^^
>> >
>> >>
>> >> RewriteEngine On
>> >> RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
>> >> /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING}
>> >> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>> >>
>> >> 
>> >
>> > remove this line ^^^
>> >
>> >>
>> >> 
>> >> 
>> >> Options +ExecCGI
>> >> AllowOverride All
>> >> SetHandler fastcgi-script
>> >> Order allow,deny
>> >> Allow from all
>> >> AuthBasicAuthoritative Off
>> >> 
>> >> 
>> >>
>> >> AllowEncodedSlashes On
>> >> ErrorLog /var/log/apache2/error.log
>> >> CustomLog /var/log/apache2/access.log combined
>> >> ServerSignature Off
>> >> 
>> >>
>> >> On Tue, Apr 23, 2013 at 5:57 PM, Yehuda Sadeh 
>> wrote:
>> >>> On Tue, Apr 23, 2013 at 7:51 AM, Igor Laskovy 
>> wrote:
>>  Sorry for delayed reply,
>> 
>>  I am not good familiar with apache.
>>  For RGW I use one of the OSD nodes. This is clear minimum
>> installation of
>>  Ubunut 12.04 and ceph deployment on it, no another services.
>>  I must to say that I use default apache2 package from Ubuntu
>> repository and
>>  have rgw print continue = false in ceph.conf .
>>  Again, all configuration made as shown at here
>>  http://ceph.com/docs/master/radosgw/manual-install/ and here
>>  http://ceph.com/docs/master/radosgw/config/ nothing more.
>>  As {fqdn} I have used FQDN for this node.
>> 
>> > try listing whatever under /etc/apache2/sites-enabled, see if
>> there's
>> > anything else there.
>>  Looks like apache works.
>> 
>>  Which exactly log files can I show for you?
>> >>>
>> >>> I think that your apache site config is the more interesting thing to
>> >>> look at it right now. The docs might be a bit unclear, we've seen some
>> >>> error there recently, can you make sure that there's only a single
>> >>> VirtualHost section in it?
>> >>>
>> 
>> 
>>  On Sun, Apr 21, 2013 at 11:49 PM, Yehuda Sadeh 
>> wrote:
>> >
>> > On Sun, Apr 21, 2013 at 10:05 AM, Igor Laskovy <
>> igor.lask...@gmail.com>
>> > wrote:
>> > >
>> > > Just initial connect to rgw server, nothing further.
>> > > Please see below behavior for CrossFTP and S3Browser cases.
>> > >
>> > > On CrossFTP side:
>> > > [R1] Connect to rgw.labspace
>> > > [R1] Current path: /
>> > > [R1] Current path: /
>> > > [R1] LIST /
>> > > [R1] Expected XML document response from S3 but received content
>> type
>> > > text/html
>> > > [R1] Disconnected
>> > >
>> > > On rgw side:
>> > > root@osd01:~# ps aux |grep rados
>> > > root  1785  0.4  0.1 2045404 6068 ?Ssl  19:47   0:00
>> > > /usr/bin/radosgw -n client.radosgw.a
>> > >
>> > > root@osd01:~# tail -f /var/log/apache2/error.log
>> > > [Sun Apr 21 19:43:56 2013] [notice] FastCGI: process manager
>> initialized
>> > > (pid 1433)
>> > > [Sun Apr 21 19:43:56 2013] [notice] Apache/2.2.22 (Ubuntu)
>> > > mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_ssl/2.2.22
>> OpenSSL/1.0.1
>> > > configured -- resuming normal operations
>> > > [Sun Apr 21 19:50:19 2013] [error] [client 192.168.1.51] File
>> does not
>> > > exist: /var/www/favicon.ico
>> >
>> > Doesn't seem that your apache is confi