Re: [ceph-users] cephfs, low performances

[Yan, Zheng]

On Mon, Dec 28, 2015 at 1:24 PM, Francois Lafont  wrote:
> Hi,
>
> Sorry for my late answer.
>
> On 23/12/2015 03:49, Yan, Zheng wrote:
>
 fio tests AIO performance in this case. cephfs does not handle AIO
 properly, AIO is actually SYNC IO. that's why cephfs is so slow in
 this case.
>>>
>>> Ah ok, thanks for this very interesting information.
>>>
>>> So, in fact, the question I ask myself is: how to test my cephfs
>>> to know if I have correct (or not) perfs as regard my hardware
>>> configuration?
>>>
>>> Because currently, in fact, I'm unable to say if I have correct perf
>>> (not incredible but in line with my hardware configuration) or if I
>>> have a problem. ;)
>>>
>>
>> It's hard to tell. basically data IO performance on cephfs should be
>> similar to data IO performance on rbd.
>
> Ok, so in a client node, I have mounted cephfs (via ceph-fuse) and a rados
> block device formatted in XFS. If I have well understood, cephfs uses sync
> IO (not async IO) and, with ceph-fuse, cephfs can't make O_DIRECT IO. So, I
> have tested this fio command in cephfs _and_ in rbd:
>
> fio --randrepeat=1 --ioengine=sync --direct=0 --gtod_reduce=1 
> --name=readwrite \
> --filename=rw.data --bs=4k --iodepth=1 --size=300MB 
> --readwrite=randrw \
>
>
> and indeed with cephfs _and_ rbd, I have approximatively the same result:
> - cephfs => ~516 iops
> - rbd=> ~587 iops
>
> Is it consistent?
>
yes

> That being said, I'm unable to know if it's good performance as regard my 
> hardware
> configuration. I'm curious to know the result in other clusters with the same 
> fio
> command.

This fio command is check performance of single thread SYNC IO. If you
want to check overall throughput, you can try using buffered IO or
increasing thread number.

FYI, I have written a patch to add AIO support to cephfs kernel client:
https://github.com/ceph/ceph-client/commits/testing

>
> Another point: I have noticed something which is very strange for me. It's 
> about
> the rados block device and this fio command:
>
> # In this case, I use libaio and (direct == 0)
> fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 
> --name=readwrite \
> --filename=rw.data --bs=4k --iodepth=16 --size=300MB 
> --readwrite=randrw  \
> --rwmixread=50
>
> This command in the rados block device gives me ~570 iops. But the curious 
> thing
> is that I have better iops if I just change "--direct=0" to "--direct=1" in 
> the
> command above. In this case, I have ~1400 iops. I don't understand this 
> difference.
> So, I have better perfs with "--direct=1":
>
> * --direct=1 => ~1400 iops
> * --direct=0 => ~570 iops
>
> Why I have this behavior? I thought it will be the opposite (better perfs with
> --direct=0). Is it normal?
>
linux kernel only supports AIO for fd opened in O_DIRECT mode, when
file is not opened in O_DIRECT mode, AIO is actually SYNC IO.

Regards
Yan, Zheng


> --
> François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[Wido den Hollander]

On 12/28/2015 07:18 AM, KeesBoog wrote:
> Hi,
> 
> I do love rados-java! I do get listings of ceph pools; I even can upload
> files to a ceph pool with a Tomcat web application. That is great!
> 

Great to hear!

> However, I get frequent SIGSEGV errors (which halt Tomcat):
> 

Ok, that is not good. Can you reproduce it with maybe some example code?
I would like to dig down to where this is happening, but if you are able
to reproduce it with some simple example code it would be great.

Or a stacktrace on which call it actually dies.

Wido

> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fd4bc54166d, pid=25308, tid=140554357282560
> #
> # JRE version: OpenJDK Runtime Environment (7.0_91-b02) (build
> 1.7.0_91-b02)
> # Java VM: OpenJDK 64-Bit Server VM (24.91-b01 mixed mode linux-amd64
> compressed oops)
> # Derivative: IcedTea 2.6.3
> # Distribution: Ubuntu 14.04 LTS, package 7u91-2.6.3-0ubuntu0.14.04.1
> # Problematic frame:
> # C  [librados.so.2+0x28766d]  Mutex::Lock(bool)+0xd
> #
> # Failed to write core dump. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid25308.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> #   http://icedtea.classpath.org/bugzilla
> #
> 
> Please find the file hs_err_pid25308.log attached. I could send a few more.
> 
> It would be great if someone could point me to a workaround of perhaps
> even fix the issue.
> 
> Best Regards,
> 
> Kees
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[KeesBoog]

Thanks Wido,

Here is the example code, taken from a Java web application.
There are no other calls to Rbd/Rados in the code.

// code altered for sake of the example 
(everything in one try/catch and random file name and initial size)
// objective: read binary file(s) from an 
inputstream to a ceph image


String file_id = "testupload_" + 
Id.getNext("file_id");

long initialsize = 10L;

try {
String config_file = "/etc/ceph/ceph.conf"; 
// nothing fancy here

String id = "admin";
String pool = "temp";

Rados rados;
IoCTX ioctx;
Rbd rbd;

// make the connection
rados = new Rados(id) {
};
rados.confReadFile(new File(config_file));
rados.connect();
ioctx = rados.ioCtxCreate(pool);
rbd = new Rbd(ioctx);

// create the file with initial size 
(larger than actual size)

int features = (1 << 0);
rbd.create(file_id, initialsize, features, 0);

// open the image
RbdImage image = rbd.open(file_id);;

// read bytes from the inputstream
int read = 0;
long bytesRead = 0;
byte[] bytes = new byte[8192];

while ((read = inputStream.read(bytes)) != 
-1) {

image.write(bytes, bytesRead, read);
bytesRead += read;
}

// adjust size to the actual read bytes
image.resize(bytesRead);

// close the image
rbd.close(image);

// close the connection
rados.ioCtxDestroy(ioctx);
rados.shutDown();
} catch (RadosException ex) {
Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE, 
null, ex);

}

This all works.

I uploaded a very small text-file.
rbd info temp/testupload_1028
rbd image 'testupload_1028':
size 5069 bytes in 1 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61b774b0dc51
format: 2
features: layering

About 40 seconds after the successful upload (the content equals the 
original content) I get an unexpected SIGSEGV-error as reported (without 
a stacktrace).


After a restart of Tomcat I did a second upload with a somewhat larger file:
rbd info temp/testupload_1030
rbd image 'testupload_1030':
size 41151 kB in 11 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61c62ae8944a
format: 2
features: layering

About 3 minutes after the successful upload I got the same SIGSEGV-error.

And a third (first restart Tomcat again):
rbd info temp/testupload_1032
rbd image 'testupload_1032':
size 127 MB in 32 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61fc2ae8944a
format: 2
features: layering

About 2 minutes after the successful upload I got the same SIGSEGV-error.

Thanks again.

Kees

Op 28-12-2015 om 10:21 schreef Wido den Hollander:

On 12/28/2015 07:18 AM, KeesBoog wrote:

Hi,

I do love rados-java! I do get listings of ceph pools; I even can upload
files to a ceph pool with a Tomcat web application. That is great!


Great to hear!


However, I get frequent SIGSEGV errors (which halt Tomcat):


Ok, that is not good. Can you reproduce it with maybe some example code?
I would like to dig down to where this is happening, but if you are able
to reproduce it with some simple example code it would be great.

Or a stacktrace on which call it actually dies.

Wido


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fd4bc54166d, pid=25308, tid=140554357282560
#
# JRE version: OpenJDK Runtime Environment (7.0_91-b02) (build
1.7.0_91-b02)
# Java VM: OpenJDK 64-Bit Server VM (24.91-b01 mixed mode linux-amd64
compressed oops)
# Derivative: IcedTea 2.6.3
# Distribution: Ubuntu 14.04 LTS, package 7u91-2.6.3-0ubuntu0.14.04.1
# Problematic frame:
# C  [librados.so.2+0x28766d]  Mutex::Lock(bool)+0xd
#
# Failed to write core dump. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[Florent Manens]

Hi Kees, 

Why don't you use plain Rados objects instead of a RBD ? 

Best regards,

- Le 28 Déc 15, à 11:54, KeesBoog techie2...@denboogert.com a écrit :

> Thanks Wido,

> Here is the example code, taken from a Java web application.
> There are no other calls to Rbd/Rados in the code.

> // code altered for sake of the example
> (everything in one try/catch and random file name and initial size)
> // objective: read binary file(s) from an
> inputstream to a ceph image

> String file_id = "testupload_" +
> Id.getNext("file_id");
> long initialsize = 10L;

> try {
> String config_file = "/etc/ceph/ceph.conf";
> // nothing fancy here
> String id = "admin";
> String pool = "temp";

> Rados rados;
> IoCTX ioctx;
> Rbd rbd;

> // make the connection
> rados = new Rados(id) {
> };
> rados.confReadFile(new File(config_file));
> rados.connect();
> ioctx = rados.ioCtxCreate(pool);
> rbd = new Rbd(ioctx);

> // create the file with initial size
> (larger than actual size)
> int features = (1 << 0);
> rbd.create(file_id, initialsize, features, 0);

> // open the image
> RbdImage image = rbd.open(file_id);;

> // read bytes from the inputstream
> int read = 0;
> long bytesRead = 0;
> byte[] bytes = new byte[8192];

> while ((read = inputStream.read(bytes)) !=
> -1) {
> image.write(bytes, bytesRead, read);
> bytesRead += read;
> }

> // adjust size to the actual read bytes
> image.resize(bytesRead);

> // close the image
> rbd.close(image);

> // close the connection
> rados.ioCtxDestroy(ioctx);
> rados.shutDown();
> } catch (RadosException ex) {
> Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
> null, ex);
> }

> This all works.

> I uploaded a very small text-file.
> rbd info temp/testupload_1028
> rbd image 'testupload_1028':
> size 5069 bytes in 1 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.1f61b774b0dc51
> format: 2
> features: layering

> About 40 seconds after the successful upload (the content equals the
> original content) I get an unexpected SIGSEGV-error as reported (without
> a stacktrace).

> After a restart of Tomcat I did a second upload with a somewhat larger file:
> rbd info temp/testupload_1030
> rbd image 'testupload_1030':
> size 41151 kB in 11 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.1f61c62ae8944a
> format: 2
> features: layering

> About 3 minutes after the successful upload I got the same SIGSEGV-error.

> And a third (first restart Tomcat again):
> rbd info temp/testupload_1032
> rbd image 'testupload_1032':
> size 127 MB in 32 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.1f61fc2ae8944a
> format: 2
> features: layering

> About 2 minutes after the successful upload I got the same SIGSEGV-error.

> Thanks again.

> Kees

> Op 28-12-2015 om 10:21 schreef Wido den Hollander:
> > On 12/28/2015 07:18 AM, KeesBoog wrote:
> >> Hi,

> >> I do love rados-java! I do get listings of ceph pools; I even can upload
> >> files to a ceph pool with a Tomcat web application. That is great!

> > Great to hear!

> >> However, I get frequent SIGSEGV errors (which halt Tomcat):

> > Ok, that is not good. Can you reproduce it with maybe some example code?
> > I would like to dig down to where this is happening, but if you are able
> > to reproduce it with some simple example code it would be great.

> > Or a stacktrace on which call it actually dies.

> > Wido

> >> #
> >> # A fatal error has been detected by the Java Runtime Environment:
> >> #
> >> # SIGSEGV (0xb) at pc=0x7fd4bc54166d, pid=25308, tid=140554357282560
> >> #
> >> # JRE version: OpenJDK Runtime Environment (7.0_91-b02) (build
> >> 1.7.0_91-b02)
> >> # Java VM: OpenJDK 64-Bit Server VM (24.91-b01 mixed mode linux-amd64
> >> compressed oops)
> >> # Derivative: IcedTea 2.6.3
> >> # Distribution: Ubuntu 14.04 LTS, package 7u91-2.6.3-0ubuntu0.14.04.1
> >> # Problematic frame:
> >> # C [librados.so.2+0x28766d] Mutex::Lock(bool)+0xd
> >> #
> >> # Failed to write core dump. Core dumps have been disabled. To enable
> >> core dumping, try "ulimit -c unlimited" before starting Java again
> >> #
> >> # An error report file with more information is saved as:
> >> # /tmp/hs_err_pid25308.log
> >> #
> >> # If you would like to submit a bug report, please include
> >> # instructions on how to reproduce the bug and visit:
> >> # http://icedtea.classpath.org/bugzilla
> >> #

> >> Please find the file hs_err_pid25308.log attached. I could send a few more.

> >> It would be great if someone could point me to a workaround of perhaps
> >> even fix the issue.

> >> Best Regards,

> >> Kees

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[KeesBoog]

Hi Florent,

Thanks.

I did not consider 'plain Rados objects' and I am not sure how that 
would work. If you think that would solve the problem, perhaps you could 
elaborate?


Would I still need Rbd to create a new image and set the size?

In the example I replaced
RbdImage image = rbd.open(file_id);
image.write(bytes, bytesRead, read);
with
ioctx.write("rbd_id."+file_id, bytes, bytesRead);

but
a) the image-file was empty after the upload (so I am sure I am doing 
something wrong)

b) the SIGSEGV-error was still there (after a few minutes)

Best Regards,

Kees


Op 28-12-2015 om 12:33 schreef Florent Manens:

Hi Kees,

Why don't you use plain Rados objects instead of a RBD ?

Best regards,

- Le 28 Déc 15, à 11:54, KeesBoog techie2...@denboogert.com a écrit :


Thanks Wido,
Here is the example code, taken from a Java web application.
There are no other calls to Rbd/Rados in the code.
// code altered for sake of the example
(everything in one try/catch and random file name and initial size)
// objective: read binary file(s) from an
inputstream to a ceph image
String file_id = "testupload_" +
Id.getNext("file_id");
long initialsize = 10L;
try {
String config_file = "/etc/ceph/ceph.conf";
// nothing fancy here
String id = "admin";
String pool = "temp";
Rados rados;
IoCTX ioctx;
Rbd rbd;
// make the connection
rados = new Rados(id) {
};
rados.confReadFile(new File(config_file));
rados.connect();
ioctx = rados.ioCtxCreate(pool);
rbd = new Rbd(ioctx);
// create the file with initial size
(larger than actual size)
int features = (1 << 0);
rbd.create(file_id, initialsize, features, 0);
// open the image
RbdImage image = rbd.open(file_id);;
// read bytes from the inputstream
int read = 0;
long bytesRead = 0;
byte[] bytes = new byte[8192];
while ((read = inputStream.read(bytes)) !=
-1) {
image.write(bytes, bytesRead, read);
bytesRead += read;
}
// adjust size to the actual read bytes
image.resize(bytesRead);
// close the image
rbd.close(image);
// close the connection
rados.ioCtxDestroy(ioctx);
rados.shutDown();
} catch (RadosException ex) {
Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
null, ex);
}
This all works.
I uploaded a very small text-file.
rbd info temp/testupload_1028
rbd image 'testupload_1028':
size 5069 bytes in 1 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61b774b0dc51
format: 2
features: layering
About 40 seconds after the successful upload (the content equals the
original content) I get an unexpected SIGSEGV-error as reported (without
a stacktrace).
After a restart of Tomcat I did a second upload with a somewhat larger file:
rbd info temp/testupload_1030
rbd image 'testupload_1030':
size 41151 kB in 11 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61c62ae8944a
format: 2
features: layering
About 3 minutes after the successful upload I got the same SIGSEGV-error.
And a third (first restart Tomcat again):
rbd info temp/testupload_1032
rbd image 'testupload_1032':
size 127 MB in 32 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61fc2ae8944a
format: 2
features: layering
About 2 minutes after the successful upload I got the same SIGSEGV-error.
Thanks again.
Kees
Op 28-12-2015 om 10:21 schreef Wido den Hollander:

On 12/28/2015 07:18 AM, KeesBoog wrote:

Hi,
I do love rados-java! I do get listings of ceph pools; I even can upload
files to a ceph pool with a Tomcat web application. That is great!

Great to hear!

However, I get frequent SIGSEGV errors (which halt Tomcat):

Ok, that is not good. Can you reproduce it with maybe some example code?
I would like to dig down to where this is happening, but if you are able
to reproduce it with some simple example code it would be great.
Or a stacktrace on which call it actually dies.
Wido

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x7fd4bc54166d, pid=25308, tid=140554357282560
#
# JRE version: OpenJDK Runtime Environment (7.0_91-b02) (build
1.7.0_91-b02)
# Java VM: OpenJDK 64-Bit Server VM (24.91-b01 mixed mode linux-amd64
compressed oops)
# Derivative: IcedTea 2.6.3
# Distribution: Ubuntu 14.04 LTS, package 7u91-2.6.3-0ubuntu0.14.04.1
# Problematic frame:
# C [librados.so.2+0x28766d] Mutex::Lock(bool)+0xd
#
# Failed to write core dump. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid25308.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
Please find the file hs_err_pid25308.log attached. I could send a few more.
It would be great if someone could point me to a workaround of perhaps
even fix the issue.
Best Regards,
Kees




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com

[ceph-users] Ceph & Hbase

[Jose M]

Hi!


Does anyone succesfully configure hbase to use ceph? I'm having some problems 
with it, maybe anyone can help.


I have ceph already running and ceph-hadoop bindings installed ('hadoop fs -ls 
/' working). I'm trying Hbase in pseudo distributed mode, but when starting 
hbase-master I'm getting this error


2015-12-27 16:41:53,962 INFO  [main] mob.MobFileCache: MobFileCache is 
initialized, and the cache size is 1000
2015-12-27 16:41:54,484 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class 
org.apache.hadoop.hbase.master.HMaster
at 
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2046)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at 
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2060)
Caused by: java.io.IOException: Permission denied
at com.ceph.fs.CephMount.native_ceph_mount(Native Method)
at com.ceph.fs.CephMount.mount(CephMount.java:154)
at org.apache.hadoop.fs.ceph.CephTalker.initialize(CephTalker.java:141)
at 
org.apache.hadoop.fs.ceph.CephFileSystem.initialize(CephFileSystem.java:98)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2643)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2680)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2662)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:379)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:1005)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:532)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:347)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at 
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2041)


This is my core-site.xml




 fs.defaultFS
 ceph://cephmaster:6789/


ceph.conf.file
 /etc/ceph/ceph.conf


 ceph.auth.id
 admin


ceph.auth.keyring
/home/developer/ceph-cluster/ceph.client.admin.keyring


 fs.ceph.impl
 org.apache.hadoop.fs.ceph.CephFileSystem


 fs.AbstractFileSystem.ceph.impl
 org.apache.hadoop.fs.ceph.CephHadoop2FileSystem


ceph.data.pools
hadoop1


ceph.mon.address
cephmaster:6789


ceph.object.size
 67108864




This is my hbase-site.xml




  hbase.cluster.distributed
  true


  hbase.rootdir
  ceph://cephmaster:6789/hbase

  
hbase.zookeeper.quorum
cephmaster
  



I already tried adding here fs.defaultFS, ceph.conf.file, ceph.auth.id, 
ceph.auth.keyfile, fs.ceph.impl and fs.AbstractFileSystem.ceph.impl in 
hbase-site.xml just in case (although i think they aren't needed here) but I 
got the same exception.


I suppose from the exception I have some permission error with ceph fs but I'm 
not sure.

Hope you can help me finding the right direction to fix this:)


Thanks in advance,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[Florent Manens]

Hi Kees,

I didn't test but try this : 

String file_id = "testupload_" + Id.getNext("file_id");

try {
String config_file = "/etc/ceph/ceph.conf";
// nothing fancy here
String id = "admin";
String pool = "temp";

Rados rados;
IoCTX ioctx;

// make the connection
rados = new Rados(id) {
};
rados.confReadFile(new File(config_file));
rados.connect();
ioctx = rados.ioCtxCreate(pool);

// read bytes from the inputstream
int read = 0;
long bytesRead = 0;
byte[] bytes = new byte[8192];

while ((read = inputStream.read(bytes)) !=
-1) {
//truncate to really read data
byte[] real_buff = Arrays.copyOf(bytes, read);
ioctx.write(file_id, real_buff, bytesRead);
bytesRead += read;
}

// close the connection
rados.ioCtxDestroy(ioctx);
rados.shutDown();
} catch (RadosException ex) {

Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
null, ex);
}

I use Rados object for my Zimbra<->Ceph connector and I don't have segfault 
problems with rados-java.

Best regards,


- Le 28 Déc 15, à 13:43, KeesBoog techie2...@denboogert.com a écrit :

> Hi Florent,

> Thanks.

> I did not consider 'plain Rados objects' and I am not sure how that
> would work. If you think that would solve the problem, perhaps you could
> elaborate?

> Would I still need Rbd to create a new image and set the size?

> In the example I replaced
> RbdImage image = rbd.open(file_id);
> image.write(bytes, bytesRead, read);
> with
> ioctx.write("rbd_id."+file_id, bytes, bytesRead);

> but
> a) the image-file was empty after the upload (so I am sure I am doing
> something wrong)
> b) the SIGSEGV-error was still there (after a few minutes)

> Best Regards,

> Kees

> Op 28-12-2015 om 12:33 schreef Florent Manens:
> > Hi Kees,

> > Why don't you use plain Rados objects instead of a RBD ?

> > Best regards,

> > - Le 28 Déc 15, à 11:54, KeesBoog techie2...@denboogert.com a écrit :

> >> Thanks Wido,
> >> Here is the example code, taken from a Java web application.
> >> There are no other calls to Rbd/Rados in the code.
> >> // code altered for sake of the example
> >> (everything in one try/catch and random file name and initial size)
> >> // objective: read binary file(s) from an
> >> inputstream to a ceph image
> >> String file_id = "testupload_" +
> >> Id.getNext("file_id");
> >> long initialsize = 10L;
> >> try {
> >> String config_file = "/etc/ceph/ceph.conf";
> >> // nothing fancy here
> >> String id = "admin";
> >> String pool = "temp";
> >> Rados rados;
> >> IoCTX ioctx;
> >> Rbd rbd;
> >> // make the connection
> >> rados = new Rados(id) {
> >> };
> >> rados.confReadFile(new File(config_file));
> >> rados.connect();
> >> ioctx = rados.ioCtxCreate(pool);
> >> rbd = new Rbd(ioctx);
> >> // create the file with initial size
> >> (larger than actual size)
> >> int features = (1 << 0);
> >> rbd.create(file_id, initialsize, features, 0);
> >> // open the image
> >> RbdImage image = rbd.open(file_id);;
> >> // read bytes from the inputstream
> >> int read = 0;
> >> long bytesRead = 0;
> >> byte[] bytes = new byte[8192];
> >> while ((read = inputStream.read(bytes)) !=
> >> -1) {
> >> image.write(bytes, bytesRead, read);
> >> bytesRead += read;
> >> }
> >> // adjust size to the actual read bytes
> >> image.resize(bytesRead);
> >> // close the image
> >> rbd.close(image);
> >> // close the connection
> >> rados.ioCtxDestroy(ioctx);
> >> rados.shutDown();
> >> } catch (RadosException ex) {
> >> Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
> >> null, ex);
> >> }
> >> This all works.
> >> I uploaded a very small text-file.
> >> rbd info temp/testupload_1028
> >> rbd image 'testupload_1028':
> >> size 5069 bytes in 1 objects
> >> order 22 (4096 kB objects)
> >> block_name_prefix: rbd_data.1f61b774b0dc51
> >> format: 2
> >> features: layering
> >> About 40 seconds after the successful upload (the content equals the
> >> original content) I get an unexpected SIGSEGV-error as reported (without
> >> a stacktrace).
> >> After a restart of Tomcat I did a second upload with a somewhat larger 
> >> file:
> >> rbd info temp/testupload_1030
> >> rbd image 'testupload_1030':
> >> size 41151 kB in 11 objects
> >> order 22 (4096 kB objects)
> >> block_name_prefix: rbd_data.1f61c62ae8944a
> >> format: 2
> >> features: layering
> >> About 3 minutes after the successful upload I got the same SIGSEGV-error.
> >> And a third (first restart Tomcat again):
> >> rbd info temp/testupload_1032
> >> rbd image 'testupload_1032':
> >> size 127 MB in 32 objects
> >> order 22 (4096 kB objects)
> >> block_name_prefix: rbd_data.1f61fc2ae8944a
> >> format: 2
>

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[KeesBoog]

Thank you Florent and Wido,

It seems that the sigsegv error occurs regardless of the code. Even if I 
do not upload a file, the error occurs after a couple of minutes... Strange!


Thanks again,

Kees


Op 28-12-2015 om 13:54 schreef Florent Manens:

Hi Kees,

I didn't test but try this :

String file_id = "testupload_" + Id.getNext("file_id");

 try {
 String config_file = "/etc/ceph/ceph.conf";
// nothing fancy here
 String id = "admin";
 String pool = "temp";

 Rados rados;
 IoCTX ioctx;

 // make the connection
 rados = new Rados(id) {
 };
 rados.confReadFile(new File(config_file));
 rados.connect();
 ioctx = rados.ioCtxCreate(pool);

 // read bytes from the inputstream
 int read = 0;
 long bytesRead = 0;
 byte[] bytes = new byte[8192];

 while ((read = inputStream.read(bytes)) !=
 -1) {
 //truncate to really read data
 byte[] real_buff = Arrays.copyOf(bytes, read);
 ioctx.write(file_id, real_buff, bytesRead);
 bytesRead += read;
 }

 // close the connection
 rados.ioCtxDestroy(ioctx);
 rados.shutDown();
 } catch (RadosException ex) {
 
Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
 null, ex);
 }

I use Rados object for my Zimbra<->Ceph connector and I don't have segfault 
problems with rados-java.

Best regards,


- Le 28 Déc 15, à 13:43, KeesBoog techie2...@denboogert.com a écrit :


Hi Florent,
Thanks.
I did not consider 'plain Rados objects' and I am not sure how that
would work. If you think that would solve the problem, perhaps you could
elaborate?
Would I still need Rbd to create a new image and set the size?
In the example I replaced
RbdImage image = rbd.open(file_id);
image.write(bytes, bytesRead, read);
with
ioctx.write("rbd_id."+file_id, bytes, bytesRead);
but
a) the image-file was empty after the upload (so I am sure I am doing
something wrong)
b) the SIGSEGV-error was still there (after a few minutes)
Best Regards,
Kees
Op 28-12-2015 om 12:33 schreef Florent Manens:

Hi Kees,
Why don't you use plain Rados objects instead of a RBD ?
Best regards,
- Le 28 Déc 15, à 11:54, KeesBoog techie2...@denboogert.com a écrit :

Thanks Wido,
Here is the example code, taken from a Java web application.
There are no other calls to Rbd/Rados in the code.
// code altered for sake of the example
(everything in one try/catch and random file name and initial size)
// objective: read binary file(s) from an
inputstream to a ceph image
String file_id = "testupload_" +
Id.getNext("file_id");
long initialsize = 10L;
try {
String config_file = "/etc/ceph/ceph.conf";
// nothing fancy here
String id = "admin";
String pool = "temp";
Rados rados;
IoCTX ioctx;
Rbd rbd;
// make the connection
rados = new Rados(id) {
};
rados.confReadFile(new File(config_file));
rados.connect();
ioctx = rados.ioCtxCreate(pool);
rbd = new Rbd(ioctx);
// create the file with initial size
(larger than actual size)
int features = (1 << 0);
rbd.create(file_id, initialsize, features, 0);
// open the image
RbdImage image = rbd.open(file_id);;
// read bytes from the inputstream
int read = 0;
long bytesRead = 0;
byte[] bytes = new byte[8192];
while ((read = inputStream.read(bytes)) !=
-1) {
image.write(bytes, bytesRead, read);
bytesRead += read;
}
// adjust size to the actual read bytes
image.resize(bytesRead);
// close the image
rbd.close(image);
// close the connection
rados.ioCtxDestroy(ioctx);
rados.shutDown();
} catch (RadosException ex) {
Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
null, ex);
}
This all works.
I uploaded a very small text-file.
rbd info temp/testupload_1028
rbd image 'testupload_1028':
size 5069 bytes in 1 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61b774b0dc51
format: 2
features: layering
About 40 seconds after the successful upload (the content equals the
original content) I get an unexpected SIGSEGV-error as reported (without
a stacktrace).
After a restart of Tomcat I did a second upload with a somewhat larger file:
rbd info temp/testupload_1030
rbd image 'testupload_1030':
size 41151 kB in 11 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61c62ae8944a
format: 2
features: layering
About 3 minutes after the successful upload I got the same SIGSEGV-error.
And a third (first restart Tomcat again):
rbd info temp/testupload_1032
rbd image 'testupload_1032':
size 127 MB in 32 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f61fc2ae8944a
format: 2
features: layering
About 2 minutes after the successful upload I got the same SIGSEGV-error.
Thanks again.
Kees
Op 28-12-2015 om 10:21 schreef Wido den Hollander:

On 12/28/2015 07:

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[Wido den Hollander]

On 28-12-15 15:25, KeesBoog wrote:
> Thank you Florent and Wido,
> 
> It seems that the sigsegv error occurs regardless of the code. Even if I
> do not upload a file, the error occurs after a couple of minutes...
> Strange!
> 

Did you open a RADOS context or connection in that case?

What version of librados are you using on the system and what is the
Ceph cluster running?

Wido

> Thanks again,
> 
> Kees
> 
> 
> Op 28-12-2015 om 13:54 schreef Florent Manens:
>> Hi Kees,
>>
>> I didn't test but try this :
>>
>> String file_id = "testupload_" + Id.getNext("file_id");
>>
>>  try {
>>  String config_file = "/etc/ceph/ceph.conf";
>> // nothing fancy here
>>  String id = "admin";
>>  String pool = "temp";
>>
>>  Rados rados;
>>  IoCTX ioctx;
>>
>>  // make the connection
>>  rados = new Rados(id) {
>>  };
>>  rados.confReadFile(new File(config_file));
>>  rados.connect();
>>  ioctx = rados.ioCtxCreate(pool);
>>
>>  // read bytes from the inputstream
>>  int read = 0;
>>  long bytesRead = 0;
>>  byte[] bytes = new byte[8192];
>>
>>  while ((read = inputStream.read(bytes)) !=
>>  -1) {
>>  //truncate to really read data
>>  byte[] real_buff = Arrays.copyOf(bytes, read);
>>  ioctx.write(file_id, real_buff, bytesRead);
>>  bytesRead += read;
>>  }
>>
>>  // close the connection
>>  rados.ioCtxDestroy(ioctx);
>>  rados.shutDown();
>>  } catch (RadosException ex) {
>> 
>> Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
>>
>>  null, ex);
>>  }
>>
>> I use Rados object for my Zimbra<->Ceph connector and I don't have
>> segfault problems with rados-java.
>>
>> Best regards,
>>
>>
>> - Le 28 Déc 15, à 13:43, KeesBoog techie2...@denboogert.com a écrit :
>>
>>> Hi Florent,
>>> Thanks.
>>> I did not consider 'plain Rados objects' and I am not sure how that
>>> would work. If you think that would solve the problem, perhaps you could
>>> elaborate?
>>> Would I still need Rbd to create a new image and set the size?
>>> In the example I replaced
>>> RbdImage image = rbd.open(file_id);
>>> image.write(bytes, bytesRead, read);
>>> with
>>> ioctx.write("rbd_id."+file_id, bytes, bytesRead);
>>> but
>>> a) the image-file was empty after the upload (so I am sure I am doing
>>> something wrong)
>>> b) the SIGSEGV-error was still there (after a few minutes)
>>> Best Regards,
>>> Kees
>>> Op 28-12-2015 om 12:33 schreef Florent Manens:
 Hi Kees,
 Why don't you use plain Rados objects instead of a RBD ?
 Best regards,
 - Le 28 Déc 15, à 11:54, KeesBoog techie2...@denboogert.com a
 écrit :
> Thanks Wido,
> Here is the example code, taken from a Java web application.
> There are no other calls to Rbd/Rados in the code.
> // code altered for sake of the example
> (everything in one try/catch and random file name and initial size)
> // objective: read binary file(s) from an
> inputstream to a ceph image
> String file_id = "testupload_" +
> Id.getNext("file_id");
> long initialsize = 10L;
> try {
> String config_file = "/etc/ceph/ceph.conf";
> // nothing fancy here
> String id = "admin";
> String pool = "temp";
> Rados rados;
> IoCTX ioctx;
> Rbd rbd;
> // make the connection
> rados = new Rados(id) {
> };
> rados.confReadFile(new File(config_file));
> rados.connect();
> ioctx = rados.ioCtxCreate(pool);
> rbd = new Rbd(ioctx);
> // create the file with initial size
> (larger than actual size)
> int features = (1 << 0);
> rbd.create(file_id, initialsize, features, 0);
> // open the image
> RbdImage image = rbd.open(file_id);;
> // read bytes from the inputstream
> int read = 0;
> long bytesRead = 0;
> byte[] bytes = new byte[8192];
> while ((read = inputStream.read(bytes)) !=
> -1) {
> image.write(bytes, bytesRead, read);
> bytesRead += read;
> }
> // adjust size to the actual read bytes
> image.resize(bytesRead);
> // close the image
> rbd.close(image);
> // close the connection
> rados.ioCtxDestroy(ioctx);
> rados.shutDown();
> } catch (RadosException ex) {
> Logger.getLogger(ApiFunction_dvd_upload.class.getName()).log(Level.SEVERE,
>
> null, ex);
> }
> This all works.
> I uploaded a very small text-file.
> rbd info temp/testupload_1028
> rbd image 'testupload_1028':
> size 5069 bytes in 1 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.1f61b774b0dc51
> format: 2
> features: layering
> About 40 seconds after the successful upload (the content 

Re: [ceph-users] Help! OSD host failure - recovery without rebuilding OSDs

[Josef Johansson]

Did you manage to work this out?
On 25 Dec 2015 9:33 am, "Josef Johansson"  wrote:

> Hi
>
> Someone here will probably lay out a detailed answer but to get you
> started,
>
> All the details for the osd are in the xfs partitions, mirror a new USB
> key and change ip etc and you should be able to recover.
>
> If the journal is linked to a /dev/sdx, make sure it's in the same spot as
> it was before..
>
> All the best of luck
> /Josef
> On 25 Dec 2015 05:39, "deeepdish"  wrote:
>
>> Hello,
>>
>> Had an interesting issue today.
>>
>> My OSD hosts are booting off a USB key which, you guessed it has a root
>> partition on there.   All OSDs are mounted.   My USB key failed on one of
>> my OSD hosts, leaving the data on OSDs inaccessible to the rest of my
>> cluster.   I have multiple monitors running other OSD hosts where data can
>> be recovered to.   However I’m wondering if there’s a way to “restore” /
>> “rebuild” the ceph install that was on this host without having all OSDs
>> resync again.
>>
>> Lesson learned = don’t use USB boot/root drives.   However, now just
>> looking at what needs to be done once the OS and Ceph packages are
>> reinstalled.
>>
>> Thank you.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help! OSD host failure - recovery without rebuilding OSDs

[deeepdish]

HI Josef,

Yes, everything came back to normal.   Thanks for following up!


> On Dec 28, 2015, at 11:25 , Josef Johansson  wrote:
> 
> Did you manage to work this out?
> 
> On 25 Dec 2015 9:33 am, "Josef Johansson"  > wrote:
> Hi
> 
> Someone here will probably lay out a detailed answer but to get you started,
> 
> All the details for the osd are in the xfs partitions, mirror a new USB key 
> and change ip etc and you should be able to recover.
> 
> If the journal is linked to a /dev/sdx, make sure it's in the same spot as it 
> was before..
> 
> All the best of luck
> /Josef
> 
> On 25 Dec 2015 05:39, "deeepdish"  > wrote:
> Hello,
> 
> Had an interesting issue today.
> 
> My OSD hosts are booting off a USB key which, you guessed it has a root 
> partition on there.   All OSDs are mounted.   My USB key failed on one of my 
> OSD hosts, leaving the data on OSDs inaccessible to the rest of my cluster.   
> I have multiple monitors running other OSD hosts where data can be recovered 
> to.   However I’m wondering if there’s a way to “restore” / “rebuild” the 
> ceph install that was on this host without having all OSDs resync again.
> 
> Lesson learned = don’t use USB boot/root drives.   However, now just looking 
> at what needs to be done once the OS and Ceph packages are reinstalled.
> 
> Thank you.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [rados-java] SIGSEGV librados.so Ubuntu

[KeesBoog]

Hi Wido and Florent,

After going back into the error-report with my new knowledge, I think I 
found the problem.


It is the finalizer in com.ceph.rados.Rados.java
Commenting out the finalize() method solved the problem.
I have no idea why it causes all this trouble, and I don't know if 
commenting it out is the solution for every use case, but I think I can 
do without the finalizer.


I tried different versions of jna, including the latest one jna-4.2.1. 
That works.


---
*You asked:**
*ceph --version
ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)
librados-config --version
ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)

---
*One more thing**
*When recompiling rados.jar according to the instructions on 
https://github.com/ceph/rados-java I encountered one test-error:
  testCreateFormatOne(com.ceph.rbd.TestRbd): Failed to remove image 
imageformat1

This may have to do with ceph.conf-setting: rbd default format = 2
 rbd info rbd/imageformat1
rbd image 'imageformat1':
size 10240 kB in 3 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1f6a9d7fdcc233
format: 2
features: layering, striping
stripe unit: 4096 kB
stripe count: 1
I don't fully understand, as the "format 2" should have failed the 
previous test: assertTrue("The image wasn't the old (1) format", oldFormat).


and this documentation is obsolete:
http://docs.ceph.com/docs/hammer/rados/api/librados-intro/#getting-librados-for-java
---

Thanks again! It great to get some friendly support; it kept me searching...

Best Regards,

Kees



Op 28-12-2015 om 15:28 schreef Wido den Hollander:


On 28-12-15 15:25, KeesBoog wrote:

Thank you Florent and Wido,

It seems that the sigsegv error occurs regardless of the code. Even if I
do not upload a file, the error occurs after a couple of minutes...
Strange!


Did you open a RADOS context or connection in that case?

What version of librados are you using on the system and what is the
Ceph cluster running?

Wido



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] My OSDs are down and not coming UP

[Ing. Martin Samek]

Hi,

I'm a newbie in a Ceph world. I try setup my first testing Ceph cluster 
but unlikely my MON server running and talking each to other but my OSDs 
are still down and won't to come up. Actually only the one OSD running 
at the same node as a elected master is able to connect and come UP.


To be technical. I have 4 physical nodes living in pure IPv6 
environment, running Gentoo Linux and Ceph 9.2. All nodes names are 
resolvable in DNS and also saved in hosts files.


I'm running OSD with command like this:

node1# /usr/bin/ceph-osd -f -i 1 --pid-file /run/ceph/osd.1.pid -c 
/etc/ceph/ceph.conf

single mon.0 is running also at node1, and OSD come up:

2015-12-28 23:37:27.931686 mon.0 [INF] osd.1 [2001:718:2:1612::50]:6800/23709 
boot

2015-12-28 23:37:27.932605 mon.0 [INF] osdmap e19: 2 osds: 1 up, 1 in

2015-12-28 23:37:27.933963 mon.0 [INF] pgmap v24: 64 pgs: 64 
stale+active+undersized+degraded; 0 bytes data, 1057 MB used, 598 GB / 599 GB 
avail

but running osd.0 at node2:

# /usr/bin/ceph-osd -f -i 0 --pid-file /run/ceph/osd.0.pid -c 
/etc/ceph/ceph.conf

did nothing, process is running, netstat shows opened connection from 
ceph-osd between node2 and node1. Here I'm lost. IPv6 connectivity is 
OK, DNS is OK, time is in sync, 1 mon running, 2 osds but only one UP. 
What is missing?


ceph-osd in debug mode show differences at node1 and node2:

node1, UP:

2015-12-28 01:42:59.084371 7f72f9873800 20 osd.1 15  clearing temps in 
0.3f_head pgid 0.3f
2015-12-28 01:42:59.084453 7f72f9873800  0 osd.1 15 load_pgs
2015-12-28 01:42:59.085248 7f72f9873800 10 osd.1 15 load_pgs ignoring 
unrecognized meta
2015-12-28 01:42:59.094690 7f72f9873800 10 osd.1 15 pgid 0.0 coll 0.0_head
2015-12-28 01:42:59.094835 7f72f9873800 30 osd.1 0 get_map 15 -cached
2015-12-28 01:42:59.094848 7f72f9873800 10 osd.1 15 _open_lock_pg 0.0
2015-12-28 01:42:59.094857 7f72f9873800 10 osd.1 15 _get_pool 0
2015-12-28 01:42:59.094928 7f72f9873800  5 osd.1 pg_epoch: 15 pg[0.0(unlocked)] 
enter Initial
2015-12-28 01:42:59.094980 7f72f9873800 20 osd.1 pg_epoch: 15 pg[0.0(unlocked)] 
enter NotTrimming
2015-12-28 01:42:59.094998 7f72f9873800 30 osd.1 pg_epoch: 15 pg[0.0( DNE empty 
local-les=0 n=0 ec=0 les/c/f 0/0/0 0/0/0) [] r=0 lpr=0 crt=0'0 inactive NIBBLEW
2015-12-28 01:42:59.095186 7f72f9873800 20 read_log coll 0.0_head log_oid 
0///head


node2, DOWN:

2015-12-28 01:36:54.437246 7f4507957800  0 osd.0 11 load_pgs
2015-12-28 01:36:54.437267 7f4507957800 10 osd.0 11 load_pgs ignoring 
unrecognized meta
2015-12-28 01:36:54.437274 7f4507957800  0 osd.0 11 load_pgs opened 0 pgs
2015-12-28 01:36:54.437278 7f4507957800 10 osd.0 11 
build_past_intervals_parallel nothing to build
2015-12-28 01:36:54.437282 7f4507957800  2 osd.0 11 superblock: i am osd.0
2015-12-28 01:36:54.437287 7f4507957800 10 osd.0 11 create_logger
2015-12-28 01:36:54.438157 7f4507957800 -1 osd.0 11 log_to_monitors 
{default=true}
2015-12-28 01:36:54.449278 7f4507957800 10 osd.0 11 set_disk_tp_priority class  
priority -1
2015-12-28 01:36:54.450813 7f44ddbff700 30 osd.0 11 heartbeat
2015-12-28 01:36:54.452558 7f44ddbff700 30 osd.0 11 heartbeat checking stats
2015-12-28 01:36:54.452592 7f44ddbff700 20 osd.0 11 update_osd_stat 
osd_stat(1056 MB used, 598 GB avail, 599 GB total, peers []/[] op hist [])
2015-12-28 01:36:54.452611 7f44ddbff700  5 osd.0 11 heartbeat: osd_stat(1056 MB 
used, 598 GB avail, 599 GB total, peers []/[] op hist [])
2015-12-28 01:36:54.452618 7f44ddbff700 30 osd.0 11 heartbeat check
2015-12-28 01:36:54.452622 7f44ddbff700 30 osd.0 11 heartbeat lonely?
2015-12-28 01:36:54.452624 7f44ddbff700 30 osd.0 11 heartbeat done
2015-12-28 01:36:54.452627 7f44ddbff700 30 osd.0 11 heartbeat_entry sleeping 
for 2.3
2015-12-28 01:36:54.452588 7f44da7fc700 10 osd.0 11 agent_entry start
2015-12-28 01:36:54.453338 7f44da7fc700 20 osd.0 11 agent_entry empty queue


My ceph.conf looks like this:

[global]

fsid = b186d870-9c6d-4a8b-ac8a-e263f4c205da

ms_bind_ipv6 = true

public_network = ::2:1612::/64

mon initial members = 0

mon host = [::2:1612::50]:6789

auth cluster required = cephx

auth service required = cephx

auth client required = cephx

osd pool default size = 2

osd pool default min size = 1

osd journal size = 1024

osd mkfs type = xfs

osd mount options xfs = rw,inode64

osd crush chooseleaf type = 1

[mon.0]

host = node1

mon addr = [::2:1612::50]:6789

[mon.1]

host = node3

mon addr = [::2:1612::30]:6789

[osd.0]

host = node2

devs = /dev/vg0/osd0

[osd.1]

host = node1

devs = /dev/vg0/osd


My ceph osd tree:

node1 # ceph osd tree

ID WEIGHT  TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 2.0 root default

-2 1.0 host node2

 0 1.0 osd.0 down0  1.0

-3 1.0 host node1

 1 1.0 osd.1   up  1.0  1.0

Any help how to cope with this is appreciated. I follow steps in this 
guides:


https://wiki.gentoo.org/wiki/Ceph/Installation#I

Re: [ceph-users] My OSDs are down and not coming UP

[Somnath Roy]

It could be a network issue..May be related to MTU (?)..Try running with 
debug_ms = 1 and see if you find anything..Also, try running command like 
'traceroute' and see if it is reporting any error..

Thanks & Regards
Somnath

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ing. 
Martin Samek
Sent: Monday, December 28, 2015 2:59 PM
To: Ceph Users
Subject: [ceph-users] My OSDs are down and not coming UP

Hi,

I'm a newbie in a Ceph world. I try setup my first testing Ceph cluster but 
unlikely my MON server running and talking each to other but my OSDs are still 
down and won't to come up. Actually only the one OSD running at the same node 
as a elected master is able to connect and come UP.

To be technical. I have 4 physical nodes living in pure IPv6 environment, 
running Gentoo Linux and Ceph 9.2. All nodes names are resolvable in DNS and 
also saved in hosts files.

I'm running OSD with command like this:

node1# /usr/bin/ceph-osd -f -i 1 --pid-file /run/ceph/osd.1.pid -c 
/etc/ceph/ceph.conf

single mon.0 is running also at node1, and OSD come up:

2015-12-28 23:37:27.931686 mon.0 [INF] osd.1 [2001:718:2:1612::50]:6800/23709 
boot

2015-12-28 23:37:27.932605 mon.0 [INF] osdmap e19: 2 osds: 1 up, 1 in

2015-12-28 23:37:27.933963 mon.0 [INF] pgmap v24: 64 pgs: 64 
stale+active+undersized+degraded; 0 bytes data, 1057 MB used, 598 GB / 599 GB 
avail

but running osd.0 at node2:

# /usr/bin/ceph-osd -f -i 0 --pid-file /run/ceph/osd.0.pid -c 
/etc/ceph/ceph.conf

did nothing, process is running, netstat shows opened connection from ceph-osd 
between node2 and node1. Here I'm lost. IPv6 connectivity is OK, DNS is OK, 
time is in sync, 1 mon running, 2 osds but only one UP. 
What is missing?

ceph-osd in debug mode show differences at node1 and node2:

node1, UP:
> 2015-12-28 01:42:59.084371 7f72f9873800 20 osd.1 15  clearing temps in 
> 0.3f_head pgid 0.3f
> 2015-12-28 01:42:59.084453 7f72f9873800  0 osd.1 15 load_pgs
> 2015-12-28 01:42:59.085248 7f72f9873800 10 osd.1 15 load_pgs ignoring 
> unrecognized meta
> 2015-12-28 01:42:59.094690 7f72f9873800 10 osd.1 15 pgid 0.0 coll 
> 0.0_head
> 2015-12-28 01:42:59.094835 7f72f9873800 30 osd.1 0 get_map 15 -cached
> 2015-12-28 01:42:59.094848 7f72f9873800 10 osd.1 15 _open_lock_pg 0.0
> 2015-12-28 01:42:59.094857 7f72f9873800 10 osd.1 15 _get_pool 0
> 2015-12-28 01:42:59.094928 7f72f9873800  5 osd.1 pg_epoch: 15 
> pg[0.0(unlocked)] enter Initial
> 2015-12-28 01:42:59.094980 7f72f9873800 20 osd.1 pg_epoch: 15 
> pg[0.0(unlocked)] enter NotTrimming
> 2015-12-28 01:42:59.094998 7f72f9873800 30 osd.1 pg_epoch: 15 pg[0.0( 
> DNE empty local-les=0 n=0 ec=0 les/c/f 0/0/0 0/0/0) [] r=0 lpr=0 
> crt=0'0 inactive NIBBLEW
> 2015-12-28 01:42:59.095186 7f72f9873800 20 read_log coll 0.0_head 
> log_oid 0///head

node2, DOWN:
> 2015-12-28 01:36:54.437246 7f4507957800  0 osd.0 11 load_pgs
> 2015-12-28 01:36:54.437267 7f4507957800 10 osd.0 11 load_pgs ignoring 
> unrecognized meta
> 2015-12-28 01:36:54.437274 7f4507957800  0 osd.0 11 load_pgs opened 0 
> pgs
> 2015-12-28 01:36:54.437278 7f4507957800 10 osd.0 11 
> build_past_intervals_parallel nothing to build
> 2015-12-28 01:36:54.437282 7f4507957800  2 osd.0 11 superblock: i am 
> osd.0
> 2015-12-28 01:36:54.437287 7f4507957800 10 osd.0 11 create_logger
> 2015-12-28 01:36:54.438157 7f4507957800 -1 osd.0 11 log_to_monitors 
> {default=true}
> 2015-12-28 01:36:54.449278 7f4507957800 10 osd.0 11 
> set_disk_tp_priority class  priority -1
> 2015-12-28 01:36:54.450813 7f44ddbff700 30 osd.0 11 heartbeat
> 2015-12-28 01:36:54.452558 7f44ddbff700 30 osd.0 11 heartbeat checking 
> stats
> 2015-12-28 01:36:54.452592 7f44ddbff700 20 osd.0 11 update_osd_stat 
> osd_stat(1056 MB used, 598 GB avail, 599 GB total, peers []/[] op hist 
> [])
> 2015-12-28 01:36:54.452611 7f44ddbff700  5 osd.0 11 heartbeat: 
> osd_stat(1056 MB used, 598 GB avail, 599 GB total, peers []/[] op hist 
> [])
> 2015-12-28 01:36:54.452618 7f44ddbff700 30 osd.0 11 heartbeat check
> 2015-12-28 01:36:54.452622 7f44ddbff700 30 osd.0 11 heartbeat lonely?
> 2015-12-28 01:36:54.452624 7f44ddbff700 30 osd.0 11 heartbeat done
> 2015-12-28 01:36:54.452627 7f44ddbff700 30 osd.0 11 heartbeat_entry 
> sleeping for 2.3
> 2015-12-28 01:36:54.452588 7f44da7fc700 10 osd.0 11 agent_entry start
> 2015-12-28 01:36:54.453338 7f44da7fc700 20 osd.0 11 agent_entry empty 
> queue

My ceph.conf looks like this:

[global]

fsid = b186d870-9c6d-4a8b-ac8a-e263f4c205da

ms_bind_ipv6 = true

public_network = ::2:1612::/64

mon initial members = 0

mon host = [::2:1612::50]:6789

auth cluster required = cephx

auth service required = cephx

auth client required = cephx

osd pool default size = 2

osd pool default min size = 1

osd journal size = 1024

osd mkfs type = xfs

osd mount options xfs = rw,inode64

osd crush chooseleaf type = 1

[mon.0]

host = node1

mon addr = [::2:1612::50]:6789

[mon.1]

host = node3

mon

Re: [ceph-users] My OSDs are down and not coming UP

[Ing. Martin Samek]

Hi,
all nodes are in one VLAN connected to one switch. Conectivity is OK, 
MTU 1500, can transfer data over netcat and mbuffer at 660 Mbps.


debug_ms, there is nothing interest:

/usr/bin/ceph-osd --debug_ms 100 -f -i 0 --pid-file /run/ceph/osd.0.pid -c 
/etc/ceph/ceph.conf

starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 
/var/lib/ceph/osd/ceph-0/journal

2015-12-29 00:18:05.878954 7fd9892e7800 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_force_aio to force use of aio 
anyway

2015-12-29 00:18:05.899633 7fd9892e7800 -1 osd.0 24 log_to_monitors 
{default=true}

Thanks,
Martin

Dne 29.12.2015 v 00:08 Somnath Roy napsal(a):

It could be a network issue..May be related to MTU (?)..Try running with 
debug_ms = 1 and see if you find anything..Also, try running command like 
'traceroute' and see if it is reporting any error..

Thanks & Regards
Somnath

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ing. 
Martin Samek
Sent: Monday, December 28, 2015 2:59 PM
To: Ceph Users
Subject: [ceph-users] My OSDs are down and not coming UP

Hi,

I'm a newbie in a Ceph world. I try setup my first testing Ceph cluster but 
unlikely my MON server running and talking each to other but my OSDs are still 
down and won't to come up. Actually only the one OSD running at the same node 
as a elected master is able to connect and come UP.

To be technical. I have 4 physical nodes living in pure IPv6 environment, 
running Gentoo Linux and Ceph 9.2. All nodes names are resolvable in DNS and 
also saved in hosts files.

I'm running OSD with command like this:

node1# /usr/bin/ceph-osd -f -i 1 --pid-file /run/ceph/osd.1.pid -c 
/etc/ceph/ceph.conf

single mon.0 is running also at node1, and OSD come up:

2015-12-28 23:37:27.931686 mon.0 [INF] osd.1 [2001:718:2:1612::50]:6800/23709 
boot

2015-12-28 23:37:27.932605 mon.0 [INF] osdmap e19: 2 osds: 1 up, 1 in

2015-12-28 23:37:27.933963 mon.0 [INF] pgmap v24: 64 pgs: 64 
stale+active+undersized+degraded; 0 bytes data, 1057 MB used, 598 GB / 599 GB 
avail

but running osd.0 at node2:

# /usr/bin/ceph-osd -f -i 0 --pid-file /run/ceph/osd.0.pid -c 
/etc/ceph/ceph.conf

did nothing, process is running, netstat shows opened connection from ceph-osd 
between node2 and node1. Here I'm lost. IPv6 connectivity is OK, DNS is OK, 
time is in sync, 1 mon running, 2 osds but only one UP.
What is missing?

ceph-osd in debug mode show differences at node1 and node2:

node1, UP:

2015-12-28 01:42:59.084371 7f72f9873800 20 osd.1 15  clearing temps in
0.3f_head pgid 0.3f
2015-12-28 01:42:59.084453 7f72f9873800  0 osd.1 15 load_pgs
2015-12-28 01:42:59.085248 7f72f9873800 10 osd.1 15 load_pgs ignoring
unrecognized meta
2015-12-28 01:42:59.094690 7f72f9873800 10 osd.1 15 pgid 0.0 coll
0.0_head
2015-12-28 01:42:59.094835 7f72f9873800 30 osd.1 0 get_map 15 -cached
2015-12-28 01:42:59.094848 7f72f9873800 10 osd.1 15 _open_lock_pg 0.0
2015-12-28 01:42:59.094857 7f72f9873800 10 osd.1 15 _get_pool 0
2015-12-28 01:42:59.094928 7f72f9873800  5 osd.1 pg_epoch: 15
pg[0.0(unlocked)] enter Initial
2015-12-28 01:42:59.094980 7f72f9873800 20 osd.1 pg_epoch: 15
pg[0.0(unlocked)] enter NotTrimming
2015-12-28 01:42:59.094998 7f72f9873800 30 osd.1 pg_epoch: 15 pg[0.0(
DNE empty local-les=0 n=0 ec=0 les/c/f 0/0/0 0/0/0) [] r=0 lpr=0
crt=0'0 inactive NIBBLEW
2015-12-28 01:42:59.095186 7f72f9873800 20 read_log coll 0.0_head
log_oid 0///head

node2, DOWN:

2015-12-28 01:36:54.437246 7f4507957800  0 osd.0 11 load_pgs
2015-12-28 01:36:54.437267 7f4507957800 10 osd.0 11 load_pgs ignoring
unrecognized meta
2015-12-28 01:36:54.437274 7f4507957800  0 osd.0 11 load_pgs opened 0
pgs
2015-12-28 01:36:54.437278 7f4507957800 10 osd.0 11
build_past_intervals_parallel nothing to build
2015-12-28 01:36:54.437282 7f4507957800  2 osd.0 11 superblock: i am
osd.0
2015-12-28 01:36:54.437287 7f4507957800 10 osd.0 11 create_logger
2015-12-28 01:36:54.438157 7f4507957800 -1 osd.0 11 log_to_monitors
{default=true}
2015-12-28 01:36:54.449278 7f4507957800 10 osd.0 11
set_disk_tp_priority class  priority -1
2015-12-28 01:36:54.450813 7f44ddbff700 30 osd.0 11 heartbeat
2015-12-28 01:36:54.452558 7f44ddbff700 30 osd.0 11 heartbeat checking
stats
2015-12-28 01:36:54.452592 7f44ddbff700 20 osd.0 11 update_osd_stat
osd_stat(1056 MB used, 598 GB avail, 599 GB total, peers []/[] op hist
[])
2015-12-28 01:36:54.452611 7f44ddbff700  5 osd.0 11 heartbeat:
osd_stat(1056 MB used, 598 GB avail, 599 GB total, peers []/[] op hist
[])
2015-12-28 01:36:54.452618 7f44ddbff700 30 osd.0 11 heartbeat check
2015-12-28 01:36:54.452622 7f44ddbff700 30 osd.0 11 heartbeat lonely?
2015-12-28 01:36:54.452624 7f44ddbff700 30 osd.0 11 heartbeat done
2015-12-28 01:36:54.452627 7f44ddbff700 30 osd.0 11 heartbeat_entry
sleeping for 2.3
2015-12-28 01:36:54.452588 7f44da7fc700 10 osd.0 11 agent_entry start
2015-12-28 01:36:54.453338 7f44da7fc700 20 osd.0 11 agent_entry empty
queue

My cep

Re: [ceph-users] Tuning ZFS + QEMU/KVM + Ceph RBD’s

[J David]

Yes, given the architectural design limitations of ZFS, there will
indeed always be performance consequences for using it in an
environment its creators never envisioned, like Ceph.  But ZFS offers
many advanced features not found on other filesystems, and for
production environments that depend on those features, it’s very
reasonable to still want them in an environment that happens to be
backed

Keep in mind also that FreeBSD and Solaris installers both create ZFS
filesystems (Solaris by default/only option, FreeBSD I’m not sure
about, it may be default in the most recent release), so this is not
just a question about ZFS on Linux.  ZFS is a *very* popular
filesystem in wide usage and is the *only* cross-platform filesystem
to offer the features it does.

So, until there’s another broadly-supported, ceph-aware,
production-quality filesystem that offers feature parity with it, the
question of how to get the best (or, if you prefer, least worst)
ZFS-on-ceph performance is worth asking.

In light of that, is it possible to do any better than just writing it
off as a lost cause?  This is work we’re absolutely willing to do, we
just don’t feel we have a good understanding of all the moving parts
involved, and how to measure and tune them all.  (And, most
importantly, how to measure the impact of the tuning.)

Thanks!

On Fri, Dec 25, 2015 at 9:06 PM, Tyler Bishop
 wrote:
> Due to the nature of distributed storage and a filesystem built to distribute 
> itself across sequential devices.. you're going to always have poor 
> performance.
>
> Are you unable to use XFS inside the vm?
>
>
> If you are not the intended recipient of this transmission you are notified 
> that disclosing, copying, distributing or taking any action in reliance on 
> the contents of this information is strictly prohibited.
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSD size and performance

[gjprabu]

Hi Team,



 We are using ceph with 3 osd and 2 replicas. Each osd size is 13TB 
and current data is reached to 2.5TB (each osd). Because of this huge size do 
we face any problem. 



OSD server configuration

Hard disk -- 13TB

RAM -- 96GB 

CPU -- 2 CPU with multi 8 core processor.




Regards

Prabu








___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ubuntu 14.04 or centos 7

[min fang]

Hi, I am looking for OS for my ceph cluster, from
http://docs.ceph.com/docs/master/start/os-recommendations/#infernalis-9-1-0,
there are two OS has been fully tested, centos 7 and ubuntu 14.04.  Which
one is better? thanks.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com