Re: [ceph-users] Hadoop and Ceph integration issues
Hi Noah,
updated the core-site.xml (bellow).
All the nodes have the same files. But the problem remains the same.
What is the value for ceph.auth.keyring? Is the path containing the
file ceph.mon.keyring?
Thanks,
Rolando
P.S.: I have the cephFS mounted locally, so the cluster is ok.
cluster d9ca74d0-d9f4-436d-92de-762af67c6534
health HEALTH_OK
monmap e1: 9 mons at
{hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
election epoch 6, quorum 0,1,2,3,4,5,6,7,8
hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
osdmap e30: 9 osds: 9 up, 9 in
pgmap v2457: 192 pgs: 192 active+clean; 10408 bytes data, 44312 MB
used, 168 GB / 221 GB avail
mdsmap e4: 1/1/1 up {0=hyrax1=up:active}
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyfile/name
value/hyrax/hadoop-ceph/ceph/admin.secret/value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.mon.keyring/value
/property
On Mon, Sep 23, 2013 at 11:42 AM, Noah Watkins noah.watk...@inktank.com wrote:
property
nameceph.root.dir/name
value/mnt/mycephfs/value
/property
This is probably causing the issue. Is this meant to be a local mount
point? The 'ceph.root.dir' property specifies the root directory
/inside/ CephFS, and the Hadoop implementation doesn't require a local
CephFS mount--it uses a client library to interact with the file
system.
The default value for this property is /, so you can probably just
remove this from your config file unless your CephFS directory
structure is carved up in a special way.
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.auth.keyfile/name
value/hyrax/hadoop-ceph/ceph/admin.secret/value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.mon.keyring/value
/property
These files will need to be available locally on every node Hadoop
runs on. I think the error below will occur after these are loaded, so
it probably isn't your issue, though I don't recall exactly at which
point different configuration files are loaded.
property
namefs.hdfs.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
I don't think this is part of the problem you are seeing, but this
'fs.hdfs.impl' property should probably be removed. We aren't
overriding HDFS, just replacing it.
property
nameceph.mon.address/name
valuehyrax1:6789/value
/property
This was already specified in your 'fs.default.name' property. I don't
think that duplicating it is an issue, but I should probably update
the documentation to make it clear that the monitor only needs to be
listed once.
Thanks!
Noah
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Hadoop and Ceph integration issues
Hi Noah,
I enabled the debugging and got:
2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2
I have the ceph.client.admin.keyring file in /etc/ceph and I tried
with and without the
parameter in core-site.xml. Unfortunately without success:(
Thanks,
Rolando
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring/value
/property
On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins noah.watk...@inktank.com wrote:
Shoot, I thought I had it figured out :)
There is a default admin user created when you first create your
cluster. After a typical install via ceph-deploy, there should be a
file called 'ceph.client.admin.keyring', usually sibling to ceph.conf.
If this is in a standard location (e.g. /etc/ceph) you shouldn't need
the keyring option, otherwise point 'ceph.auth.keyring' at that
keyring file. You shouldn't need both the keyring and the keyfile
options set, but it just depends on how your authentication / users
are all setup.
The easiest thing to do if that doesn't solve your problem is probably
to turn on logging so we can see what is blowing up.
In your ceph.conf you can add 'debug client = 20' and 'debug
javaclient = 20' to the client section. You may also need to set the
log file 'log file = /path/...'. You don't need to do this on all your
nodes, just one node where you get the failure.
- Noah
Thanks,
Rolando
P.S.: I have the cephFS mounted locally, so the cluster is ok.
cluster d9ca74d0-d9f4-436d-92de-762af67c6534
health HEALTH_OK
monmap e1: 9 mons at
{hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
election epoch 6, quorum 0,1,2,3,4,5,6,7,8
hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
osdmap e30: 9 osds: 9 up, 9 in
pgmap v2457: 192 pgs: 192 active+clean; 10408 bytes data, 44312 MB
used, 168 GB / 221 GB avail
mdsmap e4: 1/1/1 up {0=hyrax1=up:active}
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyfile/name
value/hyrax/hadoop-ceph/ceph/admin.secret/value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.mon.keyring/value
/property
On Mon, Sep 23, 2013 at 11:42 AM, Noah Watkins noah.watk...@inktank.com
wrote:
property
nameceph.root.dir/name
value/mnt/mycephfs/value
/property
This is probably causing the issue. Is this meant to be a local mount
point? The 'ceph.root.dir' property specifies the root directory
/inside/ CephFS, and the Hadoop implementation doesn't require a local
CephFS mount--it uses a client library to interact with the file
system.
The default value for this property is /, so you can probably just
remove this from your config file unless your CephFS directory
structure is carved up in a special way.
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.auth.keyfile/name
value/hyrax/hadoop-ceph/ceph/admin.secret/value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.mon.keyring/value
/property
These files will need to be available locally on every node Hadoop
runs on. I think the error below will occur after these are loaded, so
it probably isn't your issue, though I don't recall exactly at which
point different configuration files are loaded.
property
namefs.hdfs.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
I don't think this is part of the problem you are seeing, but this
'fs.hdfs.impl' property should probably be removed. We aren't
overriding HDFS, just replacing it.
property
nameceph.mon.address/name
valuehyrax1:6789/value
/property
This was already specified in your 'fs.default.name' property. I don't
think that duplicating it is an issue, but I should probably update
the documentation to make it
Re: [ceph-users] Hadoop and Ceph integration issues
bin/hadoop fs -ls
Bad connection to FS. command aborted. exception:
(no other information is thrown)
ceph log:
2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2
On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins noah.watk...@inktank.com wrote:
What happens when you run `bin/hadoop fs -ls` ? This is entirely
local, and a bit simpler and easier to grok.
On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
I am trying to start hadoop using bin/start-mapred.sh.
In the HADOOP_HOME/lib, I have:
lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so
(the first I downloaded from
http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied
from my system (after installing the ubuntu package for the ceph java
client))
I added to conf/hadoop-env.sh:
export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib
I confirmed using bin/hadoop classpath that both jar are in the classpath.
On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins noah.watk...@inktank.com
wrote:
How are you invoking Hadoop? Also, I forgot to ask, are you using the
wrappers located in github.com/ceph/hadoop-common (or the jar linked
to on http://ceph.com/docs/master/cephfs/hadoop/)?
On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
Hi Noah,
I enabled the debugging and got:
2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2
I have the ceph.client.admin.keyring file in /etc/ceph and I tried
with and without the
parameter in core-site.xml. Unfortunately without success:(
Thanks,
Rolando
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring/value
/property
On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins noah.watk...@inktank.com
wrote:
Shoot, I thought I had it figured out :)
There is a default admin user created when you first create your
cluster. After a typical install via ceph-deploy, there should be a
file called 'ceph.client.admin.keyring', usually sibling to ceph.conf.
If this is in a standard location (e.g. /etc/ceph) you shouldn't need
the keyring option, otherwise point 'ceph.auth.keyring' at that
keyring file. You shouldn't need both the keyring and the keyfile
options set, but it just depends on how your authentication / users
are all setup.
The easiest thing to do if that doesn't solve your problem is probably
to turn on logging so we can see what is blowing up.
In your ceph.conf you can add 'debug client = 20' and 'debug
javaclient = 20' to the client section. You may also need to set the
log file 'log file = /path/...'. You don't need to do this on all your
nodes, just one node where you get the failure.
- Noah
Thanks,
Rolando
P.S.: I have the cephFS mounted locally, so the cluster is ok.
cluster d9ca74d0-d9f4-436d-92de-762af67c6534
health HEALTH_OK
monmap e1: 9 mons at
{hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
election epoch 6, quorum 0,1,2,3,4,5,6,7,8
hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
osdmap e30: 9 osds: 9 up, 9 in
pgmap v2457: 192 pgs: 192 active+clean; 10408 bytes data, 44312 MB
used, 168 GB / 221 GB avail
mdsmap e4: 1/1/1 up {0=hyrax1=up:active}
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyfile/name
value/hyrax/hadoop-ceph/ceph/admin.secret/value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.mon.keyring/value
/property
On Mon, Sep 23, 2013 at 11:42 AM, Noah Watkins
noah.watk...@inktank.com wrote:
property
nameceph.root.dir/name
value/mnt/mycephfs/value
/property
This is probably causing the issue. Is this meant
Re: [ceph-users] Hadoop and Ceph integration issues
2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0
2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: /
2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2
2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0
2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: /
On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins noah.watk...@inktank.com wrote:
In the log file that you showing, do you see where the keyring file is
being set by Hadoop? You can find it by grepping for: jni: conf_set
On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
bin/hadoop fs -ls
Bad connection to FS. command aborted. exception:
(no other information is thrown)
ceph log:
2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2
On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins noah.watk...@inktank.com
wrote:
What happens when you run `bin/hadoop fs -ls` ? This is entirely
local, and a bit simpler and easier to grok.
On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
I am trying to start hadoop using bin/start-mapred.sh.
In the HADOOP_HOME/lib, I have:
lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so
(the first I downloaded from
http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied
from my system (after installing the ubuntu package for the ceph java
client))
I added to conf/hadoop-env.sh:
export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib
I confirmed using bin/hadoop classpath that both jar are in the classpath.
On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins noah.watk...@inktank.com
wrote:
How are you invoking Hadoop? Also, I forgot to ask, are you using the
wrappers located in github.com/ceph/hadoop-common (or the jar linked
to on http://ceph.com/docs/master/cephfs/hadoop/)?
On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
Hi Noah,
I enabled the debugging and got:
2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0
max 0
2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2
I have the ceph.client.admin.keyring file in /etc/ceph and I tried
with and without the
parameter in core-site.xml. Unfortunately without success:(
Thanks,
Rolando
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring/value
/property
On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins noah.watk...@inktank.com
wrote:
Shoot, I thought I had it figured out :)
There is a default admin user created when you first create your
cluster. After a typical install via ceph-deploy, there should be a
file called 'ceph.client.admin.keyring', usually sibling to ceph.conf.
If this is in a standard location (e.g. /etc/ceph) you shouldn't need
the keyring option, otherwise point 'ceph.auth.keyring' at that
keyring file. You shouldn't need both the keyring and the keyfile
options set, but it just depends on how your authentication / users
are all setup.
The easiest thing to do if that doesn't solve your problem is probably
to turn on logging so we can see what is blowing up.
In your ceph.conf you can add 'debug client = 20' and 'debug
javaclient = 20' to the client section. You may also need to set the
log file 'log file = /path/...'. You don't need to do this on all your
nodes, just one node where you get the failure.
- Noah
Thanks,
Rolando
P.S.: I have the cephFS mounted locally, so the cluster is ok.
cluster d9ca74d0-d9f4-436d-92de-762af67c6534
health HEALTH_OK
monmap e1: 9 mons at
{hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
election epoch 6, quorum 0,1,2,3,4,5,6,7,8
hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
osdmap e30: 9 osds: 9 up, 9 in
pgmap v2457: 192 pgs: 192
Re: [ceph-users] Hadoop and Ceph integration issues
My bad, I associated conf_read_file with conf_set.
No, it does not appear in the logs.
On Mon, Sep 23, 2013 at 4:20 PM, Noah Watkins noah.watk...@inktank.com wrote:
I'm not sure what you grepped for. Does this output mean that the
string conf_set didn't show up in the log?
On Mon, Sep 23, 2013 at 12:52 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0
2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: /
2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2
2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0
2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: /
On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins noah.watk...@inktank.com
wrote:
In the log file that you showing, do you see where the keyring file is
being set by Hadoop? You can find it by grepping for: jni: conf_set
On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
bin/hadoop fs -ls
Bad connection to FS. command aborted. exception:
(no other information is thrown)
ceph log:
2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2
On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins noah.watk...@inktank.com
wrote:
What happens when you run `bin/hadoop fs -ls` ? This is entirely
local, and a bit simpler and easier to grok.
On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
I am trying to start hadoop using bin/start-mapred.sh.
In the HADOOP_HOME/lib, I have:
lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so
(the first I downloaded from
http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied
from my system (after installing the ubuntu package for the ceph java
client))
I added to conf/hadoop-env.sh:
export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib
I confirmed using bin/hadoop classpath that both jar are in the
classpath.
On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins noah.watk...@inktank.com
wrote:
How are you invoking Hadoop? Also, I forgot to ask, are you using the
wrappers located in github.com/ceph/hadoop-common (or the jar linked
to on http://ceph.com/docs/master/cephfs/hadoop/)?
On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins
rolando.mart...@gmail.com wrote:
Hi Noah,
I enabled the debugging and got:
2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0
max 0
2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2
I have the ceph.client.admin.keyring file in /etc/ceph and I tried
with and without the
parameter in core-site.xml. Unfortunately without success:(
Thanks,
Rolando
property
namefs.ceph.impl/name
valueorg.apache.hadoop.fs.ceph.CephFileSystem/value
/property
property
namefs.default.name/name
valueceph://hyrax1:6789//value
/property
property
nameceph.conf.file/name
value/hyrax/hadoop-ceph/ceph/ceph.conf/value
/property
property
nameceph.root.dir/name
value//value
/property
property
nameceph.auth.keyring/name
value/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring/value
/property
On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins
noah.watk...@inktank.com wrote:
Shoot, I thought I had it figured out :)
There is a default admin user created when you first create your
cluster. After a typical install via ceph-deploy, there should be a
file called 'ceph.client.admin.keyring', usually sibling to ceph.conf.
If this is in a standard location (e.g. /etc/ceph) you shouldn't need
the keyring option, otherwise point 'ceph.auth.keyring' at that
keyring file. You shouldn't need both the keyring and the keyfile
options set, but it just depends on how your authentication / users
are all setup.
The easiest thing to do if that doesn't solve your problem is probably
to turn on logging so we can see what is blowing up.
In your ceph.conf you can add 'debug client = 20' and 'debug
javaclient = 20' to the client section. You may also need to set the
log file 'log file = /path/...'. You don't need to do this on all your
nodes, just one node where you get the failure.
- Noah
Thanks,
Rolando
P.S.: I have the cephFS mounted locally, so the cluster is ok.
cluster d9ca74d0-d9f4-436d-92de-762af67c6534
health HEALTH_OK
monmap e1: 9 mons at
{hyrax1=10.10.10.10:6789/0,hyrax2
Re: [ceph-users] Hadoop and Ceph integration issues
I tried to compile it, but the compilation failed. The error log starts with: compile-core-classes: [taskdef] 2013-09-23 20:59:25,540 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [javac] /home/ubuntu/Projects/hadoop-common/build.xml:487: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 440 source files to /home/ubuntu/Projects/hadoop-common/build/classes [javac] /home/ubuntu/Projects/hadoop-common/src/core/org/apache/hadoop/fs/ceph/CephFS.java:31: package com.ceph.fs does not exist [javac] import com.ceph.fs.CephStat; [javac] ^ What are the dependencies that I need to have installed? On Mon, Sep 23, 2013 at 4:32 PM, Noah Watkins noah.watk...@inktank.com wrote: Ok thanks. That narrows things down a lot. It seems like the keyring property is not being recognized, and I don't see so I'm wondering if it is possible that the jar file is out of date and doesn't include these configuration features. If you clone http://github.com/ceph/hadoop-common/ and checkout the cephfs/branch-1.0 branch, you can run 'ant cephfs' to make a fresh jar file. On Mon, Sep 23, 2013 at 1:22 PM, Rolando Martins rolando.mart...@gmail.com wrote: My bad, I associated conf_read_file with conf_set. No, it does not appear in the logs. On Mon, Sep 23, 2013 at 4:20 PM, Noah Watkins noah.watk...@inktank.com wrote: I'm not sure what you grepped for. Does this output mean that the string conf_set didn't show up in the log? On Mon, Sep 23, 2013 at 12:52 PM, Rolando Martins rolando.mart...@gmail.com wrote: 2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0 2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: / 2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2 2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0 2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: / On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins noah.watk...@inktank.com wrote: In the log file that you showing, do you see where the keyring file is being set by Hadoop? You can find it by grepping for: jni: conf_set On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins rolando.mart...@gmail.com wrote: bin/hadoop fs -ls Bad connection to FS. command aborted. exception: (no other information is thrown) ceph log: 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins noah.watk...@inktank.com wrote: What happens when you run `bin/hadoop fs -ls` ? This is entirely local, and a bit simpler and easier to grok. On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins rolando.mart...@gmail.com wrote: I am trying to start hadoop using bin/start-mapred.sh. In the HADOOP_HOME/lib, I have: lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so (the first I downloaded from http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied from my system (after installing the ubuntu package for the ceph java client)) I added to conf/hadoop-env.sh: export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib I confirmed using bin/hadoop classpath that both jar are in the classpath. On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins noah.watk...@inktank.com wrote: How are you invoking Hadoop? Also, I forgot to ask, are you using the wrappers located in github.com/ceph/hadoop-common (or the jar linked to on http://ceph.com/docs/master/cephfs/hadoop/)? On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins rolando.mart...@gmail.com wrote: Hi Noah, I enabled the debugging and got: 2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2 I have the ceph.client.admin.keyring file in /etc/ceph and I tried with and without the parameter in core-site.xml. Unfortunately without success:( Thanks, Rolando property namefs.ceph.impl/name valueorg.apache.hadoop.fs.ceph.CephFileSystem/value /property property namefs.default.name/name valueceph://hyrax1:6789//value /property property nameceph.conf.file/name value/hyrax/hadoop-ceph/ceph/ceph.conf/value /property property nameceph.root.dir/name
Re: [ceph-users] Hadoop and Ceph integration issues
Thanks Noah! It worked! I managed to run the wordcount example! Can you remove the jar that is posted online? It is misleading... Thanks! Rolando On Mon, Sep 23, 2013 at 5:07 PM, Noah Watkins noah.watk...@inktank.com wrote: You need to stick the CephFS jar files in the hadoop lib folder. On Mon, Sep 23, 2013 at 2:02 PM, Rolando Martins rolando.mart...@gmail.com wrote: I tried to compile it, but the compilation failed. The error log starts with: compile-core-classes: [taskdef] 2013-09-23 20:59:25,540 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [javac] /home/ubuntu/Projects/hadoop-common/build.xml:487: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 440 source files to /home/ubuntu/Projects/hadoop-common/build/classes [javac] /home/ubuntu/Projects/hadoop-common/src/core/org/apache/hadoop/fs/ceph/CephFS.java:31: package com.ceph.fs does not exist [javac] import com.ceph.fs.CephStat; [javac] ^ What are the dependencies that I need to have installed? On Mon, Sep 23, 2013 at 4:32 PM, Noah Watkins noah.watk...@inktank.com wrote: Ok thanks. That narrows things down a lot. It seems like the keyring property is not being recognized, and I don't see so I'm wondering if it is possible that the jar file is out of date and doesn't include these configuration features. If you clone http://github.com/ceph/hadoop-common/ and checkout the cephfs/branch-1.0 branch, you can run 'ant cephfs' to make a fresh jar file. On Mon, Sep 23, 2013 at 1:22 PM, Rolando Martins rolando.mart...@gmail.com wrote: My bad, I associated conf_read_file with conf_set. No, it does not appear in the logs. On Mon, Sep 23, 2013 at 4:20 PM, Noah Watkins noah.watk...@inktank.com wrote: I'm not sure what you grepped for. Does this output mean that the string conf_set didn't show up in the log? On Mon, Sep 23, 2013 at 12:52 PM, Rolando Martins rolando.mart...@gmail.com wrote: 2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0 2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: / 2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2 2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0 2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: / On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins noah.watk...@inktank.com wrote: In the log file that you showing, do you see where the keyring file is being set by Hadoop? You can find it by grepping for: jni: conf_set On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins rolando.mart...@gmail.com wrote: bin/hadoop fs -ls Bad connection to FS. command aborted. exception: (no other information is thrown) ceph log: 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins noah.watk...@inktank.com wrote: What happens when you run `bin/hadoop fs -ls` ? This is entirely local, and a bit simpler and easier to grok. On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins rolando.mart...@gmail.com wrote: I am trying to start hadoop using bin/start-mapred.sh. In the HADOOP_HOME/lib, I have: lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so (the first I downloaded from http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied from my system (after installing the ubuntu package for the ceph java client)) I added to conf/hadoop-env.sh: export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib I confirmed using bin/hadoop classpath that both jar are in the classpath. On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins noah.watk...@inktank.com wrote: How are you invoking Hadoop? Also, I forgot to ask, are you using the wrappers located in github.com/ceph/hadoop-common (or the jar linked to on http://ceph.com/docs/master/cephfs/hadoop/)? On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins rolando.mart...@gmail.com wrote: Hi Noah, I enabled the debugging and got: 2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2 I have the ceph.client.admin.keyring file in /etc/ceph and I tried with and without the parameter in core-site.xml
