Re: [ceph-users] Hadoop and Ceph integration issues
Thanks Noah! It worked! I managed to run the wordcount example! Can you remove the jar that is posted online? It is misleading... Thanks! Rolando On Mon, Sep 23, 2013 at 5:07 PM, Noah Watkins wrote: > You need to stick the CephFS jar files in the hadoop lib folder. > > On Mon, Sep 23, 2013 at 2:02 PM, Rolando Martins > wrote: >> I tried to compile it, but the compilation failed. >> The error log starts with: >> compile-core-classes: >> [taskdef] 2013-09-23 20:59:25,540 INFO mortbay.log >> (Slf4jLog.java:info(67)) - Logging to >> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> org.mortbay.log.Slf4jLog >> [javac] /home/ubuntu/Projects/hadoop-common/build.xml:487: >> warning: 'includeantruntime' was not set, defaulting to >> build.sysclasspath=last; set to false for repeatable builds >> [javac] Compiling 440 source files to >> /home/ubuntu/Projects/hadoop-common/build/classes >> [javac] >> /home/ubuntu/Projects/hadoop-common/src/core/org/apache/hadoop/fs/ceph/CephFS.java:31: >> package com.ceph.fs does not exist >> [javac] import com.ceph.fs.CephStat; >> [javac] ^ >> >> What are the dependencies that I need to have installed? >> >> >> On Mon, Sep 23, 2013 at 4:32 PM, Noah Watkins >> wrote: >>> Ok thanks. That narrows things down a lot. It seems like the keyring >>> property is not being recognized, and I don't see so I'm wondering if >>> it is possible that the jar file is out of date and doesn't include >>> these configuration features. >>> >>> If you clone http://github.com/ceph/hadoop-common/ and checkout the >>> cephfs/branch-1.0 branch, you can run 'ant cephfs' to make a fresh jar >>> file. >>> >>> On Mon, Sep 23, 2013 at 1:22 PM, Rolando Martins >>> wrote: >>>> My bad, I associated conf_read_file with conf_set. >>>> No, it does not appear in the logs. >>>> >>>> On Mon, Sep 23, 2013 at 4:20 PM, Noah Watkins >>>> wrote: >>>>> I'm not sure what you grepped for. Does this output mean that the >>>>> string "conf_set" didn't show up in the log? >>>>> >>>>> On Mon, Sep 23, 2013 at 12:52 PM, Rolando Martins >>>>> wrote: >>>>>> 2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret >>>>>> 0 >>>>>> 2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: / >>>>>> 2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR: >>>>>> missing keyring, cannot use cephx for authentication >>>>>> 2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 >>>>>> max 0 >>>>>> 2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>>>> 2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret >>>>>> 0 >>>>>> 2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: / >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins >>>>>> wrote: >>>>>>> In the log file that you showing, do you see where the keyring file is >>>>>>> being set by Hadoop? You can find it by grepping for: "jni: conf_set" >>>>>>> >>>>>>> On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins >>>>>>> wrote: >>>>>>>> bin/hadoop fs -ls >>>>>>>> >>>>>>>> Bad connection to FS. command aborted. exception: >>>>>>>> >>>>>>>> (no other information is thrown) >>>>>>>> >>>>>>>> ceph log: >>>>>>>> 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: >>>>>>>> missing keyring, cannot use cephx for authentication >>>>>>>> 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 >>>>>>>> max 0 >>>>>>>> 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>>>>>> >>>>>>>> On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins >>>>>>>> wrote: >>>>>>>>> What happens when you run `bin/hadoop fs -ls` ? This is entirely >>>>>>>>> l
Re: [ceph-users] Hadoop and Ceph integration issues
I tried to compile it, but the compilation failed. The error log starts with: compile-core-classes: [taskdef] 2013-09-23 20:59:25,540 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [javac] /home/ubuntu/Projects/hadoop-common/build.xml:487: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 440 source files to /home/ubuntu/Projects/hadoop-common/build/classes [javac] /home/ubuntu/Projects/hadoop-common/src/core/org/apache/hadoop/fs/ceph/CephFS.java:31: package com.ceph.fs does not exist [javac] import com.ceph.fs.CephStat; [javac] ^ What are the dependencies that I need to have installed? On Mon, Sep 23, 2013 at 4:32 PM, Noah Watkins wrote: > Ok thanks. That narrows things down a lot. It seems like the keyring > property is not being recognized, and I don't see so I'm wondering if > it is possible that the jar file is out of date and doesn't include > these configuration features. > > If you clone http://github.com/ceph/hadoop-common/ and checkout the > cephfs/branch-1.0 branch, you can run 'ant cephfs' to make a fresh jar > file. > > On Mon, Sep 23, 2013 at 1:22 PM, Rolando Martins > wrote: >> My bad, I associated conf_read_file with conf_set. >> No, it does not appear in the logs. >> >> On Mon, Sep 23, 2013 at 4:20 PM, Noah Watkins >> wrote: >>> I'm not sure what you grepped for. Does this output mean that the >>> string "conf_set" didn't show up in the log? >>> >>> On Mon, Sep 23, 2013 at 12:52 PM, Rolando Martins >>> wrote: >>>> 2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0 >>>> 2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: / >>>> 2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR: >>>> missing keyring, cannot use cephx for authentication >>>> 2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max >>>> 0 >>>> 2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>> 2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0 >>>> 2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: / >>>> .... >>>> >>>> >>>> On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins >>>> wrote: >>>>> In the log file that you showing, do you see where the keyring file is >>>>> being set by Hadoop? You can find it by grepping for: "jni: conf_set" >>>>> >>>>> On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins >>>>> wrote: >>>>>> bin/hadoop fs -ls >>>>>> >>>>>> Bad connection to FS. command aborted. exception: >>>>>> >>>>>> (no other information is thrown) >>>>>> >>>>>> ceph log: >>>>>> 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: >>>>>> missing keyring, cannot use cephx for authentication >>>>>> 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 >>>>>> max 0 >>>>>> 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>>>> >>>>>> On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins >>>>>> wrote: >>>>>>> What happens when you run `bin/hadoop fs -ls` ? This is entirely >>>>>>> local, and a bit simpler and easier to grok. >>>>>>> >>>>>>> On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins >>>>>>> wrote: >>>>>>>> I am trying to start hadoop using bin/start-mapred.sh. >>>>>>>> In the HADOOP_HOME/lib, I have: >>>>>>>> lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so >>>>>>>> (the first I downloaded from >>>>>>>> http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied >>>>>>>> from my system (after installing the ubuntu package for the ceph java >>>>>>>> client)) >>>>>>>> >>>>>>>> I added to conf/hadoop-env.sh: >>>>>>>> export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib >>>>>>>> >>>>>>>> I confirmed using bin/hadoop classpath that both ja
Re: [ceph-users] Hadoop and Ceph integration issues
My bad, I associated conf_read_file with conf_set. No, it does not appear in the logs. On Mon, Sep 23, 2013 at 4:20 PM, Noah Watkins wrote: > I'm not sure what you grepped for. Does this output mean that the > string "conf_set" didn't show up in the log? > > On Mon, Sep 23, 2013 at 12:52 PM, Rolando Martins > wrote: >> 2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0 >> 2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: / >> 2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR: >> missing keyring, cannot use cephx for authentication >> 2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 >> 2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >> 2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0 >> 2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: / >> >> >> >> On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins >> wrote: >>> In the log file that you showing, do you see where the keyring file is >>> being set by Hadoop? You can find it by grepping for: "jni: conf_set" >>> >>> On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins >>> wrote: >>>> bin/hadoop fs -ls >>>> >>>> Bad connection to FS. command aborted. exception: >>>> >>>> (no other information is thrown) >>>> >>>> ceph log: >>>> 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: >>>> missing keyring, cannot use cephx for authentication >>>> 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max >>>> 0 >>>> 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>> >>>> On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins >>>> wrote: >>>>> What happens when you run `bin/hadoop fs -ls` ? This is entirely >>>>> local, and a bit simpler and easier to grok. >>>>> >>>>> On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins >>>>> wrote: >>>>>> I am trying to start hadoop using bin/start-mapred.sh. >>>>>> In the HADOOP_HOME/lib, I have: >>>>>> lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so >>>>>> (the first I downloaded from >>>>>> http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied >>>>>> from my system (after installing the ubuntu package for the ceph java >>>>>> client)) >>>>>> >>>>>> I added to conf/hadoop-env.sh: >>>>>> export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib >>>>>> >>>>>> I confirmed using bin/hadoop classpath that both jar are in the >>>>>> classpath. >>>>>> >>>>>> On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins >>>>>> wrote: >>>>>>> How are you invoking Hadoop? Also, I forgot to ask, are you using the >>>>>>> wrappers located in github.com/ceph/hadoop-common (or the jar linked >>>>>>> to on http://ceph.com/docs/master/cephfs/hadoop/)? >>>>>>> >>>>>>> On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins >>>>>>> wrote: >>>>>>>> Hi Noah, >>>>>>>> I enabled the debugging and got: >>>>>>>> >>>>>>>> 2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR: >>>>>>>> missing keyring, cannot use cephx for authentication >>>>>>>> 2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 >>>>>>>> max 0 >>>>>>>> 2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>>>>>> >>>>>>>> I have the ceph.client.admin.keyring file in /etc/ceph and I tried >>>>>>>> with and without the >>>>>>>> parameter in core-site.xml. Unfortunately without success:( >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Rolando >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> fs.ceph.impl >>>>>>>> org.apache.hadoop.fs.ceph.CephFileSystem >>>>>>>> >>>>>>>> >>>>>>>> >>
Re: [ceph-users] Hadoop and Ceph integration issues
2013-09-23 19:42:22.515836 7f0b58de7700 10 jni: conf_read_file: exit ret 0 2013-09-23 19:42:22.515893 7f0b58de7700 10 jni: ceph_mount: / 2013-09-23 19:42:22.516643 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 19:42:22.516969 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 19:42:22.517210 7f0b58de7700 10 jni: ceph_mount: exit ret -2 2013-09-23 19:42:23.520569 7f0b58de7700 10 jni: conf_read_file: exit ret 0 2013-09-23 19:42:23.520601 7f0b58de7700 10 jni: ceph_mount: / On Mon, Sep 23, 2013 at 3:47 PM, Noah Watkins wrote: > In the log file that you showing, do you see where the keyring file is > being set by Hadoop? You can find it by grepping for: "jni: conf_set" > > On Mon, Sep 23, 2013 at 12:43 PM, Rolando Martins > wrote: >> bin/hadoop fs -ls >> >> Bad connection to FS. command aborted. exception: >> >> (no other information is thrown) >> >> ceph log: >> 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: >> missing keyring, cannot use cephx for authentication >> 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 >> 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >> >> On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins >> wrote: >>> What happens when you run `bin/hadoop fs -ls` ? This is entirely >>> local, and a bit simpler and easier to grok. >>> >>> On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins >>> wrote: >>>> I am trying to start hadoop using bin/start-mapred.sh. >>>> In the HADOOP_HOME/lib, I have: >>>> lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so >>>> (the first I downloaded from >>>> http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied >>>> from my system (after installing the ubuntu package for the ceph java >>>> client)) >>>> >>>> I added to conf/hadoop-env.sh: >>>> export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib >>>> >>>> I confirmed using bin/hadoop classpath that both jar are in the classpath. >>>> >>>> On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins >>>> wrote: >>>>> How are you invoking Hadoop? Also, I forgot to ask, are you using the >>>>> wrappers located in github.com/ceph/hadoop-common (or the jar linked >>>>> to on http://ceph.com/docs/master/cephfs/hadoop/)? >>>>> >>>>> On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins >>>>> wrote: >>>>>> Hi Noah, >>>>>> I enabled the debugging and got: >>>>>> >>>>>> 2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR: >>>>>> missing keyring, cannot use cephx for authentication >>>>>> 2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 >>>>>> max 0 >>>>>> 2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>>>> >>>>>> I have the ceph.client.admin.keyring file in /etc/ceph and I tried >>>>>> with and without the >>>>>> parameter in core-site.xml. Unfortunately without success:( >>>>>> >>>>>> Thanks, >>>>>> Rolando >>>>>> >>>>>> >>>>>> >>>>>> fs.ceph.impl >>>>>> org.apache.hadoop.fs.ceph.CephFileSystem >>>>>> >>>>>> >>>>>> >>>>>> fs.default.name >>>>>> ceph://hyrax1:6789/ >>>>>> >>>>>> >>>>>> >>>>>> ceph.conf.file >>>>>> /hyrax/hadoop-ceph/ceph/ceph.conf >>>>>> >>>>>> >>>>>> >>>>>> ceph.root.dir >>>>>> / >>>>>> >>>>>> >>>>>> ceph.auth.keyring >>>>>>/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring >>>>>> >>>>>> >>>>>> On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins >>>>>> wrote: >>>>>>> Shoot, I thought I had it figured out :) >>>>>>> >>>>>>> There is a default admin user created when you first create your >>>>>>> cluster. After a typical install via ceph-deploy, there sho
Re: [ceph-users] Hadoop and Ceph integration issues
bin/hadoop fs -ls Bad connection to FS. command aborted. exception: (no other information is thrown) ceph log: 2013-09-23 19:42:27.545402 7f0b58de7700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2013-09-23 19:42:27.545619 7f0b58de7700 20 client.-1 trim_cache size 0 max 0 2013-09-23 19:42:27.545733 7f0b58de7700 10 jni: ceph_mount: exit ret -2 On Mon, Sep 23, 2013 at 3:39 PM, Noah Watkins wrote: > What happens when you run `bin/hadoop fs -ls` ? This is entirely > local, and a bit simpler and easier to grok. > > On Mon, Sep 23, 2013 at 12:23 PM, Rolando Martins > wrote: >> I am trying to start hadoop using bin/start-mapred.sh. >> In the HADOOP_HOME/lib, I have: >> lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so >> (the first I downloaded from >> http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied >> from my system (after installing the ubuntu package for the ceph java >> client)) >> >> I added to conf/hadoop-env.sh: >> export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib >> >> I confirmed using bin/hadoop classpath that both jar are in the classpath. >> >> On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins >> wrote: >>> How are you invoking Hadoop? Also, I forgot to ask, are you using the >>> wrappers located in github.com/ceph/hadoop-common (or the jar linked >>> to on http://ceph.com/docs/master/cephfs/hadoop/)? >>> >>> On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins >>> wrote: >>>> Hi Noah, >>>> I enabled the debugging and got: >>>> >>>> 2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR: >>>> missing keyring, cannot use cephx for authentication >>>> 2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max >>>> 0 >>>> 2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2 >>>> >>>> I have the ceph.client.admin.keyring file in /etc/ceph and I tried >>>> with and without the >>>> parameter in core-site.xml. Unfortunately without success:( >>>> >>>> Thanks, >>>> Rolando >>>> >>>> >>>> >>>> fs.ceph.impl >>>> org.apache.hadoop.fs.ceph.CephFileSystem >>>> >>>> >>>> >>>> fs.default.name >>>> ceph://hyrax1:6789/ >>>> >>>> >>>> >>>> ceph.conf.file >>>> /hyrax/hadoop-ceph/ceph/ceph.conf >>>> >>>> >>>> >>>> ceph.root.dir >>>> / >>>> >>>> >>>> ceph.auth.keyring >>>>/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring >>>> >>>> >>>> On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins >>>> wrote: >>>>> Shoot, I thought I had it figured out :) >>>>> >>>>> There is a default admin user created when you first create your >>>>> cluster. After a typical install via ceph-deploy, there should be a >>>>> file called 'ceph.client.admin.keyring', usually sibling to ceph.conf. >>>>> If this is in a standard location (e.g. /etc/ceph) you shouldn't need >>>>> the keyring option, otherwise point 'ceph.auth.keyring' at that >>>>> keyring file. You shouldn't need both the keyring and the keyfile >>>>> options set, but it just depends on how your authentication / users >>>>> are all setup. >>>>> >>>>> The easiest thing to do if that doesn't solve your problem is probably >>>>> to turn on logging so we can see what is blowing up. >>>>> >>>>> In your ceph.conf you can add 'debug client = 20' and 'debug >>>>> javaclient = 20' to the client section. You may also need to set the >>>>> log file 'log file = /path/...'. You don't need to do this on all your >>>>> nodes, just one node where you get the failure. >>>>> >>>>> - Noah >>>>> >>>>>> Thanks, >>>>>> Rolando >>>>>> >>>>>> P.S.: I have the cephFS mounted locally, so the cluster is ok. >>>>>> >>>>>> cluster d9ca74d0-d9f4-436d-92de-762af67c6534 >>>>>>health HEALTH_OK >>>>>>monmap e1: 9 mons at >>>
Re: [ceph-users] Hadoop and Ceph integration issues
I am trying to start hadoop using bin/start-mapred.sh.
In the HADOOP_HOME/lib, I have:
lib/hadoop-cephfs.jar lib/libcephfs.jar lib/libcephfs_jni.so
(the first I downloaded from
http://ceph.com/docs/master/cephfs/hadoop/ and the other two, I copied
from my system (after installing the ubuntu package for the ceph java
client))
I added to conf/hadoop-env.sh:
export LD_LIBRARY_PATH=/hyrax/hadoop-ceph/lib
I confirmed using bin/hadoop classpath that both jar are in the classpath.
On Mon, Sep 23, 2013 at 3:17 PM, Noah Watkins wrote:
> How are you invoking Hadoop? Also, I forgot to ask, are you using the
> wrappers located in github.com/ceph/hadoop-common (or the jar linked
> to on http://ceph.com/docs/master/cephfs/hadoop/)?
>
> On Mon, Sep 23, 2013 at 12:05 PM, Rolando Martins
> wrote:
>> Hi Noah,
>> I enabled the debugging and got:
>>
>> 2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR:
>> missing keyring, cannot use cephx for authentication
>> 2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
>> 2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2
>>
>> I have the ceph.client.admin.keyring file in /etc/ceph and I tried
>> with and without the
>> parameter in core-site.xml. Unfortunately without success:(
>>
>> Thanks,
>> Rolando
>>
>>
>>
>> fs.ceph.impl
>> org.apache.hadoop.fs.ceph.CephFileSystem
>>
>>
>>
>> fs.default.name
>> ceph://hyrax1:6789/
>>
>>
>>
>> ceph.conf.file
>> /hyrax/hadoop-ceph/ceph/ceph.conf
>>
>>
>>
>> ceph.root.dir
>> /
>>
>>
>> ceph.auth.keyring
>>/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring
>>
>>
>> On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins
>> wrote:
>>> Shoot, I thought I had it figured out :)
>>>
>>> There is a default admin user created when you first create your
>>> cluster. After a typical install via ceph-deploy, there should be a
>>> file called 'ceph.client.admin.keyring', usually sibling to ceph.conf.
>>> If this is in a standard location (e.g. /etc/ceph) you shouldn't need
>>> the keyring option, otherwise point 'ceph.auth.keyring' at that
>>> keyring file. You shouldn't need both the keyring and the keyfile
>>> options set, but it just depends on how your authentication / users
>>> are all setup.
>>>
>>> The easiest thing to do if that doesn't solve your problem is probably
>>> to turn on logging so we can see what is blowing up.
>>>
>>> In your ceph.conf you can add 'debug client = 20' and 'debug
>>> javaclient = 20' to the client section. You may also need to set the
>>> log file 'log file = /path/...'. You don't need to do this on all your
>>> nodes, just one node where you get the failure.
>>>
>>> - Noah
>>>
>>>> Thanks,
>>>> Rolando
>>>>
>>>> P.S.: I have the cephFS mounted locally, so the cluster is ok.
>>>>
>>>> cluster d9ca74d0-d9f4-436d-92de-762af67c6534
>>>>health HEALTH_OK
>>>>monmap e1: 9 mons at
>>>> {hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
>>>> election epoch 6, quorum 0,1,2,3,4,5,6,7,8
>>>> hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
>>>>osdmap e30: 9 osds: 9 up, 9 in
>>>> pgmap v2457: 192 pgs: 192 active+clean; 10408 bytes data, 44312 MB
>>>> used, 168 GB / 221 GB avail
>>>>mdsmap e4: 1/1/1 up {0=hyrax1=up:active}
>>>>
>>>>
>>>>
>>>> fs.ceph.impl
>>>> org.apache.hadoop.fs.ceph.CephFileSystem
>>>>
>>>>
>>>>
>>>> fs.default.name
>>>> ceph://hyrax1:6789/
>>>>
>>>>
>>>>
>>>> ceph.conf.file
>>>> /hyrax/hadoop-ceph/ceph/ceph.conf
>>>>
>>>>
>>>>
>>>> ceph.root.dir
>>>> /
>>>>
>>>>
>>>>
>>>> ceph.auth.keyfile
>>>> /hyrax/hadoop-ceph/ceph/admin.secret
>>>>
>>>>
>>>>
>>>> ceph.auth.keyring
>>>> /hyrax/hadoop-ceph/c
Re: [ceph-users] Hadoop and Ceph integration issues
Hi Noah,
I enabled the debugging and got:
2013-09-23 18:59:34.705894 7f0b58de7700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-09-23 18:59:34.706106 7f0b58de7700 20 client.-1 trim_cache size 0 max 0
2013-09-23 18:59:34.706225 7f0b58de7700 10 jni: ceph_mount: exit ret -2
I have the ceph.client.admin.keyring file in /etc/ceph and I tried
with and without the
parameter in core-site.xml. Unfortunately without success:(
Thanks,
Rolando
fs.ceph.impl
org.apache.hadoop.fs.ceph.CephFileSystem
fs.default.name
ceph://hyrax1:6789/
ceph.conf.file
/hyrax/hadoop-ceph/ceph/ceph.conf
ceph.root.dir
/
ceph.auth.keyring
/hyrax/hadoop-ceph/ceph/ceph.client.admin.keyring
On Mon, Sep 23, 2013 at 2:24 PM, Noah Watkins wrote:
> Shoot, I thought I had it figured out :)
>
> There is a default admin user created when you first create your
> cluster. After a typical install via ceph-deploy, there should be a
> file called 'ceph.client.admin.keyring', usually sibling to ceph.conf.
> If this is in a standard location (e.g. /etc/ceph) you shouldn't need
> the keyring option, otherwise point 'ceph.auth.keyring' at that
> keyring file. You shouldn't need both the keyring and the keyfile
> options set, but it just depends on how your authentication / users
> are all setup.
>
> The easiest thing to do if that doesn't solve your problem is probably
> to turn on logging so we can see what is blowing up.
>
> In your ceph.conf you can add 'debug client = 20' and 'debug
> javaclient = 20' to the client section. You may also need to set the
> log file 'log file = /path/...'. You don't need to do this on all your
> nodes, just one node where you get the failure.
>
> - Noah
>
>> Thanks,
>> Rolando
>>
>> P.S.: I have the cephFS mounted locally, so the cluster is ok.
>>
>> cluster d9ca74d0-d9f4-436d-92de-762af67c6534
>>health HEALTH_OK
>>monmap e1: 9 mons at
>> {hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
>> election epoch 6, quorum 0,1,2,3,4,5,6,7,8
>> hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
>>osdmap e30: 9 osds: 9 up, 9 in
>> pgmap v2457: 192 pgs: 192 active+clean; 10408 bytes data, 44312 MB
>> used, 168 GB / 221 GB avail
>>mdsmap e4: 1/1/1 up {0=hyrax1=up:active}
>>
>>
>>
>> fs.ceph.impl
>> org.apache.hadoop.fs.ceph.CephFileSystem
>>
>>
>>
>> fs.default.name
>> ceph://hyrax1:6789/
>>
>>
>>
>> ceph.conf.file
>> /hyrax/hadoop-ceph/ceph/ceph.conf
>>
>>
>>
>> ceph.root.dir
>> /
>>
>>
>>
>> ceph.auth.keyfile
>> /hyrax/hadoop-ceph/ceph/admin.secret
>>
>>
>>
>> ceph.auth.keyring
>> /hyrax/hadoop-ceph/ceph/ceph.mon.keyring
>>
>>
>> On Mon, Sep 23, 2013 at 11:42 AM, Noah Watkins
>> wrote:
ceph.root.dir
/mnt/mycephfs
>>>
>>> This is probably causing the issue. Is this meant to be a local mount
>>> point? The 'ceph.root.dir' property specifies the root directory
>>> /inside/ CephFS, and the Hadoop implementation doesn't require a local
>>> CephFS mount--it uses a client library to interact with the file
>>> system.
>>>
>>> The default value for this property is "/", so you can probably just
>>> remove this from your config file unless your CephFS directory
>>> structure is carved up in a special way.
>>>
ceph.conf.file
/hyrax/hadoop-ceph/ceph/ceph.conf
ceph.auth.keyfile
/hyrax/hadoop-ceph/ceph/admin.secret
ceph.auth.keyring
/hyrax/hadoop-ceph/ceph/ceph.mon.keyring
>>>
>>> These files will need to be available locally on every node Hadoop
>>> runs on. I think the error below will occur after these are loaded, so
>>> it probably isn't your issue, though I don't recall exactly at which
>>> point different configuration files are loaded.
>>>
fs.hdfs.impl
org.apache.hadoop.fs.ceph.CephFileSystem
>>>
>>> I don't think this is part of the problem you are seeing, but this
>>> 'fs.hdfs.impl' property should probably be removed. We aren't
>>> overriding HDFS, just replacing it.
>>>
ceph.mon.address
hyrax1:6789
>>>
>>> This was already specified in your 'fs.default.name' property. I don't
>>> think that duplicating it is an issue, but I should probably update
>>> the documentation to make it clear that the monitor only needs to be
>>> listed once.
>>>
>>> Thanks!
>>> Noah
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Hadoop and Ceph integration issues
Hi Noah,
updated the core-site.xml (bellow).
All the nodes have the same files. But the problem remains the same.
What is the value for "ceph.auth.keyring"? Is the path containing the
file "ceph.mon.keyring"?
Thanks,
Rolando
P.S.: I have the cephFS mounted locally, so the cluster is ok.
cluster d9ca74d0-d9f4-436d-92de-762af67c6534
health HEALTH_OK
monmap e1: 9 mons at
{hyrax1=10.10.10.10:6789/0,hyrax2=10.10.10.12:6789/0,hyrax3=10.10.10.15:6789/0,hyrax4=10.10.10.13:6789/0,hyrax5=10.10.10.16:6789/0,hyrax6=10.10.10.14:6789/0,hyrax7=10.10.10.18:6789/0,hyrax8=10.10.10.17:6789/0,hyrax9=10.10.10.11:6789/0},
election epoch 6, quorum 0,1,2,3,4,5,6,7,8
hyrax1,hyrax2,hyrax3,hyrax4,hyrax5,hyrax6,hyrax7,hyrax8,hyrax9
osdmap e30: 9 osds: 9 up, 9 in
pgmap v2457: 192 pgs: 192 active+clean; 10408 bytes data, 44312 MB
used, 168 GB / 221 GB avail
mdsmap e4: 1/1/1 up {0=hyrax1=up:active}
fs.ceph.impl
org.apache.hadoop.fs.ceph.CephFileSystem
fs.default.name
ceph://hyrax1:6789/
ceph.conf.file
/hyrax/hadoop-ceph/ceph/ceph.conf
ceph.root.dir
/
ceph.auth.keyfile
/hyrax/hadoop-ceph/ceph/admin.secret
ceph.auth.keyring
/hyrax/hadoop-ceph/ceph/ceph.mon.keyring
On Mon, Sep 23, 2013 at 11:42 AM, Noah Watkins wrote:
>>
>> ceph.root.dir
>> /mnt/mycephfs
>>
>
> This is probably causing the issue. Is this meant to be a local mount
> point? The 'ceph.root.dir' property specifies the root directory
> /inside/ CephFS, and the Hadoop implementation doesn't require a local
> CephFS mount--it uses a client library to interact with the file
> system.
>
> The default value for this property is "/", so you can probably just
> remove this from your config file unless your CephFS directory
> structure is carved up in a special way.
>
>>
>> ceph.conf.file
>> /hyrax/hadoop-ceph/ceph/ceph.conf
>>
>>
>> ceph.auth.keyfile
>> /hyrax/hadoop-ceph/ceph/admin.secret
>>
>>
>>
>> ceph.auth.keyring
>> /hyrax/hadoop-ceph/ceph/ceph.mon.keyring
>>
>
> These files will need to be available locally on every node Hadoop
> runs on. I think the error below will occur after these are loaded, so
> it probably isn't your issue, though I don't recall exactly at which
> point different configuration files are loaded.
>
>>
>> fs.hdfs.impl
>> org.apache.hadoop.fs.ceph.CephFileSystem
>>
>
> I don't think this is part of the problem you are seeing, but this
> 'fs.hdfs.impl' property should probably be removed. We aren't
> overriding HDFS, just replacing it.
>
>>
>> ceph.mon.address
>> hyrax1:6789
>>
>
> This was already specified in your 'fs.default.name' property. I don't
> think that duplicating it is an issue, but I should probably update
> the documentation to make it clear that the monitor only needs to be
> listed once.
>
> Thanks!
> Noah
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Hadoop and Ceph integration issues
Hi all, trying to test ceph with hadoop without success. I deployed the ceph cluster using ceph-deploy. Hadoop - 1.1.3 ceph version 0.67.3 Does anyone had success on making this work? ceph.conf [global] fsid = d9ca74d0-d9f4-436d-92de-762af67c6534 mon initial members = hyrax1, hyrax2, hyrax3, hyrax4, hyrax5, hyrax6, hyrax7, hyrax8, hyrax9 mon host = 10.10.10.10,10.10.10.12,10.10.10.15,10.10.10.13,10.10.10.16,10.10.10.14,10.1 0.10.18,10.10.10.17,10.10.10.11 auth supported = cephx osd journal size = 1024 filestore xattr use omap = true fs.hdfs.impl org.apache.hadoop.fs.ceph.CephFileSystem fs.ceph.impl org.apache.hadoop.fs.ceph.CephFileSystem fs.default.name ceph://hyrax1:6789/ ceph.conf.file /hyrax/hadoop-ceph/ceph/ceph.conf ceph.root.dir /mnt/mycephfs ceph.mon.address hyrax1:6789 ceph.auth.keyfile /hyrax/hadoop-ceph/ceph/admin.secret ceph.auth.keyring /hyrax/hadoop-ceph/ceph/ceph.mon.keyring Trace: 2013-09-23 00:00:02,407 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ubuntu cause:java.io.FileNotFoundException: 2013-09-23 00:00:02,407 INFO org.apache.hadoop.mapred.JobTracker: Problem connecting to HDFS Namenode... re-trying java.io.FileNotFoundException: at com.ceph.fs.CephMount.native_ceph_mount(Native Method) at com.ceph.fs.CephMount.mount(CephMount.java:154) at org.apache.hadoop.fs.ceph.CephTalker.initialize(CephTalker.java:103) at org.apache.hadoop.fs.ceph.CephFileSystem.initialize(CephFileSystem.java:98) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1411) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1429) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:1908) at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:1906) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja va:1149) at org.apache.hadoop.mapred.JobTracker.initializeFilesystem(JobTracker.java:190 6) at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2324) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
