[ceph-users] high density machines
Hi, I am wondering if anybody in the community is running ceph cluster with high density machines e.g. Supermicro SYS-F618H-OSD288P (288 TB), Supermicro SSG-6048R-OSD432 (432 TB) or some other high density machines. I am assuming that the installation will be of petabyte scale as you would want to have at least 3 of these boxes. It would be good to hear their experiences in terms of reliability, performance (specially during node failures). As these machines have 40Gbit network connection it can be ok, but experience from real users would be great to hear. As these are mentioned in the reference architecture published by red hat and supermicro. Thanks for your time. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] high density machines
Thanks everybody for the feedback. On 09/03/2015 05:09 PM, Mark Nelson wrote: > My take is that you really only want to do these kinds of systems if you > have massive deployments. At least 10 of them, but probably more like > 20-30+. You do get massive density with them, but I think if you are > considering 5 of these, you'd be better off with 10 of the 36 drive > units. An even better solution might be ~30-40 of these: > > http://www.supermicro.com/products/system/1U/6017/SYS-6017R-73THDP_.cfm > This one does look interesting. > An extremely compelling solution would be if they took this system: > > http://www.supermicro.com/products/system/1U/5018/SSG-5018A-AR12L.cfm?parts=SHOW > This one can be really good solution for archiving purpose with replaced CPU to get more juice into it. > > and replaced the C2750 with a Xeon-D 1540 (but keep the same number of > SATA ports). > > Potentially you could have: > > - 8x 2.0GHz Xeon Broadwell-DE Cores, 45W TDP > - Up to 128GB RAM (32GB probably the sweet spot) > - 2x 10GbE > - 12x 3.5" spinning disks > - single PCIe slot for PCIe SSD/NVMe I am wondering does single PCIe SSD/NVMe device can support 12 OSDs journals and still perform the same as 4 OSD per SSD ? > > The density would be higher than the 36 drive units but lower than the > 72 drive units (though with shorter rack depth afaik). You mean the 1U solution with 12 disk is longer in length than 72 disk 4U version ? - Gurvinder Probably more > CPU per OSD and far better distribution of OSDs across servers. Given > that the 10GbE and processor are embedded on the motherboard, there's a > decent chance these systems could be priced reasonably and wouldn't have > excessive power/cooling requirements. > > Mark > > On 09/03/2015 09:13 AM, Jan Schermer wrote: >> It's not exactly a single system >> >> SSG-F618H-OSD288P* >> 4U-FatTwin, 4x 1U 72TB per node, Ceph-OSD-Storage Node >> >> This could actually be pretty good, it even has decent CPU power. >> >> I'm not a big fan of blades and blade-like systems - sooner or later a >> backplane will die and you'll need to power off everything, which is a >> huge PITA. >> But assuming you get 3 of these it could be pretty cool! >> It would be interesting to have a price comparison to a SC216 chassis >> or similiar, I'm afraid it won't be much cheaper. >> >> Jan >> >>> On 03 Sep 2015, at 16:09, Kris Gillespie wrote: >>> >>> It's funny cause in my mind, such dense servers seems like a bad idea to >>> me for exactly the reason you mention, what if it fails. Losing 400+TB >>> of storage is going to have quite some impact, 40G interfaces or not and >>> no matter what options you tweak. >>> Sure it'll be cost effective per TB, but that isn't the only aspect to >>> look at (for production use anyways). >>> >>> But I'd also be curious about real world feedback. >>> >>> Cheers >>> >>> Kris >>> >>> The 09/03/2015 16:01, Gurvinder Singh wrote: >>>> Hi, >>>> >>>> I am wondering if anybody in the community is running ceph cluster with >>>> high density machines e.g. Supermicro SYS-F618H-OSD288P (288 TB), >>>> Supermicro SSG-6048R-OSD432 (432 TB) or some other high density >>>> machines. I am assuming that the installation will be of petabyte scale >>>> as you would want to have at least 3 of these boxes. >>>> >>>> It would be good to hear their experiences in terms of reliability, >>>> performance (specially during node failures). As these machines have >>>> 40Gbit network connection it can be ok, but experience from real users >>>> would be great to hear. As these are mentioned in the reference >>>> architecture published by red hat and supermicro. >>>> >>>> Thanks for your time. >>>> ___ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> De informatie verzonden met dit e-mailbericht is uitsluitend bestemd >>> voor de geadresseerde. Gebruik van deze informatie door anderen dan >>> de geadresseerde is uitdrukkelijk verboden. Indien u dit bericht per >>> vergissing heeft ontvangen, verzoeken wij u ons onmiddelijk hiervan >>> op de hoogte te stellen en het bericht te vernietigen. >>> Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking >>>
Re: [ceph-users] high density machines
On 09/04/2015 02:31 AM, Wang, Warren wrote: > In the minority on this one. We have a number of the big SM 72 drive units w/ > 40 Gbe. Definitely not as fast as even the 36 drive units, but it isn't awful > for our average mixed workload. We can exceed all available performance with > some workloads though. > > So while we can't extract all the performance out of the box, as long as we > don't max out on performance, the cost is very appealing, I am wondering how much the cost difference you have seen with SM 72 drive compare to lets say http://www.supermicro.com/products/system/1U/6017/SYS-6017R-73THDP_.cfm or any other smaller machine which you have compared with. As with the discussion on this thread it is clear that the 72 drive box are actually 4 * 18 drive boxes sharing power and cooling. Regarding performance I think network might be bottleneck (may be cpu too), as it is 40 Gbit for whole box so you get 10 Gbit per box (18 drives each) which can be peaked. Gurvinder and as far as filling a unit, I'm not sure how many folks have filled big prod clusters, but you really don't want them even running into the 70+% range due to some inevitable uneven filling, and room for failure. > > Also, I'm betting that Ceph will continue to optimize things like the > messenger, and reduce some of the massive CPU and TCP overhead, so we can > claw back performance. I would love to see a thread count reduction. These > can see over 130K threads per box. > > Warren > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark > Nelson > Sent: Thursday, September 03, 2015 3:58 PM > To: Gurvinder Singh ; > ceph-users@lists.ceph.com > Subject: Re: [ceph-users] high density machines > > > > On 09/03/2015 02:49 PM, Gurvinder Singh wrote: >> Thanks everybody for the feedback. >> On 09/03/2015 05:09 PM, Mark Nelson wrote: >>> My take is that you really only want to do these kinds of systems if >>> you have massive deployments. At least 10 of them, but probably more >>> like >>> 20-30+. You do get massive density with them, but I think if you are >>> considering 5 of these, you'd be better off with 10 of the 36 drive >>> units. An even better solution might be ~30-40 of these: >>> >>> http://www.supermicro.com/products/system/1U/6017/SYS-6017R-73THDP_.c >>> fm >>> >> This one does look interesting. >>> An extremely compelling solution would be if they took this system: >>> >>> http://www.supermicro.com/products/system/1U/5018/SSG-5018A-AR12L.cfm >>> ?parts=SHOW >>> >> This one can be really good solution for archiving purpose with >> replaced CPU to get more juice into it. >>> >>> and replaced the C2750 with a Xeon-D 1540 (but keep the same number >>> of SATA ports). >>> >>> Potentially you could have: >>> >>> - 8x 2.0GHz Xeon Broadwell-DE Cores, 45W TDP >>> - Up to 128GB RAM (32GB probably the sweet spot) >>> - 2x 10GbE >>> - 12x 3.5" spinning disks >>> - single PCIe slot for PCIe SSD/NVMe >> I am wondering does single PCIe SSD/NVMe device can support 12 OSDs >> journals and still perform the same as 4 OSD per SSD ? > > Basically the limiting factor is how fast the device can do O_DSYNC writes. > We've seen that some PCIe SSD and NVME devices can do 1-2GB/s depending on > the capacity which is enough to reasonably support 12-24 OSDs. Whether or > not it's good to have a single PCIe card to be a point of failure is a > worthwhile topic (Probably only high write endurance cards should be > considered). There are plenty of other things that can bring the node down > too though (motherboard, ram, cpu, etc) though. A single node failure will > also have less impact if there are lots of small nodes vs a couple big ones. > >>> >>> The density would be higher than the 36 drive units but lower than >>> the >>> 72 drive units (though with shorter rack depth afaik). >> You mean the 1U solution with 12 disk is longer in length than 72 disk >> 4U version ? > > Sorry, the other way around I believe. > >> >> - Gurvinder >>Probably more >>> CPU per OSD and far better distribution of OSDs across servers. >>> Given that the 10GbE and processor are embedded on the motherboard, >>> there's a decent chance these systems could be priced reasonably and >>> wouldn't have excessive power/cooling requirements. >>> >>> Mark >>> >>> On 09/03/2015 09:13 AM, Jan Schermer
[ceph-users] osd daemon cpu threads
Hi, Just wondering if a Ceph OSD daemon supports multi threading and can get benefit from multi core Intel/ARM processor. E.g. 12 disk server with 36 Intel or 48 ARM cores. Thanks, Gurvinder ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] osd daemon cpu threads
Thanks Jan for the reply. It's good to know that Ceph can use extra cpus for throughput. I am wondering if any one in the community has used/experimented with Arm v8 2.5 GHz prosessors instead of Intel E5. On Sep 8, 2015 12:28 PM, "Jan Schermer" wrote: > In terms of throughput yes - one OSD may have thousands of threads doing > work so it will scale accross multiple clients. > But in terms of latency you are still limited by a throughput of one core, > so for database workloads or any type of synchronous or single-threaded IO > more cores will be of no help. > > Jan > > > On 08 Sep 2015, at 10:50, Gurvinder Singh < > gurvindersinghdah...@gmail.com> wrote: > > > > Hi, > > > > Just wondering if a Ceph OSD daemon supports multi threading and can get > > benefit from multi core Intel/ARM processor. E.g. 12 disk server with 36 > > Intel or 48 ARM cores. > > > > Thanks, > > Gurvinder > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph hbase issue
On 04/14/2014 11:38 PM, Noah Watkins wrote: > This strikes me as a difference in semantics between HDFS and CephFS, > and like Greg said it's probably based on HBase assumptions. It'd be > really helpful to find out what the exception is. If you are building > the Hadoop bindings from scratch, you can instrument `listStatus` in > `CephFileSystem.java` -- I believe this is being caused by > `listStatus` returning `null`. > I can't test the system now, as the cluster has been taken down. But I can confirm that it was mainly how HBase want the file system to work. Moreover the problem was when creating file with names as ".filename", so if it has "." in start e.g. hidden/temp files, ceph has been failing on that. Although I could create ceph from cephfs. Noah, you may know more about it. - Gurvinder > > > On Mon, Apr 14, 2014 at 2:27 PM, Gregory Farnum wrote: >> This looks like some kind of HBase issue to me (which I can't help >> with; I've never used it), but I guess if I were looking at Ceph I'd >> check if it was somehow configured such that the needed files are >> located in different pools (or other separate security domains) that >> might be set up wrong. >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Mon, Apr 7, 2014 at 1:49 AM, Gurvinder Singh >> wrote: >>> Hi, >>> >>> I am trying to make HBase 0.96 work on top of Ceph 0.72.2. When I start >>> the Hbase-master I am getting this error. >>> >>> 2014-04-05 23:39:39,475 DEBUG [master:pltrd023:6] wal.FSHLog: Moved >>> 1 WAL file(s) to /hbase/data/hbase/meta/1588230740/oldWALs >>> 2014-04-05 23:39:39,538 FATAL [master:host:6] master.HMaster: >>> Unhandled exception. Starting shutdown. >>> java.io.IOException: Error accessing >>> ceph://mon-host:6789/hbase/data/hbase/meta/.tabledesc >>> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1486) >>> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524) >>> at >>> org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1582) >>> at >>> org.apache.hadoop.hbase.util.FSTableDescriptors.getCurrentTableInfoStatus(FSTableDescriptors.java:348) >>> at >>> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:329) >>> at >>> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:310) >>> at >>> org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:709) >>> at >>> org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:690) >>> at >>> org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:677) >>> at >>> org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:486) >>> at >>> org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) >>> at >>> org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:127) >>> at >>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789) >>> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606) >>> at java.lang.Thread.run(Thread.java:724) >>> >>> >>> The only odd thing/warn i see in the log file is this >>> >>> wal.FSHLog: FileSystem's output stream doesn't support >>> getNumCurrentReplicas; --HDFS-826 not available; >>> fsOut=org.apache.hadoop.fs.ceph.CephOutputStream >>> >>> It has able to create hbase root and other directories such as data,meta >>> etc. So it seems Hbase is able to communicate with Ceph, but somehow it >>> is not able to create all the necessary files Any suggestions there ? >>> >>> I have added these config options in the hbase-site.xml file >>> >>> >>> fs.defaultFS >>> ceph://mon-host:6789/ >>> >>> >>> >>> ceph.conf.options >>> client_readahead_min=4193404 >>> >>> >>> >>> ceph.conf.file >>> /etc/ceph/ceph.conf >>> >>> >>> >>> ceph.auth.id >>> admin >>> >>> >>> >>> ceph.auth.keyfile >&
Re: [ceph-users] Spark/Mesos on top of Ceph/Btrfs
We are definitely interested in such a setup too. Not necessarily with btrfs but ceph combined with Spark & Mesos. Earlier last year the ceph hadoop plugin was not stable enough after that I haven't looked in to it. As hadoop plugin relies of CephFS and it is still under development. So we are currently following its development and waiting for things to get stable. - Gurvinder On 01/13/2015 02:25 PM, James wrote: > Hello, > > > I was wondering if anyone has Mesos running on top of Ceph? > I want to test/use Ceph if lieu of HDFS. > > > I'm working on Gentoo, but any experiences with Mesos on Ceph > are of keen interest to me as related to performance, stability > and any difficulties experienced. > > > James > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes
One interesting use case of combining Ceph with computing is running big data jobs on ceph itself. As with CephFS coming along, you can run Haddop/Spark jobs directly on ceph without needed to move your data to compute resources with data locality support. I am wondering if anyone in community is looking at combining storage and compute resources from this point of view. Regards, Gurvinder On 03/29/2015 09:19 PM, Nick Fisk wrote: > There's probably a middle ground where you get the best of both worlds. > Maybe 2-4 OSD's per compute node alongside dedicated Ceph nodes. That way > you get a bit of extra storage and can still use lower end CPU's, but don't > have to worry so much about resource contention. > >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Martin Millnert >> Sent: 29 March 2015 19:58 >> To: Mark Nelson >> Cc: ceph-users@lists.ceph.com >> Subject: Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same >> nodes >> >> On Thu, Mar 26, 2015 at 12:36:53PM -0500, Mark Nelson wrote: >>> Having said that, small nodes are >>> absolutely more expensive per OSD as far as raw hardware and >>> power/cooling goes. >> >> The smaller volume manufacturers have on the units, the worse the margin >> typically (from buyers side). Also, CPUs typically run up a premium the > higher >> you go. I've found a lot of local maximas, optimization-wise, over the > past >> years both in 12 OSD/U vs 18 OSD/U dedicated storage node setups, for >> instance. >> There may be local maximas along colocated low-scale storage/compute >> nodes, but the one major problem with colocating storage with compute is >> that you can't scale compute independently from storage efficiently, on >> using that building block alone. There may be temporal optimizations in >> doing so however (e.g. before you have reached sufficient scale). >> >> There's no single optimal answer when you're dealing with 20+ variables to >> consider... :) >> >> BR, >> Martin > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes
On 03/30/2015 01:29 PM, Mark Nelson wrote: > This is definitely something that we've discussed, though I don't think > anyone has really planned out what a complete solution would look like > including processor affinity, etc. Before I joined inktank I worked at > a supercomputing institute and one of the projects we worked on was to > develop grid computing tools for bioinformatics research. Moving > analytics rather than the data was a big topic for us too since genomics > data at least tends to be pretty big. Interestingly I work for supercomputing/education research company and we are thinking for similar use case and purpose. So are interesting to know other people managing resources this way. - Gurvinder Potentially ceph could be a very > interesting solution for that kind of thing. > > Mark > > On 03/30/2015 06:20 AM, Gurvinder Singh wrote: >> One interesting use case of combining Ceph with computing is running big >> data jobs on ceph itself. As with CephFS coming along, you can run >> Haddop/Spark jobs directly on ceph without needed to move your data to >> compute resources with data locality support. I am wondering if anyone >> in community is looking at combining storage and compute resources from >> this point of view. >> >> Regards, >> Gurvinder >> On 03/29/2015 09:19 PM, Nick Fisk wrote: >>> There's probably a middle ground where you get the best of both worlds. >>> Maybe 2-4 OSD's per compute node alongside dedicated Ceph nodes. That >>> way >>> you get a bit of extra storage and can still use lower end CPU's, but >>> don't >>> have to worry so much about resource contention. >>> >>>> -Original Message- >>>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On >>>> Behalf Of >>>> Martin Millnert >>>> Sent: 29 March 2015 19:58 >>>> To: Mark Nelson >>>> Cc: ceph-users@lists.ceph.com >>>> Subject: Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the >>>> same >>>> nodes >>>> >>>> On Thu, Mar 26, 2015 at 12:36:53PM -0500, Mark Nelson wrote: >>>>> Having said that, small nodes are >>>>> absolutely more expensive per OSD as far as raw hardware and >>>>> power/cooling goes. >>>> >>>> The smaller volume manufacturers have on the units, the worse the >>>> margin >>>> typically (from buyers side). Also, CPUs typically run up a premium >>>> the >>> higher >>>> you go. I've found a lot of local maximas, optimization-wise, over the >>> past >>>> years both in 12 OSD/U vs 18 OSD/U dedicated storage node setups, for >>>> instance. >>>>There may be local maximas along colocated low-scale storage/compute >>>> nodes, but the one major problem with colocating storage with >>>> compute is >>>> that you can't scale compute independently from storage efficiently, on >>>> using that building block alone. There may be temporal >>>> optimizations in >>>> doing so however (e.g. before you have reached sufficient scale). >>>> >>>> There's no single optimal answer when you're dealing with 20+ >>>> variables to >>>> consider... :) >>>> >>>> BR, >>>> Martin >>> >>> >>> >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] yarn (hadoop 2.2.x) support
Hi, I am planning to test the cephfs to replace hdfs. I am using Yarn (Hadoop 2.2.x) branch. I am wondering is there any jar available to test ceph with this version. Also how is the support of data locality in ceph. Thanks, Gurviner Singh ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] yarn (hadoop 2.2.x) support
Hi Noah, Thanks for the quick feedback. Its great to know the ceph has on way to support yarn. I am planning to test ceph, HDFS and glusterfs performance using intel hadoop benchmark. I will let you know if on the way I need any help in getting there. If you have any howto of setting up ceph as file system for yarn would be perfect. I have looked at the cephfs howto for hadoop on wiki, but there might be deprecated properties there in case of yarn. Regards, Gurvinder On 02/18/2014 03:59 PM, Noah Watkins wrote: > Hi Gurviner, > > There is a pull request for Hadoop 2 support here > > https://github.com/noahdesu/cephfs-hadoop/pull/1 > > I have not yet tested it personally, but it looks OK to me. > > Data locality support in Ceph is supported. > > On 2/18/14, 3:15 AM, Gurvinder Singh wrote: >> Hi, >> >> I am planning to test the cephfs to replace hdfs. I am using >> Yarn (Hadoop 2.2.x) branch. I am wondering is there any jar >> available to test ceph with this version. Also how is the support >> of data locality in ceph. >> >> Thanks, Gurviner Singh >> ___ ceph-users >> mailing list ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph mapreduce history server issue
Hi, I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 hadoop. I have installed the ceph hadoop binding with hadoop 2.x support. I am able to run the command such as # hadoop fs -ls / # hdfs dfs -touchz /test But when I start the mapreduce job history server I am getting the error as "Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: ceph" Here is the link to mapreduce startup message listing the lib it linked against while starting and it has both libcepfs and cephfs-hadoop jars. http://pastebin.com/wED7nMMT I have made the changes in to core-site.xml file, that's why I am able to the normal hadoop commands. Let me know if you have any suggestions how to resolve this. Regards, Gurvinder ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph mapreduce history server issue
Hi, I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 hadoop. I have installed the ceph hadoop binding with hadoop 2.x support. I am able to run the command such as # hadoop fs -ls / # hdfs dfs -touchz /test But when I start the mapreduce job history server I am getting the error as "Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: ceph" Here is the link to mapreduce startup message listing the lib it linked against while starting and it has both libcepfs and cephfs-hadoop jars. http://pastebin.com/wED7nMMT I have made the changes in to core-site.xml file, that's why I am able to the normal hadoop commands. Let me know if you have any suggestions how to resolve this. Regards, Gurvinder ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph hadoop mapred history server issue
Hi, I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 hadoop. I have installed the ceph hadoop binding with hadoop 2.x support. I am able to run the command such as # hadoop fs -ls / # hdfs dfs -touchz /test But when I start the mapreduce job history server I am getting the error as "Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: ceph" Here is the link to mapreduce startup message listing the lib it linked against while starting and it has both libcepfs and cephfs-hadoop jars. "pastebin links seems to be causing message to go to spam" I have made the changes in to core-site.xml file, that's why I am able to the normal hadoop commands. Let me know if you have any suggestions how to resolve this. Regards, Gurvinder ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph mapreduce history server issue
On 03/19/2014 03:04 PM, Alejandro Bonilla wrote: > Hi Gurvinder, > > This setup sounds interesting. Which guide did you follow? > There wasn't any specific guide to follow. But the documentation from ceph wiki http://ceph.com/docs/master/cephfs/hadoop/ has helped. I can write a step by step post once, I will get this issue fixed. To make it easy to setup ceph and test with hadoop 2.x Regards, Gurvinder > On Wed, Mar 19, 2014 at 6:44 AM, Gurvinder Singh > wrote: >> Hi, >> >> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 >> hadoop. I have installed the ceph hadoop binding with hadoop 2.x >> support. I am able to run the command such as >> >> # hadoop fs -ls / >> # hdfs dfs -touchz /test >> >> But when I start the mapreduce job history server I am getting the error as >> >> "Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No >> AbstractFileSystem for scheme: ceph" >> >> Here is the link to mapreduce startup message listing the lib it linked >> against while starting and it has both libcepfs and cephfs-hadoop jars. >> >> http://pastebin.com/wED7nMMT >> >> I have made the changes in to core-site.xml file, that's why I am able >> to the normal hadoop commands. Let me know if you have any suggestions >> how to resolve this. >> >> Regards, >> Gurvinder >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph hadoop mapred history server issue
On 03/19/2014 04:50 PM, Noah Watkins wrote: > Since `hadoop -fs ls /` seems to work on your local node, can you > verify that (1) it is in fact listing the contents of CephFS, and (2) > that on your worker nodes where the error is occuring that the > relevant dependencies (naming the Ceph hadoop bindings) are installed > and in the classpath? 1) yes it is listing the cephfs contents. 2) Currently the setup is to start the yarn manager and mapreduce history server on the same node and then setup the worker nodes. Yarn starts up fine, but the mapreduce history server gives me the error. The dependency lib are loaded in the mapreduce history server startup. Here is the link to start up message which shows that the libcephfs and cephfs-hadoop jar are loaded. http://pastebin.com/wED7nMMT I am wondering if this property value is correct fs.ceph.impl org.apache.hadoop.fs.ceph.CephFileSystem or should I set it to something for hadoop 2.x I have tried setting it up as org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as # hadoop fs -ls / -ls: Fatal internal error java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190) at org.apache.hadoop.fs.shell.Command.run(Command.java:154) at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:308) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.() at java.lang.Class.getConstructor0(Class.java:2800) at java.lang.Class.getDeclaredConstructor(Class.java:2043) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 17 more Regards, Gurvinder > > The error itself looks like a missing dependency, but that exception > being thrown might also be tirggered by other problems while loading > the bindings. > > On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh > wrote: >> On 03/19/2014 03:51 PM, Noah Watkins wrote: >>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh >>> wrote: >>>> Hi, >>>> >>>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 >>>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x >>>> support. I am able to run the command such as >>> >>> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1? >>> >> Yes. >>> Can you paste your core-site.xml? >> >> Below is the relevant part for ceph conf >> >> >> fs.defaultFS >> ceph://:6789/ >> >> >> >> ceph.conf.file >> /etc/ceph/ceph.conf >> >> >> >> ceph.auth.id >> admin >> >> >> >> ceph.auth.keyfile >> /etc/hadoop/conf/admin.secret >> >> >> >> fs.ceph.impl >> org.apache.hadoop.fs.ceph.CephFileSystem >> >> >> >>> >> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph hadoop mapred history server issue
On 03/19/2014 05:18 PM, Noah Watkins wrote: > Err, obviously switching things out for Ceph rather than Gluster. > > On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins > wrote: >> Looks like this is a configuration issue that has popped up with other >> 3rd party file systems in Hadoop 2.x with YARN. >> >> >> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E >> >> Says use this: >> >> >> fs.AbstractFileSystem.glusterfs.impl >> org.apache.hadoop.fs.glusterfs.GlusterFS >> Interestingly this solves the mapred history server issue. It created the history folder and started. fs.AbstractFileSystem.ceph.impl org.apache.hadoop.fs.ceph.CephHadoop2FileSystem But now this fails as hadoop fs -ls / ls: No FileSystem for scheme: ceph - Gurvinder >> >> Apparently rather than the `fs.ceph.impl` property in 2.x >> >> >> On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh >> wrote: >>> On 03/19/2014 04:50 PM, Noah Watkins wrote: >>>> Since `hadoop -fs ls /` seems to work on your local node, can you >>>> verify that (1) it is in fact listing the contents of CephFS, and (2) >>>> that on your worker nodes where the error is occuring that the >>>> relevant dependencies (naming the Ceph hadoop bindings) are installed >>>> and in the classpath? >>> 1) yes it is listing the cephfs contents. >>> 2) Currently the setup is to start the yarn manager and mapreduce >>> history server on the same node and then setup the worker nodes. Yarn >>> starts up fine, but the mapreduce history server gives me the error. The >>> dependency lib are loaded in the mapreduce history server startup. Here >>> is the link to start up message which shows that the libcephfs and >>> cephfs-hadoop jar are loaded. >>> >>> http://pastebin.com/wED7nMMT >>> >>> I am wondering if this property value is correct >>> >>> >>> fs.ceph.impl >>> org.apache.hadoop.fs.ceph.CephFileSystem >>> >>> >>> or should I set it to something for hadoop 2.x I have tried setting it up as >>> >>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as >>> >>> # hadoop fs -ls / >>> -ls: Fatal internal error >>> java.lang.RuntimeException: java.lang.NoSuchMethodException: >>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.() >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) >>> at >>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315) >>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) >>> at >>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) >>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) >>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) >>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) >>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) >>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) >>> at >>> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) >>> at >>> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) >>> at >>> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207) >>> at >>> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190) >>> at org.apache.hadoop.fs.shell.Command.run(Command.java:154) >>> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308) >>> Caused by: java.lang.NoSuchMethodException: >>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.() >>> at java.lang.Class.getConstructor0(Class.java:2800) >>> at java.lang.Class.getDeclaredConstructor(Class.java:2043) >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) >>> ... 17 more >>> >>> Regards, >>> Gurvinder >>> >>>> >>>> The error itself looks like a missing dependency, but that exception >>
Re: [ceph-users] ceph hadoop mapred history server issue
On 03/19/2014 05:58 PM, Noah Watkins wrote: > That certainly is odd. Does it work if you list both old and new > properties (perhaps the CLI tools are looking at an older property.. > but that seems unlikely)? Sorry I don't have more answers, I haven't > yet deploying Hadoop 2.x.. Another strange thing but.. putting the both the properties on core-site.xml makes both tools works. - Gurvinder > > On Wed, Mar 19, 2014 at 9:30 AM, Gurvinder Singh > wrote: >> On 03/19/2014 05:18 PM, Noah Watkins wrote: >>> Err, obviously switching things out for Ceph rather than Gluster. >>> >>> On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins >>> wrote: >>>> Looks like this is a configuration issue that has popped up with other >>>> 3rd party file systems in Hadoop 2.x with YARN. >>>> >>>> >>>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3c1023550423.3137743.1371825668412.javamail.r...@redhat.com%3E >>>> >>>> Says use this: >>>> >>>> >>>> fs.AbstractFileSystem.glusterfs.impl >>>> org.apache.hadoop.fs.glusterfs.GlusterFS >>>> >> Interestingly this solves the mapred history server issue. It created >> the history folder and started. >> >> >> fs.AbstractFileSystem.ceph.impl >> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem >> >> >> >> >> But now this fails as >> >> hadoop fs -ls / >> ls: No FileSystem for scheme: ceph >> >> - Gurvinder >>>> >>>> Apparently rather than the `fs.ceph.impl` property in 2.x >>>> >>>> >>>> On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh >>>> wrote: >>>>> On 03/19/2014 04:50 PM, Noah Watkins wrote: >>>>>> Since `hadoop -fs ls /` seems to work on your local node, can you >>>>>> verify that (1) it is in fact listing the contents of CephFS, and (2) >>>>>> that on your worker nodes where the error is occuring that the >>>>>> relevant dependencies (naming the Ceph hadoop bindings) are installed >>>>>> and in the classpath? >>>>> 1) yes it is listing the cephfs contents. >>>>> 2) Currently the setup is to start the yarn manager and mapreduce >>>>> history server on the same node and then setup the worker nodes. Yarn >>>>> starts up fine, but the mapreduce history server gives me the error. The >>>>> dependency lib are loaded in the mapreduce history server startup. Here >>>>> is the link to start up message which shows that the libcephfs and >>>>> cephfs-hadoop jar are loaded. >>>>> >>>>> http://pastebin.com/wED7nMMT >>>>> >>>>> I am wondering if this property value is correct >>>>> >>>>> >>>>> fs.ceph.impl >>>>> org.apache.hadoop.fs.ceph.CephFileSystem >>>>> >>>>> >>>>> or should I set it to something for hadoop 2.x I have tried setting it up >>>>> as >>>>> >>>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as >>>>> >>>>> # hadoop fs -ls / >>>>> -ls: Fatal internal error >>>>> java.lang.RuntimeException: java.lang.NoSuchMethodException: >>>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.() >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315) >>>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) >>>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) >>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) >>>>> at >>>>> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) >>>>> at >>>>> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) &g
Re: [ceph-users] ceph hadoop mapred history server issue
On 03/19/2014 03:51 PM, Noah Watkins wrote: > On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh > wrote: >> Hi, >> >> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 >> hadoop. I have installed the ceph hadoop binding with hadoop 2.x >> support. I am able to run the command such as > > From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1? > Yes. > Can you paste your core-site.xml? Below is the relevant part for ceph conf fs.defaultFS ceph://:6789/ ceph.conf.file /etc/ceph/ceph.conf ceph.auth.id admin ceph.auth.keyfile /etc/hadoop/conf/admin.secret fs.ceph.impl org.apache.hadoop.fs.ceph.CephFileSystem > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs fast on a single big file but very slow on may files
On 03/24/2014 08:02 AM, Yan, Zheng wrote:
> On Sun, Mar 23, 2014 at 8:39 PM, Sascha Frey wrote:
>> Hi list,
>>
>> I'm new to ceph and so I installed a four node ceph cluster for testing
>> purposes.
>>
>> Each node has two 6-core sandy bridge Xeons, 64 GiB of RAM, 6 15k rpm
>> SAS drives, one SSD drive for journals and 10G ethernet.
>> We're using Debian GNU/Linux 7.4 (Wheezy) with kernel 3.13 from Debian
>> backports repository and Ceph 0.72.2-1~bpo70+1.
>>
>> Every node runs six OSDs (one for every SAS disk). The SSD is partitioned
>> into six parts for journals.
>> Monitors are three of the same nodes (no extra hardware for mons and MDS
>> for testing). First, I used node #4 as a MDS and later I installed
>> Ceph-MDS on all four nodes with set_max_mds=3.
>>
>> I did increase pg_num and pgp_num to 1200 each for both data and
>> metadata pools.
>>
>> I mounted the cephfs on one node using the kernel client.
>> Writing to a single big file is fast:
>>
>> $ dd if=/dev/zero of=bigfile bs=1M count=1M
>> 1048576+0 records in
>> 1048576+0 records out
>> 1099511627776 bytes (1.1 TB) copied, 1240.52 s, 886 MB/s
>>
>> Reading is less fast:
>> $ dd if=bigfile of=/dev/null bs=1M
>> 1048576+0 records in
>> 1048576+0 records out
>> 1099511627776 bytes (1.1 TB) copied, 3226.8 s, 341 MB/s
>> (during reading, the nodes are mostly idle (>90%, 1-1.8% wa))
>>
>> After this, I tried to copy the linux kernel source tree (source and
>> dest dirs both on cephfs, 600 MiB, 45k files):
>>
>> $ time cp -a linux-3.13.6 linux-3.13.6-copy
>>
>> real35m34.184s
>> user0m1.884s
>> sys 0m11.372s
>>
>> That's much too slow.
>> The same process takes just a few seconds on one desktop class SATA
>> drive.
>>
>> I can't see any load or I/O wait on any of the four nodes. I tried
>> different mount options:
>>
>> mon1,mon2,mon3:/ on /export type ceph
>> (rw,relatime,name=someuser,secret=,nodcache,nofsc)
>> mon1,mon2,mon3:/ on /export type ceph
>> (rw,relatime,name=someuser,secret=,dcache,fsc,wsize=10485760,rsize=10485760)
>>
>> Output of 'ceph status':
>> ceph status
>> cluster 32ea6593-8cd6-40d6-ac3b-7450f1d92d16
>> health HEALTH_OK
>> monmap e1: 3 mons at
>> {howard=xxx.yyy.zzz.199:6789/0,leonard=xxx.yyy.zzz.196:6789/0,penny=xxx.yyy.zzz.198:6789/0},
>> election epoch 32, quorum 0,1,2 howard,leonard,penny
>> mdsmap e107: 1/1/1 up {0=penny=up:active}, 3 up:standby
>> osdmap e276: 24 osds: 24 up, 24 in
>> pgmap v8932: 2464 pgs, 3 pools, 1028 GB data, 514 kobjects
>> 2061 GB used, 11320 GB / 13382 GB avail
>> 2464 active+clean
>> client io 119 MB/s rd, 509 B/s wr, 43 op/s
>>
>>
>> I appreciate if someone may help me to find the reason for that
>> odd behaviour.
>
> In your case, copying each file requires sending several requests to
> the MDS/OSD, each request can take several to tens of millisecond.
> That's why only about 20 files were copied per second. One option to
> improve the overall speed is perform a parallel copy. (you can find
> some scripts from google)
I have observed the same behavior in our cluster, but by using GNU
parallel as
/mnt/ceph/linux-3.13.6#time parallel -j10 cp -r {} /mnt/ceph/copy/ ::: *
to copy the source code reduce time to
real14m22.721s
user0m1.208s
sys 0m7.200s
Hope it helps.
- Gurvinder
>
> Regards
> Yan, Zheng
>
>>
>>
>> Cheers,
>> Sascha
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph yarn setup
Hi, As mentioned earlier, here is the link to how to guide to make yarn work with Ceph emperor. http://blog.uninettlabs.no/?p=54 Feel free to ask any questions. Gurvinder Singh Uninett AS ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph hbase issue
Hi, I am trying to make HBase 0.96 work on top of Ceph 0.72.2. When I start the Hbase-master I am getting this error. 2014-04-05 23:39:39,475 DEBUG [master:pltrd023:6] wal.FSHLog: Moved 1 WAL file(s) to /hbase/data/hbase/meta/1588230740/oldWALs 2014-04-05 23:39:39,538 FATAL [master:host:6] master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException: Error accessing ceph://mon-host:6789/hbase/data/hbase/meta/.tabledesc at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1486) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524) at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1582) at org.apache.hadoop.hbase.util.FSTableDescriptors.getCurrentTableInfoStatus(FSTableDescriptors.java:348) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:329) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:310) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:709) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:690) at org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:677) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:486) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:127) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606) at java.lang.Thread.run(Thread.java:724) The only odd thing/warn i see in the log file is this wal.FSHLog: FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.ceph.CephOutputStream It has able to create hbase root and other directories such as data,meta etc. So it seems Hbase is able to communicate with Ceph, but somehow it is not able to create all the necessary files Any suggestions there ? I have added these config options in the hbase-site.xml file fs.defaultFS ceph://mon-host:6789/ ceph.conf.options client_readahead_min=4193404 ceph.conf.file /etc/ceph/ceph.conf ceph.auth.id admin ceph.auth.keyfile /etc/hbase/conf/admin.secret fs.ceph.impl org.apache.hadoop.fs.ceph.CephFileSystem fs.AbstractFileSystem.ceph.impl org.apache.hadoop.fs.ceph.CephHadoop2FileSystem hbase.rootdir ceph://mon-host:6789/hbase The directory shared by RegionServers. - Gurvinder ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
