Hi Zoltan, thanks for the answer.
Because replacing hdfs:// with ceph:// and use CephFs doesn't work for all haddop componentes out of the box (unless in my tests), for example I had issues with Hbase, then with Yarn, Hue, etc (I'm using the cloudera distribution but I also tried with separate components). And besides the need to add jars and bindings to each node to get them work, there are a lot of places (xmls, configuration) where the "hdfs for ceph" replacement need to be made. Giving this issues, I thought that mounting ceph as a local directory and then use this "virtual dirs" as the haddop dfs dirs, would be easier and will work better (less configuration problems, and only changing the dfs dirs will make all components work without any more changes). Of course I can be totally wrong, and it's a core change to do this, that's why I think I should ask here first :) Thanks! PS: If you are asking why I'm trying to use ceph here, well it's because we were given an infrastructure with the possibility yo use a big ceph storage that's working really really well (but as an object store and wasn't use until now with hadoop). ________________________________ De: Zoltan Arnold Nagy <zol...@linux.vnet.ibm.com> Enviado: jueves, 04 de febrero de 2016 06:07 p.m. Para: John Spray Cc: Jose M; ceph-users@lists.ceph.com Asunto: Re: [ceph-users] Ceph and hadoop (fstab insted of CephFS) Might be totally wrong here, but it's not layering them but replacing hdfs:// URLs with ceph:// URLs so all the mapreduce/spark/hbase/whatever is on top can use CephFS directly which is not a bad thing to do (if it works) :-) On 02 Feb 2016, at 16:50, John Spray <jsp...@redhat.com<mailto:jsp...@redhat.com>> wrote: On Tue, Feb 2, 2016 at 3:42 PM, Jose M <soloning...@hotmail.com<mailto:soloning...@hotmail.com>> wrote: Hi, One simple question, in the ceph docs says that to use Ceph as an HDFS replacement, I can use the CephFs Hadoop plugin (http://docs.ceph.com/docs/master/cephfs/hadoop/). What I would like to know if instead of using the plugin, I can mount ceph in fstab and then point hdfs dirs (namenode, datanode, etc) to this mounted "ceph" dirs, instead of native local dirs. I understand that maybe will involve more configuration steps (configuring fstab in each node), but will this work? Is there any problem with this type of configuration? Without being a big HDFS expert, it seems like you would be essentially putting one distributed filesystem on top of another distributed filesystem. I don't know if you're going to find anything that breaks as such, but it's probably not a good idea. John Thanks in advance, _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
