Manoj, thanks for the comments. I added the consolidated branch patch to HDFS-10467 (CC Brahma).
Regarding the comparison with existing approaches, I'd say that the real comparison is ViewFs (already in the docs). This is complementary to the current HDFS federation; you have multiple namespaces and you need to aggregate them. Regarding the best practices for the mount table, I think this is pretty similar to what one would do in ViewFs. Internally, what we are doing is just to have every subcluster following the same naming as the federated namespace. For example, if we mount /data/app1 in subcluster0, we mount it in /data/app1 in the federated namespace. Additionally, we are testing a Rebalancer that takes into consideration the size of the mount table (based on the USENIX ATC paper). I can extend the documentation in HDFS-12381. On Thu, Aug 31, 2017 at 4:52 PM, Iñigo Goiri <elgo...@gmail.com> wrote: > Agreed on this not being the cleanest.. > Just filed it this morning: HDFS-12384. > > > On Thu, Aug 31, 2017 at 4:36 PM, Andrew Wang <andrew.w...@cloudera.com> > wrote: > >> v) mvn install (and package) is failing with following error >>> >>> [INFO] Adding ignore: * >>> [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses >>> failed with message: >>> Duplicate classes found: >>> >>> Found in: >>> org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-beta1- >>> SNAPSHOT:compile >>> org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-beta1-SNAP >>> SHOT:compile >>> Duplicate classes: >>> org/apache/hadoop/shaded/org/apache/curator/framework/api/De >>> leteBuilder.class >>> org/apache/hadoop/shaded/org/apache/curator/framework/Curato >>> rFramework.class >>> >>> >>> I added "hadoop-client-minicluster" to ignore list to get success >>> >>> hadoop\hadoop-client-modules\hadoop-client-integration-tests\pom.xml >>> >>> <dependencies> >>> <dependency> >>> <groupId>org.apache.hadoop</groupId> >>> <artifactId>hadoop-annotations</artifactId> >>> <ignoreClasses> >>> <ignoreClass>*</ignoreClass> >>> </ignoreClasses> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.hadoop</groupId> >>> <artifactId>hadoop-client-minicluster</artifactId> >>> <ignoreClasses> >>> <ignoreClass>*</ignoreClass> >>> </ignoreClasses> >>> </dependency> >>> >> >> Is there a JIRA filed for this issue? We should engage with Sean Busbey >> on the right fix. I don't think it's right to exclude the minicluster from >> this checking. >> > >
