A question on what is documented on the crail website for configuring to use
the HDFS adapter. The website shows the use of a wildcard (in red):
spark.driver.extraClassPath
$CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
spark.executor.extraClassPath
$CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
But this didn’t work for me. I had to put all the specific jars onto this path
in order for this to work,
such as the lines below. Is there a trick to using the wildcard?
spark.driver.extraClassPath
/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
spark.executor.extraClassPath
/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
This is the error I get with using the wildcard:
Exception in thread "main" java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
org.apache.crail.hdfs.CrailHadoopFileSystem not found
And if I add that one to the path, then I get the next one it doesn’t find.
Regards,
David
