Abhishek Agarwal created FLINK-3271:
---------------------------------------
Summary: Using webhdfs in a flink topology throws classnotfound
exception
Key: FLINK-3271
URL: https://issues.apache.org/jira/browse/FLINK-3271
Project: Flink
Issue Type: Bug
Components: Core
Affects Versions: 0.10.1
Reporter: Abhishek Agarwal
I was just trying to run a storm topology on flink using flink-storm. I got
this exception -
{noformat}
Caused by: java.lang.NoClassDefFoundError: org/mortbay/util/ajax/JSON
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:727)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:610)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
{noformat}
My topology list some files on hdfs using webhdfs API.
org.mortbay.util.ajax.JSON is included in the uber jar. I noticed that flink
loads the application jar in a child classloader. This is what most likely
happened -
1. WebHdfsFileSystem class was loaded through parent class loader since it is
included in flink-dist.jar.
2. WebHdfsFileSystem has reference to the org.mortbay.util.ajax.JSON but since
it is loaded through parent class loader, WebHdfsFileSystem can't read a class
in child class loader.
Ideally all the referenced classes should be available in the distribution jar
so that these sort of issues may not occur.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
