[ https://issues.apache.org/jira/browse/PARQUET-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249700#comment-17249700 ]
Gabor Szadovszky commented on PARQUET-1953: ------------------------------------------- There are a couple of jiras about similar topics. See PARQUET-1126 for example. Unfortunately, no one had the time to really do it. > hadoop-common is not an optional dependency > ------------------------------------------- > > Key: PARQUET-1953 > URL: https://issues.apache.org/jira/browse/PARQUET-1953 > Project: Parquet > Issue Type: Bug > Reporter: Sam Halliday > Priority: Major > > parquet-hadoop provides the only mechanism to load .parquet files and has an > optional (provided) dependency on hadoop-common, implying that it is possible > to use parquet-hadoop without using hadoop. However, it is required. > The following code is needed to instantiate a ParquetFileReader > {code} > final class LocalInputFile(file: File) extends InputFile { > def getLength() = file.length() > def newStream(): SeekableInputStream = { > val input = new FileInputStream(file) > new DelegatingSeekableInputStream(input) { > def getPos(): Long = input.getChannel().position() > def seek(bs: Long): Unit = { > val _ = input.getChannel().position(bs) > } > } > } > } > {code} > but using this leads to a runtime exception because hadoop is missing > transitive dependency on org.apache.hadoop.fs.PathFilter which then depends > on org.apache.hadoop.fs.Path, both in hadoop-common. > Requiring downstream users to depend on hadoop-common is an extremely large > dependency and I would rather that this was not the case. > A search for "import org.apache.hadoop" in src/main reveals a few more places > where the dependency is hardwired, although often in deprecated static > constructors and therefore benign. -- This message was sent by Atlassian Jira (v8.3.4#803005)