Spark32 + Java 11 . Reading parquet java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'

[Pralabh Kumar]

Hi Dev team

I have a spark32 image with Java 11 (Running Spark on K8s) .  While reading
a huge parquet file via  spark.read.parquet("") .  I am getting the
following error . The same error is mentioned in Spark docs
https://spark.apache.org/docs/latest/#downloading but w.r.t to apache arrow.


   - IMHO , I think the error is coming from Parquet 1.12.1  which is based
   on Hadoop 2.10 which is not java 11 compatible.

Please let me know if this understanding is correct and is there a way to
fix it.


java.lang.NoSuchMethodError: 'sun.misc.Cleaner
sun.nio.ch.DirectBuffer.cleaner()'

            at
org.apache.hadoop.crypto.CryptoStreamUtils.freeDB(CryptoStreamUtils.java:41)

            at
org.apache.hadoop.crypto.CryptoInputStream.freeBuffers(CryptoInputStream.java:687)

            at
org.apache.hadoop.crypto.CryptoInputStream.close(CryptoInputStream.java:320)

            at java.base/java.io.FilterInputStream.close(Unknown Source)

            at
org.apache.parquet.hadoop.util.H2SeekableInputStream.close(H2SeekableInputStream.java:50)

            at
org.apache.parquet.hadoop.ParquetFileReader.close(ParquetFileReader.java:1299)

            at
org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:54)

            at
org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:44)

            at
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:467)

            at
org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)

            at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)

            at scala.util.Success.$anonfun$map$1(Try.scala:255)

            at scala.util.Success.map(Try.scala:213)

            at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

            at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

            at
scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

            at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

            at
java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown
Source)

            at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown
Source)

            at
java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown
Source)

            at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown
Source)

            at
java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)

            at
java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)