[ https://issues.apache.org/jira/browse/BEAM-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905107#comment-16905107 ]
Lukasz Gajowy commented on BEAM-5164: ------------------------------------- I think so but I'm not very confident when it comes to these decisions. [After removing shading by default|https://lists.apache.org/thread.html/00202ff6a1e6eb7c2ce8c54cafd2afe7eb5524c63f29ed3bd695860e@%3Cdev.beam.apache.org%3E] and switching to vendoring some dependencies in general I'm not sure if we should just use shadow when declaring the dependency or vendor the jars in that case same as we vendor e.g. guava. Adding [~lcwik] so that he could share his opinion. > ParquetIOIT fails on Spark and Flink > ------------------------------------ > > Key: BEAM-5164 > URL: https://issues.apache.org/jira/browse/BEAM-5164 > Project: Beam > Issue Type: Bug > Components: testing > Reporter: Lukasz Gajowy > Priority: Minor > > When run on Spark or Flink remote cluster, ParquetIOIT fails with the > following stacktrace: > {code:java} > org.apache.beam.sdk.io.parquet.ParquetIOIT > writeThenReadAll FAILED > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > org.apache.parquet.hadoop.ParquetWriter$Builder.<init>(Lorg/apache/parquet/io/OutputFile;)V > at > org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:66) > at > org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:99) > at > org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87) > at org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:116) > at org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:61) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) > at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:350) > at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331) > at > org.apache.beam.sdk.io.parquet.ParquetIOIT.writeThenReadAll(ParquetIOIT.java:133) > Caused by: > java.lang.NoSuchMethodError: > org.apache.parquet.hadoop.ParquetWriter$Builder.<init>(Lorg/apache/parquet/io/OutputFile;)V{code} > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)