novatechflow commented on PR #798: URL: https://github.com/apache/wayang/pull/798#issuecomment-5491174289
The Spark and ANTLR mismatch is fixed, but always validate the full impact of the dependency upgrade with a full test run: => The Spark 3.5.7 upgrade introduces a Parquet runtime regression. Parquet dependencies are mixed: - `parquet-hadoop`: 1.12.3 - `parquet-column`: 1.13.1 - `parquet-avro`: 1.15.2 Spark 3.5.7 references `CompressionCodecName.LZ4_RAW`, but `parquet-hadoop:1.12.3` does not provide that field. Test evidence: Full test run in that PR produced 3 errors: - `SparkParquetSinkTest.writesDatasetToParquet`: `NoSuchFieldError: LZ4_RAW` - `SparkParquetSinkTest.writesRddToParquet`: `NoClassDefFoundError` after `ParquetOptions` initialization failed. - `SparkParquetSourceDatasetOutputTest.producesDatasetChannel`: same initialization failure. Aligning the Parquet dependencies with Spark 3.5.7 should fix that, and rerun the Spark reactor afterwards ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
