[
https://issues.apache.org/jira/browse/BEAM-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-7979:
-------------------------------
Status: Open (was: Triage Needed)
> Avro incompatibilities with Spark 2.2 and Spark 2.3
> ---------------------------------------------------
>
> Key: BEAM-7979
> URL: https://issues.apache.org/jira/browse/BEAM-7979
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, io-java-parquet, sdk-java-core
> Reporter: Ryan Skraba
> Priority: Major
>
> Much of the code that depends on Avro (notably the wrappers built with
> [BeamSQL|https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L34]
> but also
> [some|https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java]
>
> [connectors|https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java#L42])
> require a version > 1.8.x
> This library is not present in Spark 2.2 and Spark 2.3 clusters, which are
> meant to be supported. These pipelines will fail with ClassNotFoundException
> / MethodNotFoundExceptions.
> Spark 2.4+ should be unaffected.
> Relocating or vendoring is probably not appropriate, since Avro is frequently
> exposed in the API through parameters and potentially in generated specific
> records.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
