Hello Gabor, These Parquet/Avro/Spark version incompatibilities are not new with Parquet 1.11.0, similar issues were present when Spark depended on Avro 1.7.x and also depended on a Parquet version which required Avro 1.8.x.
Perhaps if the parquet-avro test scope dependency did not exclude the Avro 1.9.1 transitive dependencies these runtime issues would show up in Spark unit tests rather than in downstream projects. I am testing this hypothesis today. michael > On Jan 27, 2020, at 3:52 AM, Gabor Szadovszky > <[email protected]> wrote: > > Hi Michael, > > Avro does not follow the rules of semantic versioning. 1.9.x is a > major version upgrade comparing 1.8.x (the "1" means the file format). > Therefore, Avro has nothing to do with their braking changes. > Maybe, it was not a good idea to upgrade Avro in Parquet but we already > released this upgrade 1.11.0. I don't think it either that we should have > released such an upgrade in a major release. In the past we already have > done such upgrades. > If we agree on allowing such upgrades in minor releases than the problems > with the parquet+avro upgrade falls back to the consumers so this time to > Spark. Seems that Spark has to upgrade its Avro dependencies as well and > change the code accordingly. > > Regards, > Gabor
