Which version of Parquet has this bug? Maybe we can downgrade it. On Mon, Mar 11, 2019 at 10:34 AM Mark Hamstra <m...@clearstorydata.com> wrote:
> It worked in 2.3. We broke it with 2.4.0 and were informed of that > regression late in the 2.4.0 release process. Since we didn't fix it before > the 2.4.0 release, it should have been noted as a known issue. To now claim > that there is no regression from 2.4.0 is a circular argument denying the > existence of a known regression from 2.3. > > On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sro...@gmail.com> wrote: > >> From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that: >> >> - this is a Parquet-Avro version conflict thing >> - a downstream app wants different versions of Parquet and Avro than >> Spark uses, which triggers it >> - it doesn't work in 2.4.0 >> >> It's not a regression from 2.4.0, which is the immediate question. >> There isn't even a Parquet fix available. >> But I'm not even seeing why this is excuse-making? >> >> On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <m...@clearstorydata.com> >> wrote: >> > >> > Now wait... we created a regression in 2.4.0. Arguably, we should have >> blocked that release until we had a fix; but the issue came up late in the >> release process and it looks to me like there wasn't an adequate fix >> immediately available, so we did something bad and released 2.4.0 with a >> known regression. Saying that there is now no regression from 2.4 is >> tautological and no excuse for not taking in a fix -- and it looks like >> that fix has been waiting for months. >> >
