It worked in 2.3. We broke it with 2.4.0 and were informed of that regression late in the 2.4.0 release process. Since we didn't fix it before the 2.4.0 release, it should have been noted as a known issue. To now claim that there is no regression from 2.4.0 is a circular argument denying the existence of a known regression from 2.3.
On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <sro...@gmail.com> wrote: > From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that: > > - this is a Parquet-Avro version conflict thing > - a downstream app wants different versions of Parquet and Avro than > Spark uses, which triggers it > - it doesn't work in 2.4.0 > > It's not a regression from 2.4.0, which is the immediate question. > There isn't even a Parquet fix available. > But I'm not even seeing why this is excuse-making? > > On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <m...@clearstorydata.com> > wrote: > > > > Now wait... we created a regression in 2.4.0. Arguably, we should have > blocked that release until we had a fix; but the issue came up late in the > release process and it looks to me like there wasn't an adequate fix > immediately available, so we did something bad and released 2.4.0 with a > known regression. Saying that there is now no regression from 2.4 is > tautological and no excuse for not taking in a fix -- and it looks like > that fix has been waiting for months. >
