Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21070
Based on the previous upgrade (e.g.,
https://github.com/apache/spark/pull/13280#issuecomment-223022415), we hit the
performance regressions when we upgrade Parquet and we did the revert at the
end. That might be the major reason why we have not upgraded to 1.9 yet.
Jumping to 1.10.0 from 1.8.2 is big and risky especially when it was
recently released. Even if we merge it now, we might still revert it if we hit
ANY performance regression. That is why we prefer to do the performance testing
before we do the merge.
Not sure whether the community can help the performance tests? cc @maropu
You did many performance tests in TPC-DS in the past release. Could you help
this too? If possible, we also need to do the micro-benchmark like what
@dongjoon-hyun did for ORC.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
