dongjoon-hyun commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-859136503
It's just a historical factor. IMO, I believe that we need to replace it to
`zstd-jni`.
> Does it make sense to use aircompressor for ZSTD in ORC, rather than the
zstd-
dongjoon-hyun commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-860062911
@dchristle . I'm a big supporter of ZStandard and have no doubt that we need
to upgrade ZSTD-JNI in the future. Your PR will be a part of Apache Spark
definitely.
-
ht
dongjoon-hyun commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-861791665
Well, do you know that the feature freeze of Apache Spark 3.2.0 is [July
1st](https://spark.apache.org/versioning-policy.html) for now? It seems that
that's the root cause
dongjoon-hyun commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-857226630
BTW, @dchristle . I quickly ran the micro benchmark on master branch on my
Mac before and after. We also need to check the memory usage, too.
```
OpenJDK 64-Bit
dongjoon-hyun commented on pull request #32826:
URL: https://github.com/apache/spark/pull/32826#issuecomment-857411986
Thank you for your efforts. BTW, @dchristle . Please note that your ORC PR
is not about ZSTD-JNI. It's native ZSTD library only. I commented on your ORC
PR about the diffe
