Re: [DISCUSS] "latestFirst" option and metadata growing issue in File stream source

2020-08-17 Thread Jungtaek Lim
Bump again. Unlike file stream sink which has lots of limitations and many of us have been suggesting alternatives, file stream source is the only way if end users want to read the data from files. No alternative unless they introduce another ETL & storage (probably Kafka). On Fri, Jul 31, 2020

Re: [VOTE] Release Spark 2.4.7 (RC1)

2020-08-17 Thread Xiao Li
https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release. Thanks, Xiao On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma wrote: > Thanks for letting us know. So this vote is