Re: [VOTE] Standardize Spark Exception Messages SPIP

2020-11-08 Thread Takeshi Yamamuro
+1 On Thu, Nov 5, 2020 at 3:41 AM Xinyi Yu wrote: > Hi all, > > We had the discussion of SPIP: Standardize Spark Exception Messages at > > http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-SPIP-Standardize-Spark-Exception-Messages-td30341.html > < >

Re: [DISCUSS] Disable streaming query with possible correctness issue by default

2020-11-08 Thread Jungtaek Lim
After the check logic was introduced in Spark 3.0, there's another related issue I addressed in Spark 3.1, SPARK-24634 [1]. Before SPARK-24634, there's no way to know how many rows are discarded due to being late, even whether there's any late row or not. That said, the issue has been the