Re: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

2024-04-09 Thread
Sorry this is not a bug but essentially a user error. Spark throws a really confusing error and I'm also confused. Please see the reply in the ticket for how to make things correct. https://issues.apache.org/jira/browse/SPARK-47718 刘唯 于2024年4月6日周六 11:41写道: > This indeed looks like a bug

Re: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

2024-04-06 Thread
This indeed looks like a bug. I will take some time to look into it. Mich Talebzadeh 于2024年4月3日周三 01:55写道: > > hm. you are getting below > > AnalysisException: Append output mode not supported when there are > streaming aggregations on streaming DataFrames/DataSets without watermark; > > The

Re: Bug in How to Monitor Streaming Queries in PySpark

2024-03-11 Thread
of my kno > wledge but of course cannot be guaranteed . It is essential to note that, > as with any advice, quote "one test result is worth one-thousand expert op > inions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun > <https://en.wikipedia.org/wiki

Re: Bug in How to Monitor Streaming Queries in PySpark

2024-03-10 Thread
*now -> not 刘唯 于2024年3月10日周日 22:04写道: > Have you tried using microbatch_data.get("processedRowsPerSecond")? > Camel case now snake case > > Mich Talebzadeh 于2024年3月10日周日 11:46写道: > >> >> There is a paper from Databricks on this subject >> >&g

Re: Bug in How to Monitor Streaming Queries in PySpark

2024-03-10 Thread
Have you tried using microbatch_data.get("processedRowsPerSecond")? Camel case now snake case Mich Talebzadeh 于2024年3月10日周日 11:46写道: > > There is a paper from Databricks on this subject > > > https://www.databricks.com/blog/2022/05/27/how-to-monitor-streaming-queries-in-pyspark.html > > But