Thanks for the clarification. That makes sense.. In the code below, we can
see
def onQueryProgress(self, event):
print("onQueryProgress")
# Access micro-batch data
microbatch_data = event.progress
#print("microbatch_data received") # Check if data is received
Oh I see why the confusion.
microbatch_data = event.progress
means that microbatch_data is a StreamingQueryProgress instance, it's not a
dictionary, so you should use ` microbatch_data.processedRowsPerSecond`,
instead of the `get` method which is used for dictionaries.
But weirdly, for query.las
Hi,
Thank you for your advice
This is the amended code
def onQueryProgress(self, event):
print("onQueryProgress")
# Access micro-batch data
microbatch_data = event.progress
#print("microbatch_data received") # Check if data is received
#print(microbatc
*now -> not
刘唯 于2024年3月10日周日 22:04写道：
> Have you tried using microbatch_data.get("processedRowsPerSecond")?
> Camel case now snake case
>
> Mich Talebzadeh 于2024年3月10日周日 11:46写道：
>
>>
>> There is a paper from Databricks on this subject
>>
>>
>> https://www.databricks.com/blog/2022/05/27/how-to-
Have you tried using microbatch_data.get("processedRowsPerSecond")?
Camel case now snake case
Mich Talebzadeh 于2024年3月10日周日 11:46写道：
>
> There is a paper from Databricks on this subject
>
>
> https://www.databricks.com/blog/2022/05/27/how-to-monitor-streaming-queries-in-pyspark.html
>
> But havi
There is a paper from Databricks on this subject
https://www.databricks.com/blog/2022/05/27/how-to-monitor-streaming-queries-in-pyspark.html
But having tested it, there seems to be a bug there that I reported to
Databricks forum as well (in answer to a user question)
I have come to a conclusion
