Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21469
@HeartSaVioR , may be then this should be reported in the "memoryUsedBytes"
in the StateOperatorProgress (value reported in StreamingQueryProgress) because
currently the usage reported does not reflect the memory used for the cache.
Question: in the screenshot "Estimated size of states cache in provider
total" is 3.3 MB whereas the "memory used by state total" is 20.6 KB with
"total number of state rows" = 2. This 150x difference is expected with just 2
rows in the state? Were there 100 versions of the map in the sample output you
posted?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
