Hi all, I'm currently developing a Spark Structured Streaming job and I'm performing flatMapGroupsWithState.
I'm concerned about the laziness of the Iterator[V] that is passed to my custom function (func: (K, Iterator[V], GroupState[S]) => Iterator[U]). Is it ok to collect that iterator (with a toList, for example)? I have a logic that is practically impossible to perform on a Iterator, but I do not want to break Spark lazy chain, obviously. Thank you in advance. #A.M.