jerrypeng commented on code in PR #47940:
URL: https://github.com/apache/spark/pull/47940#discussion_r1744415183
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala:
##########
@@ -447,10 +450,17 @@ case class FlatMapGroupsWithStateExec(
hasTimedOut,
watermarkPresent)
- // Call function, get the returned objects and convert them to rows
- val mappedIterator = func(keyObj, valueObjIter, groupState).map { obj =>
- numOutputRows += 1
- getOutputRow(obj)
+ val mappedIterator = try {
+ // Call function, get the returned objects and convert them to rows
+ func(keyObj, valueObjIter, groupState).map { obj =>
Review Comment:
I don't think this catches the issue I mentioned. The code in the custom
iterator will not get executed here. For example, a user can return an
iterator like the following
```
new Iterator[Int] {
override def hasNext: Boolean = {
true
}
override def next(): Int = {
throw new Exception()
}
```
Only when "next" gets call will the problem occur. I think you need to wrap
the iterator returned by the user func and catch any exceptions there.
Can you also add a test for this scenario as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
