Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21057#discussion_r181632575
--- Diff: python/pyspark/streaming/listener.py ---
@@ -22,6 +22,10 @@ class StreamingListener(object):
def __init__(self):
pass
+
+ def onStreamingStarted(self, streamingStarted):
--- End diff --
I first thought it's doc only change but I realised it's actually not after
taking a close look.
Implementing `onStreamingStarted` looks actually required:
> Add missing method to StreamingListener, which is invoked by proxy from
the Java/Scala side, and throws a strange exception when not found.
This wasn't there at the first -
https://github.com/apache/spark/blob/ace0db47141ffd457c2091751038fc291f6d5a8b/python/pyspark/streaming/listener.py
/
https://github.com/apache/spark/blob/ace0db47141ffd457c2091751038fc291f6d5a8b/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingListener.scala
; however, this method was added from
https://github.com/apache/spark/commit/ce99f51d2e8cbcb1565c9e7a729183bd74a0c2bc
but seems Python's implementation is missed.
Strictly it sounds better to have an explicit test since @aviv-ebates has a
reproducer (assuming from the description) and should be easy to add.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
