SparkStructured Streaming using withWatermark - TypeError: 'module' object is not callable

2022-02-15 Thread karan alang
Hello All, I have a Structured Streaming pyspark program running on GCP Dataproc, which reads data from Kafka, and does some data massaging, and aggregation. I'm trying to use withWatermark(), and it is giving error. py4j.Py4JException: An exception was raised by the Python Proxy. Return

Re: Apache spark 3.0.3 [Spark lower version enhancements]

2022-02-15 Thread Sean Owen
I think these are readily answerable if you look at the text of the CVEs and Spark 3.0.3 release. https://nvd.nist.gov/vuln/detail/CVE-2019-17531 concerns Jackson Databind up to 2.9.10, but you can see that 3.0.3 uses 2.10.0 https://nvd.nist.gov/vuln/detail/CVE-2020-9480 affects Spark 2.x, not

Re: Apache spark 3.0.3 [Spark lower version enhancements]

2022-02-15 Thread Rajesh Krishnamurthy
Hi Sean, I am looking for fixing the vulnerabilities such as these in the 3.0.X branch. 1) CVE-2019-17531 2)CVE-2020-9480 3)CVE-2019-0204 Rajesh Krishnamurthy | Enterprise Architect T: +1 510-833-7189 | M: +1 925-917-9208 http://www.perforce.com Visit us on: