Re: Watermarking in Structured Streaming to drop late data

Hi all I would highly recommend to all users-devs interested in the design suggestions / discussions for Structured Streaming Spark API watermarking to take a look on the following links along with the design document. It would help to understand the notions of watermark , out of order data and

Re: [Spark SQL] SQLContext getOrCreate incorrect behaviour

Hi Fyi The following 2 tickets are blocking currently (for releases up to 1.5.2) the pattern of Starting and Stopping a sparkContext inside the same driver program https://issues.apache.org/jira/browse/SPARK-11700 ->memory leak in SqlContext https://issues.apache.org/jira/browse/SPARK-11739 In

Re: A proposal for Spark 2.0

Hi Kostas With regards to your *second* point. I believe that requiring from the user apps to explicitly declare their dependencies is the most clear API approach when it comes to classpath and classloading. However what about the following API: *SparkContext.addJar(String pathToJar)* . *Is this