Spark streaming Kafka 0.11 integration

2017-09-04 Thread matus.cimerman 
Hi guys,

is there any plans to support Kafka 0.11 integration for Spark streaming
applications? I see it doesn't support yet. If there is any way how can I
help/contribute, I'll be happy if you point me right direction so that I can
give a hand.

Sincerely,
Matus Cimerman



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Unsubscribe

2017-09-04 Thread Pavel Gladkov 
Unsubscribe

-- 
С уважением,
Павел Гладков

Re: [Spark Streaming] Dynamic Broadcast Variable Update

2017-09-04 Thread takkarharsh 
Hi, 

I have the same use-case, can you please let me know the implementation for
the same.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [SS] Writing a test for a possible bug in StateStoreSaveExec with Append output mode?

2017-09-04 Thread Jacek Laskowski 
Hi,

I think I know where the issue surfaces. This is with groupBy
aggregation with Append output mode.

What should happen when a state expires for a event time (in groupBy)
with the new rows for the expired key in a streaming batch exactly
when watermark has been moved up and thus expired the state for the
key?

Example's coming up.

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
Spark Structured Streaming (Apache Spark 2.2+)
https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sun, Sep 3, 2017 at 11:04 PM, Jacek Laskowski  wrote:
> Hi,
>
> I may have found a bug in StateStoreSaveExec with Append output mode
> and would love proving myself I'm wrong or help squashing it by
> writing a test for the case.
>
> Is there a test for StateStoreSaveExec with Append output mode? If
> not, is there a streaming test template that could be very close to a
> test and that I could use?
>
> Thanks for any help you may offer!
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> Spark Structured Streaming (Apache Spark 2.2+)
> https://bit.ly/spark-structured-streaming
> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org