Re: Kafaka 0.8, 0.9 in Structured Streaming

[Michael Armbrust]

>
> Without a hell of a lot more work, Assign would be the only strategy
> usable.


How would the current "subscribe" break?

Re: Kafaka 0.8, 0.9 in Structured Streaming

[Cody Koeninger]

Without a hell of a lot more work, Assign would be the only strategy usable.

On Fri, Oct 7, 2016 at 3:25 PM, Michael Armbrust  wrote:
>> The implementation is totally and completely different however, in ways
>> that leak to the end user.
>
>
> Can you elaborate? Especially in the context of the interface provided by
> structured streaming.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Kafaka 0.8, 0.9 in Structured Streaming

[Michael Armbrust]

> 0.10 consumers won't work on an earlier broker.
> Earlier consumers will (should?) work on a 0.10 broker.
>

 This lines up with my testing.  Is there a page I'm missing that describes
this?  Like does a 0.9 client work with 0.8 broker?  Is it always old
clients can talk to new brokers but not vice versa?

Re: Kafaka 0.8, 0.9 in Structured Streaming

[Michael Armbrust]

>
> The implementation is totally and completely different however, in ways
> that leak to the end user.


Can you elaborate? Especially in the context of the interface provided by
structured streaming.

Re: Kafaka 0.8, 0.9 in Structured Streaming

[Cody Koeninger]

0.10 consumers won't work on an earlier broker.

Earlier consumers will (should?) work on a 0.10 broker.

The main things earlier consumers lack from a user perspective is
support for SSL, and pre-fetching messages.  The implementation is
totally and completely different however, in ways that leak to the end
user.

On Fri, Oct 7, 2016 at 3:15 PM, Reynold Xin  wrote:
> Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?
>
>
> On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith 
> wrote:
>>
>> +1
>>
>> We're on CDH, and it will probably be a while before they support Kafka
>> 0.10. At the same time, we don't use their Spark and we're looking forward
>> to upgrading to 2.0.x and using structured streaming.
>>
>> I was just going to write our own Kafka Source implementation which uses
>> the existing KafkaRDD but it would be much easier to get buy-in for an
>> official Spark module.
>>
>> Jeremy
>>
>> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust 
>> wrote:
>>>
>>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>>> I've been hearing a few people tell me that they are stuck on an older
>>> version of Kafka and cannot upgrade.  I'm considering revisiting
>>> SPARK-17344, but it would be good to have more information.
>>>
>>> Could people please vote or comment on the above ticket if a lack of
>>> support for older versions of kafka would block you from trying out
>>> structured streaming?
>>>
>>> Thanks!
>>>
>>> Michael
>>
>>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Kafaka 0.8, 0.9 in Structured Streaming

[Reynold Xin]

Does Kafka 0.10 work on a Kafka 0.8/0.9 cluster?


On Fri, Oct 7, 2016 at 1:14 PM, Jeremy Smith 
wrote:

> +1
>
> We're on CDH, and it will probably be a while before they support Kafka
> 0.10. At the same time, we don't use their Spark and we're looking forward
> to upgrading to 2.0.x and using structured streaming.
>
> I was just going to write our own Kafka Source implementation which uses
> the existing KafkaRDD but it would be much easier to get buy-in for an
> official Spark module.
>
> Jeremy
>
> On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust 
> wrote:
>
>> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
>> I've been hearing a few people tell me that they are stuck on an older
>> version of Kafka and cannot upgrade.  I'm considering revisiting
>> SPARK-17344 , but it
>> would be good to have more information.
>>
>> Could people please vote or comment on the above ticket if a lack of
>> support for older versions of kafka would block you from trying out
>> structured streaming?
>>
>> Thanks!
>>
>> Michael
>>
>
>

Re: Kafaka 0.8, 0.9 in Structured Streaming

[Jeremy Smith]

+1

We're on CDH, and it will probably be a while before they support Kafka
0.10. At the same time, we don't use their Spark and we're looking forward
to upgrading to 2.0.x and using structured streaming.

I was just going to write our own Kafka Source implementation which uses
the existing KafkaRDD but it would be much easier to get buy-in for an
official Spark module.

Jeremy

On Fri, Oct 7, 2016 at 12:41 PM, Michael Armbrust 
wrote:

> We recently merged support for Kafak 0.10.0 in Structured Streaming, but
> I've been hearing a few people tell me that they are stuck on an older
> version of Kafka and cannot upgrade.  I'm considering revisiting
> SPARK-17344 , but it
> would be good to have more information.
>
> Could people please vote or comment on the above ticket if a lack of
> support for older versions of kafka would block you from trying out
> structured streaming?
>
> Thanks!
>
> Michael
>