Re: Spark Streaming - Direct Approach

[Cody Koeninger]

We've been running direct stream jobs in production for over a year,
with uptimes in the range of months.

I'm pretty slammed with work right now, but when I get time to submit
a PR for the 0.10 docs i'll remove the experimental note from 0.8

On Mon, Jul 11, 2016 at 4:35 PM, Tathagata Das
 wrote:
> Aah, the docs have not been updated. They are totally in production in many
> place. Others should chime in as well.
>
> On Mon, Jul 11, 2016 at 1:43 PM, Mail.com  wrote:
>>
>> Hi All,
>>
>> Can someone please confirm if streaming direct approach for reading Kafka
>> is still experimental or can it be used for production use.
>>
>> I see the documentation and talk from TD suggesting the advantages of the
>> approach but docs state it is an "experimental" feature.
>>
>> Please suggest
>>
>> Thanks,
>> Pradeep
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Streaming - Direct Approach

[Tathagata Das]

Aah, the docs have not been updated. They are totally in production in many
place. Others should chime in as well.

On Mon, Jul 11, 2016 at 1:43 PM, Mail.com  wrote:

> Hi All,
>
> Can someone please confirm if streaming direct approach for reading Kafka
> is still experimental or can it be used for production use.
>
> I see the documentation and talk from TD suggesting the advantages of the
> approach but docs state it is an "experimental" feature.
>
> Please suggest
>
> Thanks,
> Pradeep
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: Spark Streaming - Direct Approach

[Andy Davidson]

Hi Pradeep

I can not comment about experimental or production, how ever I recently
started a POC using direct approach.

Its been running off and on for about 2 weeks. In general it seems to work
really well. One thing that is not clear to me is how the cursor is manage.
E.G. I have my topic set to delete after 1 hr. I have had some problem that
caused huge delaying. I.E. They are running after the data is deleted. After
lots of jobs fail it seems to recover. I.E. The cursor advance to valid
position.


I am running into problems how ever I do not think it has to do with the
direct approach. I think it has to do with writing to s3.

Andy

From:  "Mail.com" <pradeep.mi...@mail.com>
Date:  Monday, July 11, 2016 at 1:43 PM
To:  "user @spark" <user@spark.apache.org>
Subject:  Spark Streaming - Direct Approach

> Hi All,
> 
> Can someone please confirm if streaming direct approach for reading Kafka is
> still experimental or can it be used for production use.
> 
> I see the documentation and talk from TD suggesting the advantages of the
> approach but docs state it is an "experimental" feature.
> 
> Please suggest
> 
> Thanks,
> Pradeep
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> 
> 

Spark Streaming - Direct Approach

[Mail.com]

Hi All,

Can someone please confirm if streaming direct approach for reading Kafka is 
still experimental or can it be used for production use.

I see the documentation and talk from TD suggesting the advantages of the 
approach but docs state it is an "experimental" feature. 

Please suggest

Thanks,
Pradeep

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org