Re: does KafkaCluster can be public ?

[Cody Koeninger]

If anyone is interested in keeping tabs on it, the jira for this is

https://issues.apache.org/jira/browse/SPARK-10963

On Wed, Oct 7, 2015 at 3:16 AM, Erwan ALLAIN 
wrote:

> Thanks guys !
>
> On Wed, Oct 7, 2015 at 1:41 AM, Cody Koeninger  wrote:
>
>> Sure no prob.
>>
>> On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das 
>> wrote:
>>
>>> Given the interest, I am also inclining towards making it a public
>>> developer API. Maybe even experimental. Cody, mind submitting a patch?
>>>
>>>
>>> On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen  wrote:
>>>
 For what it's worth, I also use this class in an app, but it happens
 to be from Java code where it acts as if it's public. So no problem
 for my use case, but I suppose, another small vote for the usefulness
 of this class to the caller. I end up using getLatestLeaderOffsets to
 figure out how to initialize initial offsets.

 On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger 
 wrote:
 > I personally think KafkaCluster (or the equivalent) should be made
 public.
 > When I'm deploying spark I just sed out the private[spark] and
 rebuild.
 >
 > There's a general reluctance to make things public due to backwards
 > compatibility, but if enough people ask for it... ?
 >
 > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney 
 wrote:
 >>
 >> You can put a class in the org.apache.spark namespace to access
 anything
 >> that is private[spark]. You can then make enrichments there to access
 >> whatever you need. Just beware upgrade pain :)
 >>
 >>
 >> El martes, 6 de octubre de 2015, Erwan ALLAIN <
 eallain.po...@gmail.com>
 >> escribió:
 >>>
 >>> Hello,
 >>>
 >>> I'm currently testing spark streaming with kafka.
 >>> I'm creating DirectStream with KafkaUtils and everything's fine.
 However
 >>> I would like to use the signature where I can specify my own
 message handler
 >>> (to play with partition and offset). In this case, I need to manage
 >>> offset/partition by myself to fill fromOffsets argument.
 >>> I have found a Jira on this usecase
 >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been
 closed
 >>> telling that it's too specific.
 >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest
 and
 >>> OffsetRequest) but what I have to do is almost the same as the
 KafkaCluster
 >>> which is private.
 >>>
 >>> is it possible to :
 >>>  - add another signature in KafkaUtils ?
 >>>  - make KafkaCluster public ?
 >>>
 >>> or do you have any other srmart solution where I don't need to
 copy/paste
 >>> KafkaCluster ?
 >>>
 >>> Thanks.
 >>>
 >>> Regards,
 >>> Erwan ALLAIN
 >
 >

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org


>>>
>>
>

Re: does KafkaCluster can be public ?

[Erwan ALLAIN]

Thanks guys !

On Wed, Oct 7, 2015 at 1:41 AM, Cody Koeninger  wrote:

> Sure no prob.
>
> On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das  wrote:
>
>> Given the interest, I am also inclining towards making it a public
>> developer API. Maybe even experimental. Cody, mind submitting a patch?
>>
>>
>> On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen  wrote:
>>
>>> For what it's worth, I also use this class in an app, but it happens
>>> to be from Java code where it acts as if it's public. So no problem
>>> for my use case, but I suppose, another small vote for the usefulness
>>> of this class to the caller. I end up using getLatestLeaderOffsets to
>>> figure out how to initialize initial offsets.
>>>
>>> On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger 
>>> wrote:
>>> > I personally think KafkaCluster (or the equivalent) should be made
>>> public.
>>> > When I'm deploying spark I just sed out the private[spark] and rebuild.
>>> >
>>> > There's a general reluctance to make things public due to backwards
>>> > compatibility, but if enough people ask for it... ?
>>> >
>>> > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney 
>>> wrote:
>>> >>
>>> >> You can put a class in the org.apache.spark namespace to access
>>> anything
>>> >> that is private[spark]. You can then make enrichments there to access
>>> >> whatever you need. Just beware upgrade pain :)
>>> >>
>>> >>
>>> >> El martes, 6 de octubre de 2015, Erwan ALLAIN <
>>> eallain.po...@gmail.com>
>>> >> escribió:
>>> >>>
>>> >>> Hello,
>>> >>>
>>> >>> I'm currently testing spark streaming with kafka.
>>> >>> I'm creating DirectStream with KafkaUtils and everything's fine.
>>> However
>>> >>> I would like to use the signature where I can specify my own message
>>> handler
>>> >>> (to play with partition and offset). In this case, I need to manage
>>> >>> offset/partition by myself to fill fromOffsets argument.
>>> >>> I have found a Jira on this usecase
>>> >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been
>>> closed
>>> >>> telling that it's too specific.
>>> >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest
>>> and
>>> >>> OffsetRequest) but what I have to do is almost the same as the
>>> KafkaCluster
>>> >>> which is private.
>>> >>>
>>> >>> is it possible to :
>>> >>>  - add another signature in KafkaUtils ?
>>> >>>  - make KafkaCluster public ?
>>> >>>
>>> >>> or do you have any other srmart solution where I don't need to
>>> copy/paste
>>> >>> KafkaCluster ?
>>> >>>
>>> >>> Thanks.
>>> >>>
>>> >>> Regards,
>>> >>> Erwan ALLAIN
>>> >
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>
>

Re: does KafkaCluster can be public ?

[Ted Yu]

Or maybe annotate with @DeveloperApi

Cheers

On Tue, Oct 6, 2015 at 7:24 AM, Cody Koeninger  wrote:

> I personally think KafkaCluster (or the equivalent) should be made
> public.  When I'm deploying spark I just sed out the private[spark] and
> rebuild.
>
> There's a general reluctance to make things public due to backwards
> compatibility, but if enough people ask for it... ?
>
> On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney 
> wrote:
>
>> You can put a class in the org.apache.spark namespace to access anything
>> that is private[spark]. You can then make enrichments there to access
>> whatever you need. Just beware upgrade pain :)
>>
>>
>> El martes, 6 de octubre de 2015, Erwan ALLAIN 
>> escribió:
>>
>>> Hello,
>>>
>>> I'm currently testing spark streaming with kafka.
>>> I'm creating DirectStream with KafkaUtils and everything's fine. However
>>> I would like to use the signature where I can specify my own message
>>> handler (to play with partition and offset). In this case, I need to manage
>>> offset/partition by myself to fill fromOffsets argument.
>>> I have found a Jira on this usecase
>>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed
>>> telling that it's too specific.
>>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and
>>> OffsetRequest) but what I have to do is almost the same as the KafkaCluster
>>> which is private.
>>>
>>> is it possible to :
>>>  - add another signature in KafkaUtils ?
>>>  - make KafkaCluster public ?
>>>
>>> or do you have any other srmart solution where I don't need to
>>> copy/paste KafkaCluster ?
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Erwan ALLAIN
>>>
>>
>

Re: does KafkaCluster can be public ?

[Sean Owen]

For what it's worth, I also use this class in an app, but it happens
to be from Java code where it acts as if it's public. So no problem
for my use case, but I suppose, another small vote for the usefulness
of this class to the caller. I end up using getLatestLeaderOffsets to
figure out how to initialize initial offsets.

On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger  wrote:
> I personally think KafkaCluster (or the equivalent) should be made public.
> When I'm deploying spark I just sed out the private[spark] and rebuild.
>
> There's a general reluctance to make things public due to backwards
> compatibility, but if enough people ask for it... ?
>
> On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney  wrote:
>>
>> You can put a class in the org.apache.spark namespace to access anything
>> that is private[spark]. You can then make enrichments there to access
>> whatever you need. Just beware upgrade pain :)
>>
>>
>> El martes, 6 de octubre de 2015, Erwan ALLAIN 
>> escribió:
>>>
>>> Hello,
>>>
>>> I'm currently testing spark streaming with kafka.
>>> I'm creating DirectStream with KafkaUtils and everything's fine. However
>>> I would like to use the signature where I can specify my own message handler
>>> (to play with partition and offset). In this case, I need to manage
>>> offset/partition by myself to fill fromOffsets argument.
>>> I have found a Jira on this usecase
>>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed
>>> telling that it's too specific.
>>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and
>>> OffsetRequest) but what I have to do is almost the same as the KafkaCluster
>>> which is private.
>>>
>>> is it possible to :
>>>  - add another signature in KafkaUtils ?
>>>  - make KafkaCluster public ?
>>>
>>> or do you have any other srmart solution where I don't need to copy/paste
>>> KafkaCluster ?
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Erwan ALLAIN
>
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: does KafkaCluster can be public ?

[Cody Koeninger]

I personally think KafkaCluster (or the equivalent) should be made public.
When I'm deploying spark I just sed out the private[spark] and rebuild.

There's a general reluctance to make things public due to backwards
compatibility, but if enough people ask for it... ?

On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney  wrote:

> You can put a class in the org.apache.spark namespace to access anything
> that is private[spark]. You can then make enrichments there to access
> whatever you need. Just beware upgrade pain :)
>
>
> El martes, 6 de octubre de 2015, Erwan ALLAIN 
> escribió:
>
>> Hello,
>>
>> I'm currently testing spark streaming with kafka.
>> I'm creating DirectStream with KafkaUtils and everything's fine. However
>> I would like to use the signature where I can specify my own message
>> handler (to play with partition and offset). In this case, I need to manage
>> offset/partition by myself to fill fromOffsets argument.
>> I have found a Jira on this usecase
>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed
>> telling that it's too specific.
>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and
>> OffsetRequest) but what I have to do is almost the same as the KafkaCluster
>> which is private.
>>
>> is it possible to :
>>  - add another signature in KafkaUtils ?
>>  - make KafkaCluster public ?
>>
>> or do you have any other srmart solution where I don't need to copy/paste
>> KafkaCluster ?
>>
>> Thanks.
>>
>> Regards,
>> Erwan ALLAIN
>>
>

does KafkaCluster can be public ?

[Erwan ALLAIN]

Hello,

I'm currently testing spark streaming with kafka.
I'm creating DirectStream with KafkaUtils and everything's fine. However I
would like to use the signature where I can specify my own message handler
(to play with partition and offset). In this case, I need to manage
offset/partition by myself to fill fromOffsets argument.
I have found a Jira on this usecase
https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed
telling that it's too specific.
I'm aware that it can be done using kafka api (TopicMetaDataRequest and
OffsetRequest) but what I have to do is almost the same as the KafkaCluster
which is private.

is it possible to :
 - add another signature in KafkaUtils ?
 - make KafkaCluster public ?

or do you have any other srmart solution where I don't need to copy/paste
KafkaCluster ?

Thanks.

Regards,
Erwan ALLAIN

Re: does KafkaCluster can be public ?

[Tathagata Das]

Given the interest, I am also inclining towards making it a public
developer API. Maybe even experimental. Cody, mind submitting a patch?


On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen  wrote:

> For what it's worth, I also use this class in an app, but it happens
> to be from Java code where it acts as if it's public. So no problem
> for my use case, but I suppose, another small vote for the usefulness
> of this class to the caller. I end up using getLatestLeaderOffsets to
> figure out how to initialize initial offsets.
>
> On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger  wrote:
> > I personally think KafkaCluster (or the equivalent) should be made
> public.
> > When I'm deploying spark I just sed out the private[spark] and rebuild.
> >
> > There's a general reluctance to make things public due to backwards
> > compatibility, but if enough people ask for it... ?
> >
> > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney 
> wrote:
> >>
> >> You can put a class in the org.apache.spark namespace to access anything
> >> that is private[spark]. You can then make enrichments there to access
> >> whatever you need. Just beware upgrade pain :)
> >>
> >>
> >> El martes, 6 de octubre de 2015, Erwan ALLAIN 
> >> escribió:
> >>>
> >>> Hello,
> >>>
> >>> I'm currently testing spark streaming with kafka.
> >>> I'm creating DirectStream with KafkaUtils and everything's fine.
> However
> >>> I would like to use the signature where I can specify my own message
> handler
> >>> (to play with partition and offset). In this case, I need to manage
> >>> offset/partition by myself to fill fromOffsets argument.
> >>> I have found a Jira on this usecase
> >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been
> closed
> >>> telling that it's too specific.
> >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and
> >>> OffsetRequest) but what I have to do is almost the same as the
> KafkaCluster
> >>> which is private.
> >>>
> >>> is it possible to :
> >>>  - add another signature in KafkaUtils ?
> >>>  - make KafkaCluster public ?
> >>>
> >>> or do you have any other srmart solution where I don't need to
> copy/paste
> >>> KafkaCluster ?
> >>>
> >>> Thanks.
> >>>
> >>> Regards,
> >>> Erwan ALLAIN
> >
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: does KafkaCluster can be public ?

[Cody Koeninger]

Sure no prob.

On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das  wrote:

> Given the interest, I am also inclining towards making it a public
> developer API. Maybe even experimental. Cody, mind submitting a patch?
>
>
> On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen  wrote:
>
>> For what it's worth, I also use this class in an app, but it happens
>> to be from Java code where it acts as if it's public. So no problem
>> for my use case, but I suppose, another small vote for the usefulness
>> of this class to the caller. I end up using getLatestLeaderOffsets to
>> figure out how to initialize initial offsets.
>>
>> On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger 
>> wrote:
>> > I personally think KafkaCluster (or the equivalent) should be made
>> public.
>> > When I'm deploying spark I just sed out the private[spark] and rebuild.
>> >
>> > There's a general reluctance to make things public due to backwards
>> > compatibility, but if enough people ask for it... ?
>> >
>> > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney 
>> wrote:
>> >>
>> >> You can put a class in the org.apache.spark namespace to access
>> anything
>> >> that is private[spark]. You can then make enrichments there to access
>> >> whatever you need. Just beware upgrade pain :)
>> >>
>> >>
>> >> El martes, 6 de octubre de 2015, Erwan ALLAIN > >
>> >> escribió:
>> >>>
>> >>> Hello,
>> >>>
>> >>> I'm currently testing spark streaming with kafka.
>> >>> I'm creating DirectStream with KafkaUtils and everything's fine.
>> However
>> >>> I would like to use the signature where I can specify my own message
>> handler
>> >>> (to play with partition and offset). In this case, I need to manage
>> >>> offset/partition by myself to fill fromOffsets argument.
>> >>> I have found a Jira on this usecase
>> >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been
>> closed
>> >>> telling that it's too specific.
>> >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest
>> and
>> >>> OffsetRequest) but what I have to do is almost the same as the
>> KafkaCluster
>> >>> which is private.
>> >>>
>> >>> is it possible to :
>> >>>  - add another signature in KafkaUtils ?
>> >>>  - make KafkaCluster public ?
>> >>>
>> >>> or do you have any other srmart solution where I don't need to
>> copy/paste
>> >>> KafkaCluster ?
>> >>>
>> >>> Thanks.
>> >>>
>> >>> Regards,
>> >>> Erwan ALLAIN
>> >
>> >
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Re: does KafkaCluster can be public ?

[Jonathan Coveney]

You can put a class in the org.apache.spark namespace to access anything
that is private[spark]. You can then make enrichments there to access
whatever you need. Just beware upgrade pain :)

El martes, 6 de octubre de 2015, Erwan ALLAIN 
escribió:

> Hello,
>
> I'm currently testing spark streaming with kafka.
> I'm creating DirectStream with KafkaUtils and everything's fine. However I
> would like to use the signature where I can specify my own message handler
> (to play with partition and offset). In this case, I need to manage
> offset/partition by myself to fill fromOffsets argument.
> I have found a Jira on this usecase
> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed
> telling that it's too specific.
> I'm aware that it can be done using kafka api (TopicMetaDataRequest and
> OffsetRequest) but what I have to do is almost the same as the KafkaCluster
> which is private.
>
> is it possible to :
>  - add another signature in KafkaUtils ?
>  - make KafkaCluster public ?
>
> or do you have any other srmart solution where I don't need to copy/paste
> KafkaCluster ?
>
> Thanks.
>
> Regards,
> Erwan ALLAIN
>