Re: does KafkaCluster can be public ?
If anyone is interested in keeping tabs on it, the jira for this is https://issues.apache.org/jira/browse/SPARK-10963 On Wed, Oct 7, 2015 at 3:16 AM, Erwan ALLAIN wrote: > Thanks guys ! > > On Wed, Oct 7, 2015 at 1:41 AM, Cody Koeninger wrote: > >> Sure no prob. >> >> On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das >> wrote: >> >>> Given the interest, I am also inclining towards making it a public >>> developer API. Maybe even experimental. Cody, mind submitting a patch? >>> >>> >>> On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen wrote: >>> For what it's worth, I also use this class in an app, but it happens to be from Java code where it acts as if it's public. So no problem for my use case, but I suppose, another small vote for the usefulness of this class to the caller. I end up using getLatestLeaderOffsets to figure out how to initialize initial offsets. On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger wrote: > I personally think KafkaCluster (or the equivalent) should be made public. > When I'm deploying spark I just sed out the private[spark] and rebuild. > > There's a general reluctance to make things public due to backwards > compatibility, but if enough people ask for it... ? > > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney wrote: >> >> You can put a class in the org.apache.spark namespace to access anything >> that is private[spark]. You can then make enrichments there to access >> whatever you need. Just beware upgrade pain :) >> >> >> El martes, 6 de octubre de 2015, Erwan ALLAIN < eallain.po...@gmail.com> >> escribió: >>> >>> Hello, >>> >>> I'm currently testing spark streaming with kafka. >>> I'm creating DirectStream with KafkaUtils and everything's fine. However >>> I would like to use the signature where I can specify my own message handler >>> (to play with partition and offset). In this case, I need to manage >>> offset/partition by myself to fill fromOffsets argument. >>> I have found a Jira on this usecase >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed >>> telling that it's too specific. >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and >>> OffsetRequest) but what I have to do is almost the same as the KafkaCluster >>> which is private. >>> >>> is it possible to : >>> - add another signature in KafkaUtils ? >>> - make KafkaCluster public ? >>> >>> or do you have any other srmart solution where I don't need to copy/paste >>> KafkaCluster ? >>> >>> Thanks. >>> >>> Regards, >>> Erwan ALLAIN > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org >>> >> >
Re: does KafkaCluster can be public ?
Thanks guys ! On Wed, Oct 7, 2015 at 1:41 AM, Cody Koeninger wrote: > Sure no prob. > > On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das wrote: > >> Given the interest, I am also inclining towards making it a public >> developer API. Maybe even experimental. Cody, mind submitting a patch? >> >> >> On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen wrote: >> >>> For what it's worth, I also use this class in an app, but it happens >>> to be from Java code where it acts as if it's public. So no problem >>> for my use case, but I suppose, another small vote for the usefulness >>> of this class to the caller. I end up using getLatestLeaderOffsets to >>> figure out how to initialize initial offsets. >>> >>> On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger >>> wrote: >>> > I personally think KafkaCluster (or the equivalent) should be made >>> public. >>> > When I'm deploying spark I just sed out the private[spark] and rebuild. >>> > >>> > There's a general reluctance to make things public due to backwards >>> > compatibility, but if enough people ask for it... ? >>> > >>> > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney >>> wrote: >>> >> >>> >> You can put a class in the org.apache.spark namespace to access >>> anything >>> >> that is private[spark]. You can then make enrichments there to access >>> >> whatever you need. Just beware upgrade pain :) >>> >> >>> >> >>> >> El martes, 6 de octubre de 2015, Erwan ALLAIN < >>> eallain.po...@gmail.com> >>> >> escribió: >>> >>> >>> >>> Hello, >>> >>> >>> >>> I'm currently testing spark streaming with kafka. >>> >>> I'm creating DirectStream with KafkaUtils and everything's fine. >>> However >>> >>> I would like to use the signature where I can specify my own message >>> handler >>> >>> (to play with partition and offset). In this case, I need to manage >>> >>> offset/partition by myself to fill fromOffsets argument. >>> >>> I have found a Jira on this usecase >>> >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been >>> closed >>> >>> telling that it's too specific. >>> >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest >>> and >>> >>> OffsetRequest) but what I have to do is almost the same as the >>> KafkaCluster >>> >>> which is private. >>> >>> >>> >>> is it possible to : >>> >>> - add another signature in KafkaUtils ? >>> >>> - make KafkaCluster public ? >>> >>> >>> >>> or do you have any other srmart solution where I don't need to >>> copy/paste >>> >>> KafkaCluster ? >>> >>> >>> >>> Thanks. >>> >>> >>> >>> Regards, >>> >>> Erwan ALLAIN >>> > >>> > >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
Re: does KafkaCluster can be public ?
Sure no prob. On Tue, Oct 6, 2015 at 6:35 PM, Tathagata Das wrote: > Given the interest, I am also inclining towards making it a public > developer API. Maybe even experimental. Cody, mind submitting a patch? > > > On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen wrote: > >> For what it's worth, I also use this class in an app, but it happens >> to be from Java code where it acts as if it's public. So no problem >> for my use case, but I suppose, another small vote for the usefulness >> of this class to the caller. I end up using getLatestLeaderOffsets to >> figure out how to initialize initial offsets. >> >> On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger >> wrote: >> > I personally think KafkaCluster (or the equivalent) should be made >> public. >> > When I'm deploying spark I just sed out the private[spark] and rebuild. >> > >> > There's a general reluctance to make things public due to backwards >> > compatibility, but if enough people ask for it... ? >> > >> > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney >> wrote: >> >> >> >> You can put a class in the org.apache.spark namespace to access >> anything >> >> that is private[spark]. You can then make enrichments there to access >> >> whatever you need. Just beware upgrade pain :) >> >> >> >> >> >> El martes, 6 de octubre de 2015, Erwan ALLAIN > > >> >> escribió: >> >>> >> >>> Hello, >> >>> >> >>> I'm currently testing spark streaming with kafka. >> >>> I'm creating DirectStream with KafkaUtils and everything's fine. >> However >> >>> I would like to use the signature where I can specify my own message >> handler >> >>> (to play with partition and offset). In this case, I need to manage >> >>> offset/partition by myself to fill fromOffsets argument. >> >>> I have found a Jira on this usecase >> >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been >> closed >> >>> telling that it's too specific. >> >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest >> and >> >>> OffsetRequest) but what I have to do is almost the same as the >> KafkaCluster >> >>> which is private. >> >>> >> >>> is it possible to : >> >>> - add another signature in KafkaUtils ? >> >>> - make KafkaCluster public ? >> >>> >> >>> or do you have any other srmart solution where I don't need to >> copy/paste >> >>> KafkaCluster ? >> >>> >> >>> Thanks. >> >>> >> >>> Regards, >> >>> Erwan ALLAIN >> > >> > >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: does KafkaCluster can be public ?
Given the interest, I am also inclining towards making it a public developer API. Maybe even experimental. Cody, mind submitting a patch? On Tue, Oct 6, 2015 at 7:45 AM, Sean Owen wrote: > For what it's worth, I also use this class in an app, but it happens > to be from Java code where it acts as if it's public. So no problem > for my use case, but I suppose, another small vote for the usefulness > of this class to the caller. I end up using getLatestLeaderOffsets to > figure out how to initialize initial offsets. > > On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger wrote: > > I personally think KafkaCluster (or the equivalent) should be made > public. > > When I'm deploying spark I just sed out the private[spark] and rebuild. > > > > There's a general reluctance to make things public due to backwards > > compatibility, but if enough people ask for it... ? > > > > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney > wrote: > >> > >> You can put a class in the org.apache.spark namespace to access anything > >> that is private[spark]. You can then make enrichments there to access > >> whatever you need. Just beware upgrade pain :) > >> > >> > >> El martes, 6 de octubre de 2015, Erwan ALLAIN > >> escribió: > >>> > >>> Hello, > >>> > >>> I'm currently testing spark streaming with kafka. > >>> I'm creating DirectStream with KafkaUtils and everything's fine. > However > >>> I would like to use the signature where I can specify my own message > handler > >>> (to play with partition and offset). In this case, I need to manage > >>> offset/partition by myself to fill fromOffsets argument. > >>> I have found a Jira on this usecase > >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been > closed > >>> telling that it's too specific. > >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and > >>> OffsetRequest) but what I have to do is almost the same as the > KafkaCluster > >>> which is private. > >>> > >>> is it possible to : > >>> - add another signature in KafkaUtils ? > >>> - make KafkaCluster public ? > >>> > >>> or do you have any other srmart solution where I don't need to > copy/paste > >>> KafkaCluster ? > >>> > >>> Thanks. > >>> > >>> Regards, > >>> Erwan ALLAIN > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: does KafkaCluster can be public ?
For what it's worth, I also use this class in an app, but it happens to be from Java code where it acts as if it's public. So no problem for my use case, but I suppose, another small vote for the usefulness of this class to the caller. I end up using getLatestLeaderOffsets to figure out how to initialize initial offsets. On Tue, Oct 6, 2015 at 3:24 PM, Cody Koeninger wrote: > I personally think KafkaCluster (or the equivalent) should be made public. > When I'm deploying spark I just sed out the private[spark] and rebuild. > > There's a general reluctance to make things public due to backwards > compatibility, but if enough people ask for it... ? > > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney wrote: >> >> You can put a class in the org.apache.spark namespace to access anything >> that is private[spark]. You can then make enrichments there to access >> whatever you need. Just beware upgrade pain :) >> >> >> El martes, 6 de octubre de 2015, Erwan ALLAIN >> escribió: >>> >>> Hello, >>> >>> I'm currently testing spark streaming with kafka. >>> I'm creating DirectStream with KafkaUtils and everything's fine. However >>> I would like to use the signature where I can specify my own message handler >>> (to play with partition and offset). In this case, I need to manage >>> offset/partition by myself to fill fromOffsets argument. >>> I have found a Jira on this usecase >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed >>> telling that it's too specific. >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and >>> OffsetRequest) but what I have to do is almost the same as the KafkaCluster >>> which is private. >>> >>> is it possible to : >>> - add another signature in KafkaUtils ? >>> - make KafkaCluster public ? >>> >>> or do you have any other srmart solution where I don't need to copy/paste >>> KafkaCluster ? >>> >>> Thanks. >>> >>> Regards, >>> Erwan ALLAIN > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: does KafkaCluster can be public ?
Or maybe annotate with @DeveloperApi Cheers On Tue, Oct 6, 2015 at 7:24 AM, Cody Koeninger wrote: > I personally think KafkaCluster (or the equivalent) should be made > public. When I'm deploying spark I just sed out the private[spark] and > rebuild. > > There's a general reluctance to make things public due to backwards > compatibility, but if enough people ask for it... ? > > On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney > wrote: > >> You can put a class in the org.apache.spark namespace to access anything >> that is private[spark]. You can then make enrichments there to access >> whatever you need. Just beware upgrade pain :) >> >> >> El martes, 6 de octubre de 2015, Erwan ALLAIN >> escribió: >> >>> Hello, >>> >>> I'm currently testing spark streaming with kafka. >>> I'm creating DirectStream with KafkaUtils and everything's fine. However >>> I would like to use the signature where I can specify my own message >>> handler (to play with partition and offset). In this case, I need to manage >>> offset/partition by myself to fill fromOffsets argument. >>> I have found a Jira on this usecase >>> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed >>> telling that it's too specific. >>> I'm aware that it can be done using kafka api (TopicMetaDataRequest and >>> OffsetRequest) but what I have to do is almost the same as the KafkaCluster >>> which is private. >>> >>> is it possible to : >>> - add another signature in KafkaUtils ? >>> - make KafkaCluster public ? >>> >>> or do you have any other srmart solution where I don't need to >>> copy/paste KafkaCluster ? >>> >>> Thanks. >>> >>> Regards, >>> Erwan ALLAIN >>> >> >
Re: does KafkaCluster can be public ?
I personally think KafkaCluster (or the equivalent) should be made public. When I'm deploying spark I just sed out the private[spark] and rebuild. There's a general reluctance to make things public due to backwards compatibility, but if enough people ask for it... ? On Tue, Oct 6, 2015 at 6:51 AM, Jonathan Coveney wrote: > You can put a class in the org.apache.spark namespace to access anything > that is private[spark]. You can then make enrichments there to access > whatever you need. Just beware upgrade pain :) > > > El martes, 6 de octubre de 2015, Erwan ALLAIN > escribió: > >> Hello, >> >> I'm currently testing spark streaming with kafka. >> I'm creating DirectStream with KafkaUtils and everything's fine. However >> I would like to use the signature where I can specify my own message >> handler (to play with partition and offset). In this case, I need to manage >> offset/partition by myself to fill fromOffsets argument. >> I have found a Jira on this usecase >> https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed >> telling that it's too specific. >> I'm aware that it can be done using kafka api (TopicMetaDataRequest and >> OffsetRequest) but what I have to do is almost the same as the KafkaCluster >> which is private. >> >> is it possible to : >> - add another signature in KafkaUtils ? >> - make KafkaCluster public ? >> >> or do you have any other srmart solution where I don't need to copy/paste >> KafkaCluster ? >> >> Thanks. >> >> Regards, >> Erwan ALLAIN >> >
Re: does KafkaCluster can be public ?
You can put a class in the org.apache.spark namespace to access anything that is private[spark]. You can then make enrichments there to access whatever you need. Just beware upgrade pain :) El martes, 6 de octubre de 2015, Erwan ALLAIN escribió: > Hello, > > I'm currently testing spark streaming with kafka. > I'm creating DirectStream with KafkaUtils and everything's fine. However I > would like to use the signature where I can specify my own message handler > (to play with partition and offset). In this case, I need to manage > offset/partition by myself to fill fromOffsets argument. > I have found a Jira on this usecase > https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed > telling that it's too specific. > I'm aware that it can be done using kafka api (TopicMetaDataRequest and > OffsetRequest) but what I have to do is almost the same as the KafkaCluster > which is private. > > is it possible to : > - add another signature in KafkaUtils ? > - make KafkaCluster public ? > > or do you have any other srmart solution where I don't need to copy/paste > KafkaCluster ? > > Thanks. > > Regards, > Erwan ALLAIN >