Spring Integration Kafka support
Hello, Spring Integration extensions has a new module to support Kafka 0.8 integration. Currently, adapters for producer and the high level consumer are available. It is still in development. Here is the current snapshot of this support: https://github.com/SpringSource/spring-integration-extensions/tree/master/spring-integration-kafka If you are a Spring/Java user and also use Kafka 0.8, please take a look. Any feedback, suggestions, and PR's are welcomed!! Thank you, Soby Chacko
Re: kafka 0.8
Jun, That is great to hear. Looking forward to it. Thanks, Soby On Tue, Jun 11, 2013 at 12:20 PM, Jun Rao wrote: > Soby, > > Sorry for the delay. This week and early next week, all Kafka committers at > LinkedIn are busy with upgrading Kafka to 0.8 internally at LinkedIn. We > will try to start the release process as soon as we have things under > control. > > Thanks, > > Jun > > > On Mon, Jun 10, 2013 at 6:54 PM, Soby Chacko wrote: > > > Hello, > > > > Any updates on the 0.8 beta release? > > > > Soby Chacko > > > > > > On Tue, Jun 4, 2013 at 12:24 PM, Neha Narkhede > >wrote: > > > > > I was just about to send an update. We can release beta right away. > > > > > > Joe, > > > > > > I remember you were interested in helping out. Let me know if you are > > still > > > up for managing the release. > > > > > > Thanks, > > > Neha > > > On Jun 4, 2013 9:22 AM, "Soby Chacko" wrote: > > > > > > > Hello, > > > > > > > > Just wanted to know where we are with the beta release for 0.8? More > > > > importantly, is 0.8 going to be publicly available from a maven > > > repository? > > > > How about different versions of 0.8 built for different versions of > > > scala? > > > > (for example scala 2.8 vs 2.9 etc.) > > > > > > > > Much appreciated. > > > > > > > > Soby Chacko > > > > > > > > > >
Re: kafka 0.8
Hello, Any updates on the 0.8 beta release? Soby Chacko On Tue, Jun 4, 2013 at 12:24 PM, Neha Narkhede wrote: > I was just about to send an update. We can release beta right away. > > Joe, > > I remember you were interested in helping out. Let me know if you are still > up for managing the release. > > Thanks, > Neha > On Jun 4, 2013 9:22 AM, "Soby Chacko" wrote: > > > Hello, > > > > Just wanted to know where we are with the beta release for 0.8? More > > importantly, is 0.8 going to be publicly available from a maven > repository? > > How about different versions of 0.8 built for different versions of > scala? > > (for example scala 2.8 vs 2.9 etc.) > > > > Much appreciated. > > > > Soby Chacko > > >
kafka 0.8
Hello, Just wanted to know where we are with the beta release for 0.8? More importantly, is 0.8 going to be publicly available from a maven repository? How about different versions of 0.8 built for different versions of scala? (for example scala 2.8 vs 2.9 etc.) Much appreciated. Soby Chacko
Re: kafka 0.8 beta release status
Any status on the mavenization of 0.8? On Mon, Apr 29, 2013 at 11:04 AM, Jun Rao wrote: > Yes. > > Thanks, > > Jun > > > On Mon, Apr 29, 2013 at 6:13 AM, Oleg Ruchovets >wrote: > > > Does Kafka 0.8 become official beta? > > > > > > On Mon, Apr 29, 2013 at 8:52 AM, Jun Rao wrote: > > > > > We have updated the 0.8 documentation in our website ( > > > http://kafka.apache.org/index.html). Please review the docs. We have > the > > > following blockers for the 0.8 beta release: > > > > > > additional docs: > > > * examples of using the 0.8 high level consumer api > > > * description of additional 0.8 tools > > > > > > KAFKA-885 (sbt package builds two kafka jars) > > > > > > It would be great if people can help on the blockers. > > > > > > Thanks, > > > > > > Jun > > > > > >
Re: kafka key serializer
Hi Neha, I could be understanding it wrong. I am looking at https://issues.apache.org/jira/browse/KAFKA-544 and see the following comment. This patch does the following: 1. Change Encoder and Decoder to map between object and byte[] rather than between Message and object. 2. Require two encoders for the producer and two decoders for the consumer. This follows the same pattern as before: in the producer there is now serializer.class and key.serializer.class. *By default key.serializer takes the same value as serializer.class*. If no key is specified then this parameter is essentially ignored. In the consumer ConsumerConnector now requires two decoders, one for the key and one for the value. So, if this is true, then doesn't that mean key.serializer gets default to whatever value serializer.class takes, not the DefaultEncoder? Thanks again, Soby Chacko On Fri, Apr 12, 2013 at 11:38 AM, Neha Narkhede wrote: > It defaults both key and value serializer to DefaultEncoder, but you > can customize both independently through "key.serializer.class" and > "serializer.class" config options. > > Thanks, > Neha > > On Fri, Apr 12, 2013 at 8:33 AM, Soby Chacko wrote: > > Thanks for the reply. But, when I did some more research, it seems like > its > > using the same encoder for both. For example, if I provide > serializer.class > > explicitly, this serializer is used for both key and value. However, if I > > don't specify any serializer, then it appears that Kafka defaults to > > DefaultEncoder. Is that what you ment? > > > > Thanks again!! > > Soby Chacko > > > > > > On Wed, Apr 10, 2013 at 1:59 PM, Neha Narkhede >wrote: > > > >> It will use DefaultEncoder. > >> > >> Thanks, > >> Neha > >> > >> On Wed, Apr 10, 2013 at 8:27 AM, Soby Chacko > wrote: > >> > If I don't provide an explicit key serializer but a serializer class > (for > >> > value encoding), and then use a key in KeyedMessage, what will be the > >> > encoder used for key? Is it going to default to the same encoder used > for > >> > value or the DefaultEncoder? > >> > > >> > Thanks, > >> > Soby Chacko > >> >
Re: kafka key serializer
Thanks for the reply. But, when I did some more research, it seems like its using the same encoder for both. For example, if I provide serializer.class explicitly, this serializer is used for both key and value. However, if I don't specify any serializer, then it appears that Kafka defaults to DefaultEncoder. Is that what you ment? Thanks again!! Soby Chacko On Wed, Apr 10, 2013 at 1:59 PM, Neha Narkhede wrote: > It will use DefaultEncoder. > > Thanks, > Neha > > On Wed, Apr 10, 2013 at 8:27 AM, Soby Chacko wrote: > > If I don't provide an explicit key serializer but a serializer class (for > > value encoding), and then use a key in KeyedMessage, what will be the > > encoder used for key? Is it going to default to the same encoder used for > > value or the DefaultEncoder? > > > > Thanks, > > Soby Chacko >
kafka key serializer
If I don't provide an explicit key serializer but a serializer class (for value encoding), and then use a key in KeyedMessage, what will be the encoder used for key? Is it going to default to the same encoder used for value or the DefaultEncoder? Thanks, Soby Chacko
Re: kafka 0.8 release schedule?
Thanks!! On Thu, Apr 4, 2013 at 12:25 AM, Jun Rao wrote: > Soby, > > We plan to release 0.8 beta in Apr. Our build tool supports building on > different versions of scala and the default one is 2.8. > > Thanks, > > Jun > > > On Wed, Apr 3, 2013 at 4:02 PM, Soby Chacko wrote: > > > Hello, > > > > I am sure this question has been asked before. But, can someone tell me > the > > tentative release schedule for Kafka 0.8? And also, when its released, is > > it going to be based on Scala 2.9.2 or default to 2.8? > > > > Regards, > > Soby Chacko. > > >
kafka 0.8 release schedule?
Hello, I am sure this question has been asked before. But, can someone tell me the tentative release schedule for Kafka 0.8? And also, when its released, is it going to be based on Scala 2.9.2 or default to 2.8? Regards, Soby Chacko.
Re: Kafka 0.8 build artifacts
That will be awesome. Thanks, Soby Chacko On Wed, Mar 27, 2013 at 11:51 AM, Neha Narkhede wrote: > We are looking into the possibility of changing Kafka 0.8 to depend on > metrics 2.2.0 instead. This will allow us to mavenize Kafka. > > THanks, > Neha > > On Wed, Mar 27, 2013 at 8:45 AM, Soby Chacko wrote: > > Hello, > > > > Whats the likelihood of more changes to these two yammer metrics > libraries? > > Or are they going to stay the same for kafka 0.8? > > > > Regards, > > Soby Chacko > > > > On Tue, Mar 26, 2013 at 7:34 PM, Soby Chacko > wrote: > > > >> Thanks Dragos!! > >> > >> Is the zkclient 0.2 change is pushed? > >> > >> I will see what I can do with the metrics jars locally. > >> > >> Thanks again!! > >> > >> Soby Chacko. > >> > >> > >> On Tue, Mar 26, 2013 at 7:18 PM, Dragos Manolescu < > >> dragos.manole...@servicenow.com> wrote: > >> > >>> Hi Soby -- > >>> > >>> As far as I can tell the zkclient dependency is a leftover from 0.7.2. > >>> With the exception of the migration tool, which has a comment along > these > >>> lines but I haven't looked into 0.8 uses the 0.2 version of the library > >>> (available from maven). I submitted a patch to address this, it is in > the > >>> pipeline (http://j.mp/13uX6rs). > >>> > >>> It is not as simple with the metics library as there are significant > API > >>> changes between the 3.0.0-c0c8be71 used in the Kafka code and > >>> 3.0.0-SNAPSHOT. There's an open item on that (http://j.mp/10ccTnE); > >>> meanwhile the path of least resistance is to manually add the 2 metrics > >>> JARs to the class path. > >>> > >>> I have a fork that addresses the above and has a couple of other minor > >>> fixes. With the caveat that it hasn't been reviewed by the Kafka team > yore > >>> welcome to have a look: http://j.mp/ZsSld2 > >>> > >>> Welcome to the club ;) > >>> > >>> -Dragos > >>> > >>> On 3/26/13 3:28 PM, "Soby Chacko" wrote: > >>> > >>> >Hello, > >>> > > >>> >I built kafka 0.8 with scala 2.9.2. However, I ran into some issues > after > >>> >publishing the 0.8-SNAPSHOT into my local maven repository. From my > >>> >application, I simply depended on this new jar thinking that it would > >>> pull > >>> >all the other dependencies transitively from a public maven > repository. > >>> (I > >>> >look in a handful of repos in my build script) But, it could not find > the > >>> >following 3 jars from any repository: > >>> > > >>> >zkclient-20120522.jar > >>> >metrics-annotation-3.0.0-c0c8be71.jar > >>> >metrics-core-3.0.0-c0c8be71.jar > >>> > > >>> >I found these in the kafka 0.8 artifact that I built and was able to > >>> >manually copy them. Are there any plans to make these jars available > >>> >through any maven repositories, as they seem to contain custom > changes? > >>> > > >>> >Appreciate any help. > >>> > > >>> >Regards, > >>> >Soby Chacko > >>> > >>> > >> >
Re: Kafka 0.8 build artifacts
Hello, Whats the likelihood of more changes to these two yammer metrics libraries? Or are they going to stay the same for kafka 0.8? Regards, Soby Chacko On Tue, Mar 26, 2013 at 7:34 PM, Soby Chacko wrote: > Thanks Dragos!! > > Is the zkclient 0.2 change is pushed? > > I will see what I can do with the metrics jars locally. > > Thanks again!! > > Soby Chacko. > > > On Tue, Mar 26, 2013 at 7:18 PM, Dragos Manolescu < > dragos.manole...@servicenow.com> wrote: > >> Hi Soby -- >> >> As far as I can tell the zkclient dependency is a leftover from 0.7.2. >> With the exception of the migration tool, which has a comment along these >> lines but I haven't looked into 0.8 uses the 0.2 version of the library >> (available from maven). I submitted a patch to address this, it is in the >> pipeline (http://j.mp/13uX6rs). >> >> It is not as simple with the metics library as there are significant API >> changes between the 3.0.0-c0c8be71 used in the Kafka code and >> 3.0.0-SNAPSHOT. There's an open item on that (http://j.mp/10ccTnE); >> meanwhile the path of least resistance is to manually add the 2 metrics >> JARs to the class path. >> >> I have a fork that addresses the above and has a couple of other minor >> fixes. With the caveat that it hasn't been reviewed by the Kafka team yore >> welcome to have a look: http://j.mp/ZsSld2 >> >> Welcome to the club ;) >> >> -Dragos >> >> On 3/26/13 3:28 PM, "Soby Chacko" wrote: >> >> >Hello, >> > >> >I built kafka 0.8 with scala 2.9.2. However, I ran into some issues after >> >publishing the 0.8-SNAPSHOT into my local maven repository. From my >> >application, I simply depended on this new jar thinking that it would >> pull >> >all the other dependencies transitively from a public maven repository. >> (I >> >look in a handful of repos in my build script) But, it could not find the >> >following 3 jars from any repository: >> > >> >zkclient-20120522.jar >> >metrics-annotation-3.0.0-c0c8be71.jar >> >metrics-core-3.0.0-c0c8be71.jar >> > >> >I found these in the kafka 0.8 artifact that I built and was able to >> >manually copy them. Are there any plans to make these jars available >> >through any maven repositories, as they seem to contain custom changes? >> > >> >Appreciate any help. >> > >> >Regards, >> >Soby Chacko >> >> >
Re: Kafka 0.8 build artifacts
Thanks Dragos!! Is the zkclient 0.2 change is pushed? I will see what I can do with the metrics jars locally. Thanks again!! Soby Chacko. On Tue, Mar 26, 2013 at 7:18 PM, Dragos Manolescu < dragos.manole...@servicenow.com> wrote: > Hi Soby -- > > As far as I can tell the zkclient dependency is a leftover from 0.7.2. > With the exception of the migration tool, which has a comment along these > lines but I haven't looked into 0.8 uses the 0.2 version of the library > (available from maven). I submitted a patch to address this, it is in the > pipeline (http://j.mp/13uX6rs). > > It is not as simple with the metics library as there are significant API > changes between the 3.0.0-c0c8be71 used in the Kafka code and > 3.0.0-SNAPSHOT. There's an open item on that (http://j.mp/10ccTnE); > meanwhile the path of least resistance is to manually add the 2 metrics > JARs to the class path. > > I have a fork that addresses the above and has a couple of other minor > fixes. With the caveat that it hasn't been reviewed by the Kafka team yore > welcome to have a look: http://j.mp/ZsSld2 > > Welcome to the club ;) > > -Dragos > > On 3/26/13 3:28 PM, "Soby Chacko" wrote: > > >Hello, > > > >I built kafka 0.8 with scala 2.9.2. However, I ran into some issues after > >publishing the 0.8-SNAPSHOT into my local maven repository. From my > >application, I simply depended on this new jar thinking that it would pull > >all the other dependencies transitively from a public maven repository. (I > >look in a handful of repos in my build script) But, it could not find the > >following 3 jars from any repository: > > > >zkclient-20120522.jar > >metrics-annotation-3.0.0-c0c8be71.jar > >metrics-core-3.0.0-c0c8be71.jar > > > >I found these in the kafka 0.8 artifact that I built and was able to > >manually copy them. Are there any plans to make these jars available > >through any maven repositories, as they seem to contain custom changes? > > > >Appreciate any help. > > > >Regards, > >Soby Chacko > >
Kafka 0.8 build artifacts
Hello, I built kafka 0.8 with scala 2.9.2. However, I ran into some issues after publishing the 0.8-SNAPSHOT into my local maven repository. From my application, I simply depended on this new jar thinking that it would pull all the other dependencies transitively from a public maven repository. (I look in a handful of repos in my build script) But, it could not find the following 3 jars from any repository: zkclient-20120522.jar metrics-annotation-3.0.0-c0c8be71.jar metrics-core-3.0.0-c0c8be71.jar I found these in the kafka 0.8 artifact that I built and was able to manually copy them. Are there any plans to make these jars available through any maven repositories, as they seem to contain custom changes? Appreciate any help. Regards, Soby Chacko