Re: Kafka tuning - consultant work

[Stephen Boesch]

Please refrain from using this list as a job board. thank you.

Am Mo., 15. Apr. 2019 um 07:00 Uhr schrieb Manoj Murumkar <
manoj.murum...@gmail.com>:

> Damian,
>
> Let me know when we can talk. I have done extensive work on Kafka and run
> a boutique consulting firm specializes in this work. Let me know when it's
> a good time to talk.
>
> I am located in San Francisco Bay area and can do a bit of travel, if
> needed.
>
> Thanks,
>
> Manoj
> (650) 417.5847
>
> > On Apr 15, 2019, at 6:44 AM, Damian Martinez 
> wrote:
> >
> > Hi all, we are looking for one experienced developer with hands on
> > experience installing and configuring Kafka in cloud and hybrid
> > environments. This service is part of the backbone for data processing
> and
> > analytics information.
> >
> > *Primary Skills*: Kafka
> > *Duration:* 3 Months (possibility of extension)
> >
> > The ideal candidate has built reliable, scalable and maintainable data
> > intensive distributed systems using Kafka, managing and processing data
> at
> > the internet company scale. You should have hands on experience with
> >
> >   - Kafka, installation and tuning for high performance
> >   - troubleshooting issues in complex, distributed, multi-tier
> >   architectures.
> >   - CI / CD and containerization
> >   - active and passive monitoring, and alerting
> >
> > --
> >
> > *Damián Martínez Gelabert*
> >
> > Director, Software Engineering  |   Medallia
> >
> > Mobile: +54911 61412409
> >
> >     .>
> >   
> > 
> > *[image:
> https://www.flashbrand.me/send#feedbackto=dmarti...@medallia.com]
> > *
>

Re: [DISCUSS] Java 8 as a minimum requirement

[Stephen Boesch]

@Jeff Klukas What is the concern about scala 2.11 vs 2.12?   2.11 runs on
both java7 and java8

2016-06-16 14:12 GMT-07:00 Jeff Klukas :

> Would the move to Java 8 be for all modules? I'd have some concern about
> removing Java 7 compatibility for kafka-clients and for kafka streams
> (though less so since it's still so new). I don't know how hard it will be
> to transition a Scala 2.11 application to Scala 2.12. Are we comfortable
> with the idea of applications stuck on Scala 2.11 or otherwise unable to
> update to Java 8 not having access to new client releases?
>
> On Thu, Jun 16, 2016 at 5:05 PM, Philippe Derome 
> wrote:
>
> > I strongly support motion having difficulty running (Apache Kafka as
> > opposed to Confluent) Stream examples with JDK 8 today.
> > On 16 Jun 2016 4:46 p.m., "Ismael Juma"  wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on making Java 8 a minimum
> requirement
> > > for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> This
> > > is the first discussion on the topic so the idea is to understand how
> > > people feel about it. If people feel it's too soon, then we can pick up
> > the
> > > conversation again after Kafka 0.10.1.0. If the feedback is mostly
> > > positive, I will start a vote thread.
> > >
> > > Let's start with some dates. Java 7 hasn't received public updates
> since
> > > April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> > scheduled
> > > to be released in March 2017[3].
> > >
> > > The first argument for dropping support for Java 7 is that the last
> > public
> > > release by Oracle contains a large number of known security
> > > vulnerabilities. The effectiveness of Kafka's security features is
> > reduced
> > > if the underlying runtime is not itself secure.
> > >
> > > The second argument for moving to Java 8 is that it adds a number of
> > > compelling features:
> > >
> > > * Lambda expressions and method references (particularly useful for the
> > > Kafka Streams DSL)
> > > * Default methods (very useful for maintaining compatibility when
> adding
> > > methods to interfaces)
> > > * java.util.stream (helpful for making collection transformations more
> > > concise)
> > > * Lots of improvements to java.util.concurrent (CompletableFuture,
> > > DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> LongAccumulator)
> > > * Other nice things: SplittableRandom, Optional (and many others I have
> > not
> > > mentioned)
> > >
> > > The third argument is that it will simplify our testing matrix, we
> won't
> > > have to test with Java 7 any longer (this is particularly useful for
> > system
> > > tests that take hours to run). It will also make it easier to support
> > Scala
> > > 2.12, which requires Java 8.
> > >
> > > The fourth argument is that many other open-source projects have taken
> > the
> > > leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> 3[7],
> > > Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> will
> > > support Java 8 in the next version (although it will take a while
> before
> > > most phones will use that version sadly). This reduces (but does not
> > > eliminate) the chance that we would be the first project that would
> > cause a
> > > user to consider a Java upgrade.
> > >
> > > The main argument for not making the change is that a reasonable number
> > of
> > > users may still be using Java 7 by the time Kafka 0.10.1.0 is released.
> > > More specifically, we care about the subset who would be able to
> upgrade
> > to
> > > Kafka 0.10.1.0, but would not be able to upgrade the Java version. It
> > would
> > > be great if we could quantify this in some way.
> > >
> > > What do you think?
> > >
> > > Ismael
> > >
> > > [1] https://java.com/en/download/faq/java_7.xml
> > > [2] https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > [3] http://openjdk.java.net/projects/jdk9/
> > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > [5]
> https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > [8] https://webtide.com/jetty-9-3-features/
> > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > [10]
> > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > >
> >
>

Re: Dropping support for Scala 2.9.x

[Stephen Boesch]

+1  I was on a project that ended up not using kafka - and this was one
reason: there are many other third party libraries that do not even have
2.9 versions so the interdependencies did not work.

2015-03-27 7:34 GMT-07:00 Stevo Slavić ssla...@gmail.com:

 +1 for dropping 2.9.x support

 Kind regards,
 Stevo Slavic.

 On Fri, Mar 27, 2015 at 3:20 PM, Ismael Juma mli...@juma.me.uk wrote:

  Hi all,
 
  The Kafka build currently includes support for Scala 2.9, which means
 that
  it cannot take advantage of features introduced in Scala 2.10 or depend
 on
  libraries that require it.
 
  This restricts the solutions available while trying to solve existing
  issues. I was browsing JIRA looking for areas to contribute and I quickly
  ran into two issues where this is the case:
 
  * KAFKA-1351: String.format is very expensive in Scala could be solved
  nicely by using the String interpolation feature introduced in Scala
 2.10.
 
  * KAFKA-1595: Remove deprecated and slower scala JSON parser from
  kafka.consumer.TopicCount could be solved by using an existing JSON
  library, but both jackson-scala and play-json require 2.10 (argonaut
  supports Scala 2.9, but it brings other dependencies like scalaz). We can
  workaround this by writing our own code instead of using libraries, of
  course, but it's not ideal.
 
  Other features like Scala Futures and value classes would also be useful
 in
  some situations, I would think (for a more extensive list of new
 features,
  see
 
 
 http://scala-language.1934581.n4.nabble.com/Scala-2-10-0-now-available-td4634126.html
  ).
 
  Another pain point of supporting 2.9.x is that it doubles the number of
  build and test configurations required from 2 to 4 (because the 2.9.x
  series was not necessarily binary compatible).
 
  A strong argument for maintaining support for 2.9.x was the client
 library,
  but that has been rewritten in Java.
 
  It's also worth mentioning that Scala 2.9.1 was released in August 2011
  (more than 3.5 years ago) and the 2.9.x series hasn't received updates of
  any sort since early 2013. Scala 2.10.0, in turn, was released in January
  2013 (over 2 years ago) and 2.10.5, the last planned release in the
 2.10.x
  series, has been recently released (so even 2.10.x won't be receiving
  updates any longer).
 
  All in all, I think it would not be unreasonable to drop support for
 Scala
  2.9.x in a future release, but I may be missing something. What do others
  think?
 
  Ismael
 

Re: Why The Division Between Scala And Java

[Stephen Boesch]

so there will be both scala and java clients?  or will scala users simply
import the java libraries (which is after all not too bad)

2015-02-22 16:30 GMT-08:00 Guozhang Wang wangg...@gmail.com:

 Alex,

 Before 0.8 Kafka is written in Scala, and in 0.8.2 we are re-writing the
 clients in Java for better clients adoption while the server is still under
 Scala. The plan after the Java clients also includes migrating the common
 utils / error code / request formats to Java that will be used for both
 clients and servers.

 Guozhang

 On Sat, Feb 21, 2015 at 3:35 PM, Alex Melville amelvi...@g.hmc.edu
 wrote:

  Hi All,
 
 
  Why does the Kafka codebase contain both Scala and Java code? There are
  even some cases where the same class (i.e. javaapi.SimpleConsumer and
  kafka.consumer.SimpleConsumer). Is it just to allow a Scala developer to
  write Scala and a Java developer to use Java? We are trying to use the
  SimpleConsumer to gather more in-depth information about the partitions
 on
  a topic, and the documentation is fragmented and consequently it's
 unclear
  whether we have to use one of either Scala or Java, or if we have
 complete
  freedom to choose either (none of us have programmed in Scala, so we're
  inclined to choose the Java one).
 
  Functionally are both the scala and java codebases the same?
 
  Thanks in advance,
 
  Alex Melville
 



 --
 -- Guozhang

Re: Delete topic functionality can't use in 0.8.1

[Stephen Boesch]

Ryco, you are correct: delete topic is a new feature for 0.8.2

2015-02-09 19:53 GMT-08:00 Ryco Xiao ryco.x...@appcoachs.com:

 when I exec the delete command,return information is below:
 It mark the kafka-topic.sh not support the delete parameter.
 my package is compiled by myself.


 ​

Re: Poll: Producer/Consumer impl/language you use?

[Stephen Boesch]

The scala API going away would be a minus. As Koert mentioned we could use
the java api but it is less ..  well .. functional.

Kafka is included in the Spark examples and external modules and is popular
as a component of ecosystems on Spark (for which scala is the primary
language).

2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com:

 Hi,

 I don't have a good excuse here. :(
 I thought about including Scala, but for some reason didn't do it.  I see
 12-13% of people chose Other.  Do you think that is because I didn't
 include Scala?

 Also, is the Scala API reeally going away?

 Otis
 --
 Monitoring * Alerting * Anomaly Detection * Centralized Log Management
 Solr  Elasticsearch Support * http://sematext.com/


 On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote:

  no scala? although scala can indeed use the java api, its ugly we
  prefer to use the scala api (which i believe will go away unfortunately)
 
  On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic 
  otis.gospodne...@gmail.com wrote:
 
   Hi,
  
   I was wondering which implementations/languages people use for their
  Kafka
   Producer/Consumers not everyone is using the Java APIs.  So here's
 a
   1-question poll:
  
  
 http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/
  
   Will share the results in about a week when we have enough votes.
  
   Thanks!
   Otis
   --
   Monitoring * Alerting * Anomaly Detection * Centralized Log Management
   Solr  Elasticsearch Support * http://sematext.com/
  
 

Re: Right Tool

[Stephen Boesch]

Hi Patrick,   Kafka can be used at any scale including small ones
(initially anyways). The issues I ran into personally various issues with
ZooKeeper management and a bug in deleting topics (is that fixed yet?)  In
any case you might try out Kafka  - given its highly performant, scalable,
and flexible backbone.   After that you will have little worry about scale
- given Kafka's use within massive web scale deployments.

2014-09-12 15:18 GMT-07:00 Patrick Barker patrickbarke...@gmail.com:

 Hey, I'm new to kafka and I'm trying to get a handle on how it all works. I
 want to integrate polyglot persistence into my application. Kafka looks
 like exactly what I want just on a smaller scale. I am currently only
 dealing with about 2,000 users, which may grow,  but is kafka a good use
 case here, or is there another technology thats better suited?

 Thanks

Unable to get off the ground following the quick start section

[Stephen Boesch]

I have tried to use kafka both building form source via gradle as well as
un tgz-ing the tarball. Either way, none of the scripts work in the way
described in the Quick Start section.

Following is an example :

6:15:40/kafka:91 $bin/zookeeper-server-start.sh config/zookeeper.properties
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/zookeeper/server/quorum/QuorumPeerMain
Caused by: java.lang.ClassNotFoundException:
org.apache.zookeeper.server.quorum.QuorumPeerMain
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Re: Unable to get off the ground following the quick start section

[Stephen Boesch]

Thanks I will look into that resource.  Using the tgz is working it turns
out I was symlinked back to the src dir when encountering the issue.


2014-04-17 6:37 GMT-07:00 Bert Corderman b...@corderman.com:

 I cant speak to the quick start , however I found the following very
 helpful when I was getting started.


 http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/


 On Thu, Apr 17, 2014 at 9:22 AM, Stephen Boesch java...@gmail.com wrote:

  I have tried to use kafka both building form source via gradle as well as
  un tgz-ing the tarball. Either way, none of the scripts work in the way
  described in the Quick Start section.
 
  Following is an example :
 
  6:15:40/kafka:91 $bin/zookeeper-server-start.sh
 config/zookeeper.properties
  Exception in thread main java.lang.NoClassDefFoundError:
  org/apache/zookeeper/server/quorum/QuorumPeerMain
  Caused by: java.lang.ClassNotFoundException:
  org.apache.zookeeper.server.quorum.QuorumPeerMain
  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)