date:20170329

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Damian Guy

Hi Jon,

If you are able to get a handle on the store, i.e., via
KafkaStreams.store(...) and call fetch without any exceptions, then the
store is available.
The time params to fetch are the boundaries to search for windows for the
given key. They relate to the start time of the window, so if you did
fetch(key, t1, t2) - it will find all the windows for key that start in the
inclusive time range t1 - t2.

Are you running more than one instance? If yes, then you want to make sure
that you are querying the correct instance. For that you can use:
KafkaStreams.metadataForKey(...) to find the instance that has the key you
are looking for.

Thanks,
Damian

On Tue, 28 Mar 2017 at 22:37 Jon Yeargers  wrote:

> Im probing about trying to find a way to solve my aggregation -> db issue.
> Looking at the '.fetch()'  function Im wondering about the 'timeFrom' and
> 'timeTo' params as not a lot is mentioned about 'proper' usage.
>
> The test in
>
> https://github.com/confluentinc/examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/interactivequeries/WordCountInteractiveQueriesExampleTest.java#L200-L212
> makes it appear that the params are boundaries and that it will return an
> inclusive list of every key/window combination. Truth?
>
> My tests to this end haven't returned anything.
>
> Im watching the values coming out of the KTable so I can
> send them back as request params. What Ive tried:
>
> - Window.key(), Window.key().start() and Window.key().end()
> - Window.key(), (Window.key().start() - 1) and (Window.key().end() + 1)
> - Window.key(), 0 and Window.key().end()
> - Window.key(), 0 and (Window.key().end() + 1)
>
> None of these seem to hit anything in the StateStore.
>
> Is there a delay before Store values become available for '.fetch()'?
>

Kafka broker went down with "No space left on device" when there is a lot more

2017-03-29 Thread Nomar Morado

Two of my brokers went down today with the same error - see attachment for
details.

The device though is 55% free which is about over 100GB in space. The
entire kafka logs is only 1.3GB.


Any thoughts on what might be tripping this one?

I am using kafka 0.9.0.1.


Thanks

Nomar

Re: Kafka broker went down with "No space left on device" when there is a lot more

2017-03-29 Thread Manikumar

apache mailing list doesn't allow attachments. Can you paste the error
message here?
also check for free inodes on disk
https://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html

On Wed, Mar 29, 2017 at 5:48 PM, Nomar Morado 
wrote:

> Two of my brokers went down today with the same error - see attachment for
> details.
>
> The device though is 55% free which is about over 100GB in space. The
> entire kafka logs is only 1.3GB.
>
>
> Any thoughts on what might be tripping this one?
>
> I am using kafka 0.9.0.1.
>
>
> Thanks
>
> Nomar
>

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Jon Yeargers

Im only running one instance (locally) to keep things simple.

Reduction:

KTable, String> hourAggStore =
sourceStream.groupByKey().reduce(rowReducer,
TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
1000).until(70 * 60 * 1000L),
"HourAggStore");

then I get values to look for via:

hourAggStore.foreach((k, v) -> {
LogLine logLine = objectMapper.readValue(v, logLine.class);
LOGGER.debug("{}", k.key());
});

Ive kept it easy by requesting everything from 0 to
'System.currentTimeMillis()'. Retrieval is done using a snip from your
sample code "windowedByKey".

Requests are sent in via curl and output through the same channel. I pass
in the key and ask for any values.

Ive looked at the values passed in / out of the reduction function and they
look sane.

My assumption is that if a value shows up in the 'forEach' loop this
implies it exists in the StateStore. Accurate?

In fact, only about one in 10 requests actually return any values. No
errors - just no data.



On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy  wrote:

> Hi Jon,
>
> If you are able to get a handle on the store, i.e., via
> KafkaStreams.store(...) and call fetch without any exceptions, then the
> store is available.
> The time params to fetch are the boundaries to search for windows for the
> given key. They relate to the start time of the window, so if you did
> fetch(key, t1, t2) - it will find all the windows for key that start in the
> inclusive time range t1 - t2.
>
> Are you running more than one instance? If yes, then you want to make sure
> that you are querying the correct instance. For that you can use:
> KafkaStreams.metadataForKey(...) to find the instance that has the key you
> are looking for.
>
> Thanks,
> Damian
>
>
>
> On Tue, 28 Mar 2017 at 22:37 Jon Yeargers 
> wrote:
>
> > Im probing about trying to find a way to solve my aggregation -> db
> issue.
> > Looking at the '.fetch()'  function Im wondering about the 'timeFrom' and
> > 'timeTo' params as not a lot is mentioned about 'proper' usage.
> >
> > The test in
> >
> > https://github.com/confluentinc/examples/blob/
> master/kafka-streams/src/test/java/io/confluent/examples/
> streams/interactivequeries/WordCountInteractiveQueriesExa
> mpleTest.java#L200-L212
> > makes it appear that the params are boundaries and that it will return an
> > inclusive list of every key/window combination. Truth?
> >
> > My tests to this end haven't returned anything.
> >
> > Im watching the values coming out of the KTable so I can
> > send them back as request params. What Ive tried:
> >
> > - Window.key(), Window.key().start() and Window.key().end()
> > - Window.key(), (Window.key().start() - 1) and (Window.key().end() + 1)
> > - Window.key(), 0 and Window.key().end()
> > - Window.key(), 0 and (Window.key().end() + 1)
> >
> > None of these seem to hit anything in the StateStore.
> >
> > Is there a delay before Store values become available for '.fetch()'?
> >
>

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Michael Noll

Jon,

there's a related example, using a window store and a key-value store, at
https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/test/java/io/confluent/examples/streams/ValidateStateWithInteractiveQueriesLambdaIntegrationTest.java
(this is for Confluent 3.2 / Kafka 0.10.2).

-Michael



On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers 
wrote:

> Im only running one instance (locally) to keep things simple.
>
> Reduction:
>
> KTable, String> hourAggStore =
> sourceStream.groupByKey().reduce(rowReducer,
> TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
> 1000).until(70 * 60 * 1000L),
> "HourAggStore");
>
> then I get values to look for via:
>
> hourAggStore.foreach((k, v) -> {
> LogLine logLine = objectMapper.readValue(v, logLine.class);
> LOGGER.debug("{}", k.key());
> });
>
> Ive kept it easy by requesting everything from 0 to
> 'System.currentTimeMillis()'. Retrieval is done using a snip from your
> sample code "windowedByKey".
>
> Requests are sent in via curl and output through the same channel. I pass
> in the key and ask for any values.
>
> Ive looked at the values passed in / out of the reduction function and they
> look sane.
>
> My assumption is that if a value shows up in the 'forEach' loop this
> implies it exists in the StateStore. Accurate?
>
> In fact, only about one in 10 requests actually return any values. No
> errors - just no data.
>
>
>
> On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy  wrote:
>
> > Hi Jon,
> >
> > If you are able to get a handle on the store, i.e., via
> > KafkaStreams.store(...) and call fetch without any exceptions, then the
> > store is available.
> > The time params to fetch are the boundaries to search for windows for the
> > given key. They relate to the start time of the window, so if you did
> > fetch(key, t1, t2) - it will find all the windows for key that start in
> the
> > inclusive time range t1 - t2.
> >
> > Are you running more than one instance? If yes, then you want to make
> sure
> > that you are querying the correct instance. For that you can use:
> > KafkaStreams.metadataForKey(...) to find the instance that has the key
> you
> > are looking for.
> >
> > Thanks,
> > Damian
> >
> >
> >
> > On Tue, 28 Mar 2017 at 22:37 Jon Yeargers 
> > wrote:
> >
> > > Im probing about trying to find a way to solve my aggregation -> db
> > issue.
> > > Looking at the '.fetch()'  function Im wondering about the 'timeFrom'
> and
> > > 'timeTo' params as not a lot is mentioned about 'proper' usage.
> > >
> > > The test in
> > >
> > > https://github.com/confluentinc/examples/blob/
> > master/kafka-streams/src/test/java/io/confluent/examples/
> > streams/interactivequeries/WordCountInteractiveQueriesExa
> > mpleTest.java#L200-L212
> > > makes it appear that the params are boundaries and that it will return
> an
> > > inclusive list of every key/window combination. Truth?
> > >
> > > My tests to this end haven't returned anything.
> > >
> > > Im watching the values coming out of the KTable so I
> can
> > > send them back as request params. What Ive tried:
> > >
> > > - Window.key(), Window.key().start() and Window.key().end()
> > > - Window.key(), (Window.key().start() - 1) and (Window.key().end() + 1)
> > > - Window.key(), 0 and Window.key().end()
> > > - Window.key(), 0 and (Window.key().end() + 1)
> > >
> > > None of these seem to hit anything in the StateStore.
> > >
> > > Is there a delay before Store values become available for '.fetch()'?
> > >
> >
>

Kafka running but not listening to port 9092

2017-03-29 Thread Rafael Telles

Hello there!

I have two clusters of Kafka brokers, one of them (with 15 brokers + 3
Zookeeper servers) became sick (a lot of under-replicated partitions,
throwing a lot of NotEnoughReplicasExceptions). I logged in some of the
brokers that other couldn't connect to, and I found out that they were all
running their Kafka process, but they were not listening to the default TCP
port (9092) as expected:

root@dc3-kafka-02:/home/kafka/kafka_2.11-0.10.1.1# ps aux | grep kafka
root 14055 21.6 33.6 23001236 5513176 ? Sl Mar23 1866:20
/usr/lib/jvm/java-8-oracle/bin/java -Xms2G -Xmx6G -server -XX:+UseG1GC
-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC -Djava.awt.headless=true
-Xloggc:/home/kafka/kafka_2.11-0.10.1.1/bin/../logs/kafkaServer-gc.log
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=17264
-Dkafka.logs.dir=/home/kafka/kafka_2.11-0.10.1.1/bin/../logs
-Dlog4j.configuration=
file:/home/kafka/kafka_2.11-0.10.1.1/bin/../config/log4j.properties -cp
:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/argparse4j-0.5.0.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/connect-api-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/connect-file-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/connect-json-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/connect-runtime-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/guava-18.0.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/hk2-api-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/hk2-locator-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/hk2-utils-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jackson-annotations-2.6.0.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jackson-core-2.6.3.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jackson-databind-2.6.3.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/javassist-3.18.2-GA.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/javax.inject-1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/javax.inject-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-client-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-common-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-container-servlet-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-guava-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-media-jaxb-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jersey-server-2.22.2.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/jopt-simple-4.9.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka_2.11-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka_2.11-0.10.1.1-sources.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka_2.11-0.10.1.1-test-sources.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka-clients-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka-log4j-appender-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka-streams-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka-streams-examples-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/kafka-tools-0.10.1.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/log4j-1.2.17.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/lz4-1.3.0.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/metrics-core-2.2.0.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/raven-7.8.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/raven-log4j-7.8.1.jar:/home/kafka/kafka_2.11-0.10.1.1/bin/../libs/reflections-0.9.10.jar:/h

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Jon Yeargers

But if a key shows up in a KTable->forEach should it be available in the
StateStore (from the KTable)?

On Wed, Mar 29, 2017 at 6:31 AM, Michael Noll  wrote:

> Jon,
>
> there's a related example, using a window store and a key-value store, at
> https://github.com/confluentinc/examples/blob/3.
> 2.x/kafka-streams/src/test/java/io/confluent/examples/streams/
> ValidateStateWithInteractiveQueriesLambdaIntegrationTest.java
> (this is for Confluent 3.2 / Kafka 0.10.2).
>
> -Michael
>
>
>
> On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers 
> wrote:
>
> > Im only running one instance (locally) to keep things simple.
> >
> > Reduction:
> >
> > KTable, String> hourAggStore =
> > sourceStream.groupByKey().reduce(rowReducer,
> > TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
> > 1000).until(70 * 60 * 1000L),
> > "HourAggStore");
> >
> > then I get values to look for via:
> >
> > hourAggStore.foreach((k, v) -> {
> > LogLine logLine = objectMapper.readValue(v,
> logLine.class);
> > LOGGER.debug("{}", k.key());
> > });
> >
> > Ive kept it easy by requesting everything from 0 to
> > 'System.currentTimeMillis()'. Retrieval is done using a snip from your
> > sample code "windowedByKey".
> >
> > Requests are sent in via curl and output through the same channel. I pass
> > in the key and ask for any values.
> >
> > Ive looked at the values passed in / out of the reduction function and
> they
> > look sane.
> >
> > My assumption is that if a value shows up in the 'forEach' loop this
> > implies it exists in the StateStore. Accurate?
> >
> > In fact, only about one in 10 requests actually return any values. No
> > errors - just no data.
> >
> >
> >
> > On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy 
> wrote:
> >
> > > Hi Jon,
> > >
> > > If you are able to get a handle on the store, i.e., via
> > > KafkaStreams.store(...) and call fetch without any exceptions, then the
> > > store is available.
> > > The time params to fetch are the boundaries to search for windows for
> the
> > > given key. They relate to the start time of the window, so if you did
> > > fetch(key, t1, t2) - it will find all the windows for key that start in
> > the
> > > inclusive time range t1 - t2.
> > >
> > > Are you running more than one instance? If yes, then you want to make
> > sure
> > > that you are querying the correct instance. For that you can use:
> > > KafkaStreams.metadataForKey(...) to find the instance that has the key
> > you
> > > are looking for.
> > >
> > > Thanks,
> > > Damian
> > >
> > >
> > >
> > > On Tue, 28 Mar 2017 at 22:37 Jon Yeargers 
> > > wrote:
> > >
> > > > Im probing about trying to find a way to solve my aggregation -> db
> > > issue.
> > > > Looking at the '.fetch()'  function Im wondering about the 'timeFrom'
> > and
> > > > 'timeTo' params as not a lot is mentioned about 'proper' usage.
> > > >
> > > > The test in
> > > >
> > > > https://github.com/confluentinc/examples/blob/
> > > master/kafka-streams/src/test/java/io/confluent/examples/
> > > streams/interactivequeries/WordCountInteractiveQueriesExa
> > > mpleTest.java#L200-L212
> > > > makes it appear that the params are boundaries and that it will
> return
> > an
> > > > inclusive list of every key/window combination. Truth?
> > > >
> > > > My tests to this end haven't returned anything.
> > > >
> > > > Im watching the values coming out of the KTable so I
> > can
> > > > send them back as request params. What Ive tried:
> > > >
> > > > - Window.key(), Window.key().start() and Window.key().end()
> > > > - Window.key(), (Window.key().start() - 1) and (Window.key().end() +
> 1)
> > > > - Window.key(), 0 and Window.key().end()
> > > > - Window.key(), 0 and (Window.key().end() + 1)
> > > >
> > > > None of these seem to hit anything in the StateStore.
> > > >
> > > > Is there a delay before Store values become available for '.fetch()'?
> > > >
> > >
> >
>

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Jon Yeargers

To be a bit more specific:

If I call this: KTable kt =
sourceStream.groupByKey().reduce(..., "somekeystore");

and then call this:

kt.forEach()-> ...

Can I assume that everything that comes out will be available in
"somekeystore"? If not, what subset should I expect to find there?

On Wed, Mar 29, 2017 at 8:34 AM, Jon Yeargers 
wrote:

> But if a key shows up in a KTable->forEach should it be available in the
> StateStore (from the KTable)?
>
> On Wed, Mar 29, 2017 at 6:31 AM, Michael Noll 
> wrote:
>
>> Jon,
>>
>> there's a related example, using a window store and a key-value store, at
>> https://github.com/confluentinc/examples/blob/3.2.x/kafka-
>> streams/src/test/java/io/confluent/examples/streams/Val
>> idateStateWithInteractiveQueriesLambdaIntegrationTest.java
>> (this is for Confluent 3.2 / Kafka 0.10.2).
>>
>> -Michael
>>
>>
>>
>> On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers 
>> wrote:
>>
>> > Im only running one instance (locally) to keep things simple.
>> >
>> > Reduction:
>> >
>> > KTable, String> hourAggStore =
>> > sourceStream.groupByKey().reduce(rowReducer,
>> > TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
>> > 1000).until(70 * 60 * 1000L),
>> > "HourAggStore");
>> >
>> > then I get values to look for via:
>> >
>> > hourAggStore.foreach((k, v) -> {
>> > LogLine logLine = objectMapper.readValue(v,
>> logLine.class);
>> > LOGGER.debug("{}", k.key());
>> > });
>> >
>> > Ive kept it easy by requesting everything from 0 to
>> > 'System.currentTimeMillis()'. Retrieval is done using a snip from your
>> > sample code "windowedByKey".
>> >
>> > Requests are sent in via curl and output through the same channel. I
>> pass
>> > in the key and ask for any values.
>> >
>> > Ive looked at the values passed in / out of the reduction function and
>> they
>> > look sane.
>> >
>> > My assumption is that if a value shows up in the 'forEach' loop this
>> > implies it exists in the StateStore. Accurate?
>> >
>> > In fact, only about one in 10 requests actually return any values. No
>> > errors - just no data.
>> >
>> >
>> >
>> > On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy 
>> wrote:
>> >
>> > > Hi Jon,
>> > >
>> > > If you are able to get a handle on the store, i.e., via
>> > > KafkaStreams.store(...) and call fetch without any exceptions, then
>> the
>> > > store is available.
>> > > The time params to fetch are the boundaries to search for windows for
>> the
>> > > given key. They relate to the start time of the window, so if you did
>> > > fetch(key, t1, t2) - it will find all the windows for key that start
>> in
>> > the
>> > > inclusive time range t1 - t2.
>> > >
>> > > Are you running more than one instance? If yes, then you want to make
>> > sure
>> > > that you are querying the correct instance. For that you can use:
>> > > KafkaStreams.metadataForKey(...) to find the instance that has the
>> key
>> > you
>> > > are looking for.
>> > >
>> > > Thanks,
>> > > Damian
>> > >
>> > >
>> > >
>> > > On Tue, 28 Mar 2017 at 22:37 Jon Yeargers 
>> > > wrote:
>> > >
>> > > > Im probing about trying to find a way to solve my aggregation -> db
>> > > issue.
>> > > > Looking at the '.fetch()'  function Im wondering about the
>> 'timeFrom'
>> > and
>> > > > 'timeTo' params as not a lot is mentioned about 'proper' usage.
>> > > >
>> > > > The test in
>> > > >
>> > > > https://github.com/confluentinc/examples/blob/
>> > > master/kafka-streams/src/test/java/io/confluent/examples/
>> > > streams/interactivequeries/WordCountInteractiveQueriesExa
>> > > mpleTest.java#L200-L212
>> > > > makes it appear that the params are boundaries and that it will
>> return
>> > an
>> > > > inclusive list of every key/window combination. Truth?
>> > > >
>> > > > My tests to this end haven't returned anything.
>> > > >
>> > > > Im watching the values coming out of the KTable so I
>> > can
>> > > > send them back as request params. What Ive tried:
>> > > >
>> > > > - Window.key(), Window.key().start() and Window.key().end()
>> > > > - Window.key(), (Window.key().start() - 1) and (Window.key().end()
>> + 1)
>> > > > - Window.key(), 0 and Window.key().end()
>> > > > - Window.key(), 0 and (Window.key().end() + 1)
>> > > >
>> > > > None of these seem to hit anything in the StateStore.
>> > > >
>> > > > Is there a delay before Store values become available for
>> '.fetch()'?
>> > > >
>> > >
>> >
>>
>
>

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Damian Guy

Jon,

You should be able to query anything that has not expired, i.e., based on
TimeWindows.until(..).

Thanks,
Damian

On Wed, 29 Mar 2017 at 17:24 Jon Yeargers  wrote:

> To be a bit more specific:
>
> If I call this: KTable kt =
> sourceStream.groupByKey().reduce(..., "somekeystore");
>
> and then call this:
>
> kt.forEach()-> ...
>
> Can I assume that everything that comes out will be available in
> "somekeystore"? If not, what subset should I expect to find there?
>
> On Wed, Mar 29, 2017 at 8:34 AM, Jon Yeargers 
> wrote:
>
> > But if a key shows up in a KTable->forEach should it be available in the
> > StateStore (from the KTable)?
> >
> > On Wed, Mar 29, 2017 at 6:31 AM, Michael Noll 
> > wrote:
> >
> >> Jon,
> >>
> >> there's a related example, using a window store and a key-value store,
> at
> >> https://github.com/confluentinc/examples/blob/3.2.x/kafka-
> >> streams/src/test/java/io/confluent/examples/streams/Val
> >> idateStateWithInteractiveQueriesLambdaIntegrationTest.java
> >> (this is for Confluent 3.2 / Kafka 0.10.2).
> >>
> >> -Michael
> >>
> >>
> >>
> >> On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers  >
> >> wrote:
> >>
> >> > Im only running one instance (locally) to keep things simple.
> >> >
> >> > Reduction:
> >> >
> >> > KTable, String> hourAggStore =
> >> > sourceStream.groupByKey().reduce(rowReducer,
> >> > TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
> >> > 1000).until(70 * 60 * 1000L),
> >> > "HourAggStore");
> >> >
> >> > then I get values to look for via:
> >> >
> >> > hourAggStore.foreach((k, v) -> {
> >> > LogLine logLine = objectMapper.readValue(v,
> >> logLine.class);
> >> > LOGGER.debug("{}", k.key());
> >> > });
> >> >
> >> > Ive kept it easy by requesting everything from 0 to
> >> > 'System.currentTimeMillis()'. Retrieval is done using a snip from your
> >> > sample code "windowedByKey".
> >> >
> >> > Requests are sent in via curl and output through the same channel. I
> >> pass
> >> > in the key and ask for any values.
> >> >
> >> > Ive looked at the values passed in / out of the reduction function and
> >> they
> >> > look sane.
> >> >
> >> > My assumption is that if a value shows up in the 'forEach' loop this
> >> > implies it exists in the StateStore. Accurate?
> >> >
> >> > In fact, only about one in 10 requests actually return any values. No
> >> > errors - just no data.
> >> >
> >> >
> >> >
> >> > On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy 
> >> wrote:
> >> >
> >> > > Hi Jon,
> >> > >
> >> > > If you are able to get a handle on the store, i.e., via
> >> > > KafkaStreams.store(...) and call fetch without any exceptions, then
> >> the
> >> > > store is available.
> >> > > The time params to fetch are the boundaries to search for windows
> for
> >> the
> >> > > given key. They relate to the start time of the window, so if you
> did
> >> > > fetch(key, t1, t2) - it will find all the windows for key that start
> >> in
> >> > the
> >> > > inclusive time range t1 - t2.
> >> > >
> >> > > Are you running more than one instance? If yes, then you want to
> make
> >> > sure
> >> > > that you are querying the correct instance. For that you can use:
> >> > > KafkaStreams.metadataForKey(...) to find the instance that has the
> >> key
> >> > you
> >> > > are looking for.
> >> > >
> >> > > Thanks,
> >> > > Damian
> >> > >
> >> > >
> >> > >
> >> > > On Tue, 28 Mar 2017 at 22:37 Jon Yeargers  >
> >> > > wrote:
> >> > >
> >> > > > Im probing about trying to find a way to solve my aggregation ->
> db
> >> > > issue.
> >> > > > Looking at the '.fetch()'  function Im wondering about the
> >> 'timeFrom'
> >> > and
> >> > > > 'timeTo' params as not a lot is mentioned about 'proper' usage.
> >> > > >
> >> > > > The test in
> >> > > >
> >> > > > https://github.com/confluentinc/examples/blob/
> >> > > master/kafka-streams/src/test/java/io/confluent/examples/
> >> > > streams/interactivequeries/WordCountInteractiveQueriesExa
> >> > > mpleTest.java#L200-L212
> >> > > > makes it appear that the params are boundaries and that it will
> >> return
> >> > an
> >> > > > inclusive list of every key/window combination. Truth?
> >> > > >
> >> > > > My tests to this end haven't returned anything.
> >> > > >
> >> > > > Im watching the values coming out of the KTable
> so I
> >> > can
> >> > > > send them back as request params. What Ive tried:
> >> > > >
> >> > > > - Window.key(), Window.key().start() and Window.key().end()
> >> > > > - Window.key(), (Window.key().start() - 1) and (Window.key().end()
> >> + 1)
> >> > > > - Window.key(), 0 and Window.key().end()
> >> > > > - Window.key(), 0 and (Window.key().end() + 1)
> >> > > >
> >> > > > None of these seem to hit anything in the StateStore.
> >> > > >
> >> > > > Is there a delay before Store values become available for
> >> '.fetch()'?
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

ThoughWorks Tech Radar: Assess Kafka Streams

2017-03-29 Thread Jan Filipiak


Regardless of how usefull you find the tech radar.

Well deserved! even though we all here agree that trial or adopt is in reach

https://www.thoughtworks.com/radar/platforms/kafka-streams

Best Jan

Re: ThoughWorks Tech Radar: Assess Kafka Streams

2017-03-29 Thread Eno Thereska

Thanks for the heads up Jan!

Eno

> On 29 Mar 2017, at 19:08, Jan Filipiak  wrote:
> 
> Regardless of how usefull you find the tech radar.
> 
> Well deserved! even though we all here agree that trial or adopt is in reach
> 
> https://www.thoughtworks.com/radar/platforms/kafka-streams
> 
> Best Jan
> 
>

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Jon Yeargers

So '.until()' is based on clock time / elapsed time (IE record age) /
something else?

The fact that Im seeing lots of records come through that can't be found in
the Store - these are 'old' and already expired?

Going forward - it would be useful to have different forms of '.until()' so
one could consume old records (EG if one was catching up from lag) without
having to worry about them immediately disappearing.

On Wed, Mar 29, 2017 at 10:37 AM, Damian Guy  wrote:

> Jon,
>
> You should be able to query anything that has not expired, i.e., based on
> TimeWindows.until(..).
>
> Thanks,
> Damian
>
> On Wed, 29 Mar 2017 at 17:24 Jon Yeargers 
> wrote:
>
> > To be a bit more specific:
> >
> > If I call this: KTable kt =
> > sourceStream.groupByKey().reduce(..., "somekeystore");
> >
> > and then call this:
> >
> > kt.forEach()-> ...
> >
> > Can I assume that everything that comes out will be available in
> > "somekeystore"? If not, what subset should I expect to find there?
> >
> > On Wed, Mar 29, 2017 at 8:34 AM, Jon Yeargers 
> > wrote:
> >
> > > But if a key shows up in a KTable->forEach should it be available in
> the
> > > StateStore (from the KTable)?
> > >
> > > On Wed, Mar 29, 2017 at 6:31 AM, Michael Noll 
> > > wrote:
> > >
> > >> Jon,
> > >>
> > >> there's a related example, using a window store and a key-value store,
> > at
> > >> https://github.com/confluentinc/examples/blob/3.2.x/kafka-
> > >> streams/src/test/java/io/confluent/examples/streams/Val
> > >> idateStateWithInteractiveQueriesLambdaIntegrationTest.java
> > >> (this is for Confluent 3.2 / Kafka 0.10.2).
> > >>
> > >> -Michael
> > >>
> > >>
> > >>
> > >> On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers <
> jon.yearg...@cedexis.com
> > >
> > >> wrote:
> > >>
> > >> > Im only running one instance (locally) to keep things simple.
> > >> >
> > >> > Reduction:
> > >> >
> > >> > KTable, String> hourAggStore =
> > >> > sourceStream.groupByKey().reduce(rowReducer,
> > >> > TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
> > >> > 1000).until(70 * 60 * 1000L),
> > >> > "HourAggStore");
> > >> >
> > >> > then I get values to look for via:
> > >> >
> > >> > hourAggStore.foreach((k, v) -> {
> > >> > LogLine logLine = objectMapper.readValue(v,
> > >> logLine.class);
> > >> > LOGGER.debug("{}", k.key());
> > >> > });
> > >> >
> > >> > Ive kept it easy by requesting everything from 0 to
> > >> > 'System.currentTimeMillis()'. Retrieval is done using a snip from
> your
> > >> > sample code "windowedByKey".
> > >> >
> > >> > Requests are sent in via curl and output through the same channel. I
> > >> pass
> > >> > in the key and ask for any values.
> > >> >
> > >> > Ive looked at the values passed in / out of the reduction function
> and
> > >> they
> > >> > look sane.
> > >> >
> > >> > My assumption is that if a value shows up in the 'forEach' loop this
> > >> > implies it exists in the StateStore. Accurate?
> > >> >
> > >> > In fact, only about one in 10 requests actually return any values.
> No
> > >> > errors - just no data.
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy 
> > >> wrote:
> > >> >
> > >> > > Hi Jon,
> > >> > >
> > >> > > If you are able to get a handle on the store, i.e., via
> > >> > > KafkaStreams.store(...) and call fetch without any exceptions,
> then
> > >> the
> > >> > > store is available.
> > >> > > The time params to fetch are the boundaries to search for windows
> > for
> > >> the
> > >> > > given key. They relate to the start time of the window, so if you
> > did
> > >> > > fetch(key, t1, t2) - it will find all the windows for key that
> start
> > >> in
> > >> > the
> > >> > > inclusive time range t1 - t2.
> > >> > >
> > >> > > Are you running more than one instance? If yes, then you want to
> > make
> > >> > sure
> > >> > > that you are querying the correct instance. For that you can use:
> > >> > > KafkaStreams.metadataForKey(...) to find the instance that has
> the
> > >> key
> > >> > you
> > >> > > are looking for.
> > >> > >
> > >> > > Thanks,
> > >> > > Damian
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Tue, 28 Mar 2017 at 22:37 Jon Yeargers <
> jon.yearg...@cedexis.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Im probing about trying to find a way to solve my aggregation ->
> > db
> > >> > > issue.
> > >> > > > Looking at the '.fetch()'  function Im wondering about the
> > >> 'timeFrom'
> > >> > and
> > >> > > > 'timeTo' params as not a lot is mentioned about 'proper' usage.
> > >> > > >
> > >> > > > The test in
> > >> > > >
> > >> > > > https://github.com/confluentinc/examples/blob/
> > >> > > master/kafka-streams/src/test/java/io/confluent/examples/
> > >> > > streams/interactivequeries/WordCountInteractiveQueriesExa
> > >> > > mpleTest.java#L200-L212
> > >> > > > makes it appear that the params are boundaries and that it will
> > >> return
> > >> > an
> > >> > > > inclusive list of

weird SerializationException when consumer is fetching and parsing record in streams application

2017-03-29 Thread Sachin Mittal

Hi,
This is for first time we are getting a weird exception.
After this the streams caches.

Only work around is to manually seek and commit offset to a greater number
and we are needing this manual intervention again and again.

Any idea what is causing it and how can we circumvent this.

Note this error happens in both cases when 10.2 client or 10.1.1 client
connect to kafka server 10.1.1

So this does not looks like version issue.

Also we have following setting
message.max.bytes=513
ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, "5048576"
ProducerConfig.MAX_REQUEST_SIZE_CONFIG, "5048576"

Rest is all default and also increasing the value for
ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG did not help.

Stack trace below.

Thanks
Sachin


org.apache.kafka.common.errors.SerializationException: Error deserializing
key/value for partition advice-stream-6 at offset 45153795
java.lang.IllegalArgumentException: null
  at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_122-ea]
  at org.apache.kafka.common.utils.Utils.sizeDelimited(Utils.java:791)
~[kafka-clients-0.10.2.0.jar:na]
  at org.apache.kafka.common.record.Record.value(Record.java:268)
~[kafka-clients-0.10.2.0.jar:na]
  at 
org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:867)
~[kafka-clients-0.10.2.0.jar:na]
  at org.apache.kafka.clients.consumer.internals.Fetcher.
parseCompletedFetch(Fetcher.java:775) ~[kafka-clients-0.10.2.0.jar:na]
  at org.apache.kafka.clients.consumer.internals.Fetcher.
fetchedRecords(Fetcher.java:473) ~[kafka-clients-0.10.2.0.jar:na]
  at org.apache.kafka.clients.consumer.KafkaConsumer.
pollOnce(KafkaConsumer.java:1062) ~[kafka-clients-0.10.2.0.jar:na]
  at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
~[kafka-clients-0.10.2.0.jar:na]
  at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:592)
~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na]
  at org.apache.kafka.streams.processor.internals.
StreamThread.run(StreamThread.java:378) ~[kafka-streams-0.10.2.1-
SNAPSHOT.jar:na]

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Matthias J. Sax

It's based in "stream time", ie, the internally tracked progress based
on the timestamps return by TimestampExtractor.

-Matthias

On 3/29/17 12:52 PM, Jon Yeargers wrote:
> So '.until()' is based on clock time / elapsed time (IE record age) /
> something else?
> 
> The fact that Im seeing lots of records come through that can't be found in
> the Store - these are 'old' and already expired?
> 
> Going forward - it would be useful to have different forms of '.until()' so
> one could consume old records (EG if one was catching up from lag) without
> having to worry about them immediately disappearing.
> 
> On Wed, Mar 29, 2017 at 10:37 AM, Damian Guy  wrote:
> 
>> Jon,
>>
>> You should be able to query anything that has not expired, i.e., based on
>> TimeWindows.until(..).
>>
>> Thanks,
>> Damian
>>
>> On Wed, 29 Mar 2017 at 17:24 Jon Yeargers 
>> wrote:
>>
>>> To be a bit more specific:
>>>
>>> If I call this: KTable kt =
>>> sourceStream.groupByKey().reduce(..., "somekeystore");
>>>
>>> and then call this:
>>>
>>> kt.forEach()-> ...
>>>
>>> Can I assume that everything that comes out will be available in
>>> "somekeystore"? If not, what subset should I expect to find there?
>>>
>>> On Wed, Mar 29, 2017 at 8:34 AM, Jon Yeargers 
>>> wrote:
>>>
 But if a key shows up in a KTable->forEach should it be available in
>> the
 StateStore (from the KTable)?

 On Wed, Mar 29, 2017 at 6:31 AM, Michael Noll 
 wrote:

> Jon,
>
> there's a related example, using a window store and a key-value store,
>>> at
> https://github.com/confluentinc/examples/blob/3.2.x/kafka-
> streams/src/test/java/io/confluent/examples/streams/Val
> idateStateWithInteractiveQueriesLambdaIntegrationTest.java
> (this is for Confluent 3.2 / Kafka 0.10.2).
>
> -Michael
>
>
>
> On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers <
>> jon.yearg...@cedexis.com

> wrote:
>
>> Im only running one instance (locally) to keep things simple.
>>
>> Reduction:
>>
>> KTable, String> hourAggStore =
>> sourceStream.groupByKey().reduce(rowReducer,
>> TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
>> 1000).until(70 * 60 * 1000L),
>> "HourAggStore");
>>
>> then I get values to look for via:
>>
>> hourAggStore.foreach((k, v) -> {
>> LogLine logLine = objectMapper.readValue(v,
> logLine.class);
>> LOGGER.debug("{}", k.key());
>> });
>>
>> Ive kept it easy by requesting everything from 0 to
>> 'System.currentTimeMillis()'. Retrieval is done using a snip from
>> your
>> sample code "windowedByKey".
>>
>> Requests are sent in via curl and output through the same channel. I
> pass
>> in the key and ask for any values.
>>
>> Ive looked at the values passed in / out of the reduction function
>> and
> they
>> look sane.
>>
>> My assumption is that if a value shows up in the 'forEach' loop this
>> implies it exists in the StateStore. Accurate?
>>
>> In fact, only about one in 10 requests actually return any values.
>> No
>> errors - just no data.
>>
>>
>>
>> On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy 
> wrote:
>>
>>> Hi Jon,
>>>
>>> If you are able to get a handle on the store, i.e., via
>>> KafkaStreams.store(...) and call fetch without any exceptions,
>> then
> the
>>> store is available.
>>> The time params to fetch are the boundaries to search for windows
>>> for
> the
>>> given key. They relate to the start time of the window, so if you
>>> did
>>> fetch(key, t1, t2) - it will find all the windows for key that
>> start
> in
>> the
>>> inclusive time range t1 - t2.
>>>
>>> Are you running more than one instance? If yes, then you want to
>>> make
>> sure
>>> that you are querying the correct instance. For that you can use:
>>> KafkaStreams.metadataForKey(...) to find the instance that has
>> the
> key
>> you
>>> are looking for.
>>>
>>> Thanks,
>>> Damian
>>>
>>>
>>>
>>> On Tue, 28 Mar 2017 at 22:37 Jon Yeargers <
>> jon.yearg...@cedexis.com

>>> wrote:
>>>
 Im probing about trying to find a way to solve my aggregation ->
>>> db
>>> issue.
 Looking at the '.fetch()'  function Im wondering about the
> 'timeFrom'
>> and
 'timeTo' params as not a lot is mentioned about 'proper' usage.

 The test in

 https://github.com/confluentinc/examples/blob/
>>> master/kafka-streams/src/test/java/io/confluent/examples/
>>> streams/interactivequeries/WordCountInteractiveQueriesExa
>>> mpleTest.java#L200-L212
 makes it appear that the params are boundaries and that it will
> return
>> an
 inclusive list of every key/window c

Re: Understanding ReadOnlyWindowStore.fetch

2017-03-29 Thread Jon Yeargers

I remain more than mystified about the workings of the StateStore. I tried
making aggregations with a 1minute window, 10 second advance and a _12
hour_ retention (which is longer than the retention.ms of the topic).  I
still couldn't get more than a 15% hit rate on the StateStore.

Are there configuration settings? Some properties file to setup RocksDB? Im
not getting any errors - just not getting any data.

On Wed, Mar 29, 2017 at 12:52 PM, Jon Yeargers 
wrote:

> So '.until()' is based on clock time / elapsed time (IE record age) /
> something else?
>
> The fact that Im seeing lots of records come through that can't be found
> in the Store - these are 'old' and already expired?
>
> Going forward - it would be useful to have different forms of '.until()'
> so one could consume old records (EG if one was catching up from lag)
> without having to worry about them immediately disappearing.
>
> On Wed, Mar 29, 2017 at 10:37 AM, Damian Guy  wrote:
>
>> Jon,
>>
>> You should be able to query anything that has not expired, i.e., based on
>> TimeWindows.until(..).
>>
>> Thanks,
>> Damian
>>
>> On Wed, 29 Mar 2017 at 17:24 Jon Yeargers 
>> wrote:
>>
>> > To be a bit more specific:
>> >
>> > If I call this: KTable kt =
>> > sourceStream.groupByKey().reduce(..., "somekeystore");
>> >
>> > and then call this:
>> >
>> > kt.forEach()-> ...
>> >
>> > Can I assume that everything that comes out will be available in
>> > "somekeystore"? If not, what subset should I expect to find there?
>> >
>> > On Wed, Mar 29, 2017 at 8:34 AM, Jon Yeargers > >
>> > wrote:
>> >
>> > > But if a key shows up in a KTable->forEach should it be available in
>> the
>> > > StateStore (from the KTable)?
>> > >
>> > > On Wed, Mar 29, 2017 at 6:31 AM, Michael Noll 
>> > > wrote:
>> > >
>> > >> Jon,
>> > >>
>> > >> there's a related example, using a window store and a key-value
>> store,
>> > at
>> > >> https://github.com/confluentinc/examples/blob/3.2.x/kafka-
>> > >> streams/src/test/java/io/confluent/examples/streams/Val
>> > >> idateStateWithInteractiveQueriesLambdaIntegrationTest.java
>> > >> (this is for Confluent 3.2 / Kafka 0.10.2).
>> > >>
>> > >> -Michael
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Mar 29, 2017 at 3:12 PM, Jon Yeargers <
>> jon.yearg...@cedexis.com
>> > >
>> > >> wrote:
>> > >>
>> > >> > Im only running one instance (locally) to keep things simple.
>> > >> >
>> > >> > Reduction:
>> > >> >
>> > >> > KTable, String> hourAggStore =
>> > >> > sourceStream.groupByKey().reduce(rowReducer,
>> > >> > TimeWindows.of(65 * 60 * 1000L).advanceBy(5 * 60 *
>> > >> > 1000).until(70 * 60 * 1000L),
>> > >> > "HourAggStore");
>> > >> >
>> > >> > then I get values to look for via:
>> > >> >
>> > >> > hourAggStore.foreach((k, v) -> {
>> > >> > LogLine logLine = objectMapper.readValue(v,
>> > >> logLine.class);
>> > >> > LOGGER.debug("{}", k.key());
>> > >> > });
>> > >> >
>> > >> > Ive kept it easy by requesting everything from 0 to
>> > >> > 'System.currentTimeMillis()'. Retrieval is done using a snip from
>> your
>> > >> > sample code "windowedByKey".
>> > >> >
>> > >> > Requests are sent in via curl and output through the same channel.
>> I
>> > >> pass
>> > >> > in the key and ask for any values.
>> > >> >
>> > >> > Ive looked at the values passed in / out of the reduction function
>> and
>> > >> they
>> > >> > look sane.
>> > >> >
>> > >> > My assumption is that if a value shows up in the 'forEach' loop
>> this
>> > >> > implies it exists in the StateStore. Accurate?
>> > >> >
>> > >> > In fact, only about one in 10 requests actually return any values.
>> No
>> > >> > errors - just no data.
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Mar 29, 2017 at 2:15 AM, Damian Guy 
>> > >> wrote:
>> > >> >
>> > >> > > Hi Jon,
>> > >> > >
>> > >> > > If you are able to get a handle on the store, i.e., via
>> > >> > > KafkaStreams.store(...) and call fetch without any exceptions,
>> then
>> > >> the
>> > >> > > store is available.
>> > >> > > The time params to fetch are the boundaries to search for windows
>> > for
>> > >> the
>> > >> > > given key. They relate to the start time of the window, so if you
>> > did
>> > >> > > fetch(key, t1, t2) - it will find all the windows for key that
>> start
>> > >> in
>> > >> > the
>> > >> > > inclusive time range t1 - t2.
>> > >> > >
>> > >> > > Are you running more than one instance? If yes, then you want to
>> > make
>> > >> > sure
>> > >> > > that you are querying the correct instance. For that you can use:
>> > >> > > KafkaStreams.metadataForKey(...) to find the instance that has
>> the
>> > >> key
>> > >> > you
>> > >> > > are looking for.
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Damian
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Tue, 28 Mar 2017 at 22:37 Jon Yeargers <
>> jon.yearg...@cedexis.com
>> > >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Im probing about trying to find a way to solve my aggr

working of Kafka quota for clients and users

2017-03-29 Thread Archie

Hi,

I am pretty new to kafka and want to understand how the quota system works
for kafka.

Till now I have been following the document here


I have been able to set the quotas (produce and consume) for new clients
using the following command

bin/kafka-configs.sh --zookeeper 10.11.10.2:2181 --alter --add-config
'producer_byte_rate=1024,consumer_byte_rate=1024' --entity-type clients
--entity-name clientA
I am also using the following command to measure the throughput of Kafka
topics

bin/kafka-producer-perf-test.sh --topic topic1 --num-records 10
--record-size 10 --throughput 50 --producer-props acks=0
bootstrap.servers=10.11.10.2:9092

Now I am not sure how to assign a client-id for a particular producer.
Basically I want to run the kafka-producer-perf-test with a particular
client id.

Is it possible to do this. Also what is the difference between the
user-quota and client-quota? Can I assign user-id to a particular producer?

Thanks,
Archie

Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

2017-03-29 Thread Mina Aslani

Hi,

Do we have an example of a container with an instance of the jar file by
any chance? I am wondering if I should have a container of headless java or
should I have a container of Kafka?

And after I have the container running, in my container should I run Java
-cp ... same as https://github.com/confluentinc/examples/blob/3.
2.x/kafka-streams/src/main/java/io/confluent/examples/streams/
WordCountLambdaExample.java#L55-L62?

Regards,
Mina

On Tue, Mar 21, 2017 at 4:49 PM, Mina Aslani  wrote:

> Hi Michael,
>
> Thank you very much for the prompt response, really appreciate it!
>
> From https://github.com/confluentinc/examples/blob/3.2.x/
> kafka-streams/src/main/java/io/confluent/examples/streams/
> WordCountLambdaExample.java#L55-L62 and
> https://github.com/confluentinc/examples/tree/3.2.x/kafka-
> streams#packaging-and-running I missed the fact that the jar should be
> run in a separate container.
>
> Best regards,
> Mina
>
> On Tue, Mar 21, 2017 at 4:34 PM, Michael Noll 
> wrote:
>
>> Typically you'd containerize your app and then launch e.g. 10 containers
>> if
>> you need to run 10 instances of your app.
>>
>> Also, what do you mean by "in a cluster of Kafka containers" and "in the
>> cluster of Kafkas"?
>>
>> On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani 
>> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand how I can use a kafka stream app(jar file) in
>> a
>> > cluster of kafka containers.
>> >
>> > Kafka does not have master/slave concept (unlike spark), how I should
>> run
>> > my app in the cluster of kafkas (e.g. on one or multiple
>> docker-machine/s)?
>> >
>> > I use below command line when having one VM/node with one kafka
>> container
>> > https://github.com/confluentinc/examples/blob/3.
>> > 2.x/kafka-streams/src/main/
>> > java/io/confluent/examples/streams/WordCountLambdaExample.java#L55-L62
>> >
>> > Best regards,
>> > Mina
>> >
>>
>
>

Re: Understanding ReadOnlyWindowStore.fetch

Kafka broker went down with "No space left on device" when there is a lot more

Re: Kafka broker went down with "No space left on device" when there is a lot more

Re: Understanding ReadOnlyWindowStore.fetch

Re: Understanding ReadOnlyWindowStore.fetch

Kafka running but not listening to port 9092

Re: Understanding ReadOnlyWindowStore.fetch

Re: Understanding ReadOnlyWindowStore.fetch

Re: Understanding ReadOnlyWindowStore.fetch

ThoughWorks Tech Radar: Assess Kafka Streams

Re: ThoughWorks Tech Radar: Assess Kafka Streams

Re: Understanding ReadOnlyWindowStore.fetch

weird SerializationException when consumer is fetching and parsing record in streams application

Re: Understanding ReadOnlyWindowStore.fetch

Re: Understanding ReadOnlyWindowStore.fetch

working of Kafka quota for clients and users

Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

17 matches

Site Navigation

Mail list logo

Footer information