GitHub user stathmarxis opened a pull request:
https://github.com/apache/kafka/pull/1558
org.apache.kafka.streams.errors.StreamsException: Failed to rebalance
<p>I am running the below simple example in kafka streams and i got a weird
exception which i cannot handle</p>
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-pipe");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
"192.168.1.3:9092");
props.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
// setting offset reset to earliest so that we can re-run the
demo code with the same pre-loaded data
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
KStreamBuilder builder = new KStreamBuilder();
builder.stream("streams-file-input").to("streams-pipe-output");
KafkaStreams streams = new KafkaStreams(builder, props);
streams.start();
// usually the stream application would be running forever,
// in this example we just let it run for some time and stop
since the input data is finite.
Thread.sleep(5000L);
streams.close();
[Exception in thread "StreamThread-1"
org.apache.kafka.streams.errors.StreamsException: Failed to rebalance
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:299)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException:
Error while creating the state manager
at
org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:71)
at
org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:86)
at
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
at
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
at
org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
at
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:295)
... 1 more
Caused by: java.io.FileNotFoundException:
C:\tmp\kafka-streams\my-streapplication\0_0\.lock (The system cannot find the
path specified)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.lockStateDirectory(ProcessorStateManager.java:125)
at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:93)
at
org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:69)](url)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/kafka 0.10.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1558.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1558
----
commit 1b5879653e0d956c79556301d1d11987baf6f2d7
Author: Gwen Shapira <[email protected]>
Date: 2016-03-21T16:48:02Z
Changing version to 0.10.0.0
commit 4f8b3aed8dc6a77293c526253123d999a23af149
Author: Guozhang Wang <[email protected]>
Date: 2016-03-21T19:06:07Z
MINOR: Add InterfaceStability.Unstable annotations to some Kafka Streams
public APIs
Also improves Java docs for the Streams high-level DSL.
Author: Guozhang Wang <[email protected]>
Reviewers: Ismael Juma, Michael G. Noll
Closes #1097 from guozhangwang/KNewJavaDoc
(cherry picked from commit b6c29e3810bd59f39fa93c429817396cf8c324b7)
Signed-off-by: Guozhang Wang <[email protected]>
commit 3710b12e6e6a6d8dddb7402087ee550cd14909db
Author: Gwen Shapira <[email protected]>
Date: 2016-03-21T20:15:53Z
MINOR: fix documentation version
This will need to be double-committed.
Author: Gwen Shapira <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Ewen Cheslack-Postava
<[email protected]>
Closes #1107 from gwenshap/fix-doc-version
(cherry picked from commit 69df734470f1db9ba1732cb4bedaa1541d721103)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit 3cfd20b7b9d609c1055f6c23de86b6133e5cac75
Author: Gwen Shapira <[email protected]>
Date: 2016-03-22T01:58:45Z
MINOR: update new version in additional places
matching set of version fixes. ewencp junrao
Author: Gwen Shapira <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Geoff Anderson
<[email protected]>, Ewen Cheslack-Postava <[email protected]>
Closes #1110 from gwenshap/minor-fix-version-010
commit 808f85f03d8f69047914eb21438d1458e23c4325
Author: Jason Gustafson <[email protected]>
Date: 2016-03-22T03:47:25Z
KAFKA-3412: multiple asynchronous commits causes send failures
Author: Jason Gustafson <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Ewen Cheslack-Postava
<[email protected]>
Closes #1108 from hachikuji/KAFKA-3412
(cherry picked from commit 8d8e3aaa6172d314230a8d61e6892e9c09dc45b6)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit c0f870f8509253564c73ca76cd9fabb7df4ce6ad
Author: Grant Henke <[email protected]>
Date: 2016-03-22T18:53:12Z
KAFKA-3435: Remove `Unstable` annotation from new Java Consumer
Author: Grant Henke <[email protected]>
Reviewers: Gwen Shapira
Closes #1113 from granthenke/remove-unstable
(cherry picked from commit 99cfb99aa161007f2ceae33945d11b968d932175)
Signed-off-by: Gwen Shapira <[email protected]>
commit 66ac5157d914aacf9b544fe7de91020f445cef3c
Author: gaob13 <[email protected]>
Date: 2016-03-22T18:58:01Z
MINOR: Remove the very misleading comment lines
It is not true in practice. Maybe the implied feature is not yet
implemented or removed.
These lines can be super misleading.
Please merge.
Thank you.
Author: gaob13 <[email protected]>
Reviewers: Ismael Juma, Ewen Cheslack-Postava
Closes #793 from gaob13/trunk
(cherry picked from commit 73470b028cd659c1c405e6b6ba52bf483113b90b)
Signed-off-by: Gwen Shapira <[email protected]>
commit e2d7f9e445a2cbb5e1e0ed8437af849b731677bf
Author: Ismael Juma <[email protected]>
Date: 2016-03-22T19:39:04Z
KAFKA-3426; Improve protocol type errors when invalid sizes are received
Author: Ismael Juma <[email protected]>
Reviewers: Grant Henke, Gwen Shapira
Closes #1100 from
ijuma/kafka-3426-invalid-protocol-type-errors-invalid-sizes
(cherry picked from commit 73c79000edddd929cd0af25f4a29fcc682a1c9c0)
Signed-off-by: Gwen Shapira <[email protected]>
commit 687d2494ff27b05f21ea29e645a0b1b000334db6
Author: Jason Gustafson <[email protected]>
Date: 2016-03-22T20:09:13Z
KAFKA-3319: improve session timeout broker/client config documentation
Author: Jason Gustafson <[email protected]>
Reviewers: Grant Henke, Ismael Juma, Guozhang Wang
Closes #1106 from hachikuji/KAFKA-3319
(cherry picked from commit ca77d67058726fc9df9bdd7cc0217ee62ccc5106)
Signed-off-by: Guozhang Wang <[email protected]>
commit 05984de69ee8eb06f214a2e42abd789a4fbc8cef
Author: Vahid Hashemian <[email protected]>
Date: 2016-03-22T20:10:07Z
KAFKA-3219: Fix long topic name validation
This fixes an issue with long topic names by considering, during topic
validation, the '-' and the partition id that is appended to the log
folder created for each topic partition.
Author: Vahid Hashemian <[email protected]>
Reviewers: Gwen Shapira, Grant Henke
Closes #898 from vahidhashemian/KAFKA-3219
(cherry picked from commit ad3dfc6ab25c3f80d2425e24e72ae732b850dc60)
Signed-off-by: Gwen Shapira <[email protected]>
commit a81e5d05e6fea7fb74dfb19b67729b6dd4ca2423
Author: Ewen Cheslack-Postava <[email protected]>
Date: 2016-03-22T21:09:10Z
KAFKA-3435: Follow up to fix checkstyle
Author: Ewen Cheslack-Postava <[email protected]>
Reviewers: Gwen Shapira
Closes #1116 from ewencp/kafka-3435-follow-up
(cherry picked from commit d6b450ced57bb5ab8a7164358e6a360f97a7b5cc)
Signed-off-by: Gwen Shapira <[email protected]>
commit d9cbe566371ed1f519fa983c14fc48d088f66b6a
Author: Grant Henke <[email protected]>
Date: 2016-03-22T22:01:05Z
KAFKA-3301; CommonClientConfigs.METRICS_SAMPLE_WINDOW_MS_DOC is incorâ¦
â¦rect
Author: Grant Henke <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #1114 from granthenke/window-docs
(cherry picked from commit 983e9486ef139f8c9797679732d09feb3f7bfb87)
Signed-off-by: Jun Rao <[email protected]>
commit d1e74890caaf3b76b0d9a888742a2cf1c6e6e883
Author: Ismael Juma <[email protected]>
Date: 2016-03-23T01:20:07Z
KAFKA-3447; partitionState in UpdateMetadataRequest not logged properly
state-change log
Author: Ismael Juma <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #1117 from ijuma/kafka-3447-metadata-cache-logging
(cherry picked from commit adca4d7df3155ac29ae510c237adf06c9521d221)
Signed-off-by: Jun Rao <[email protected]>
commit aead28a54e55ca0541d696abced523b453bb50f7
Author: Ismael Juma <[email protected]>
Date: 2016-03-23T02:13:26Z
KAFKA-3431: Remove `o.a.k.common.BrokerEndPoint` in favour of `Node`
Also included a minor efficiency improvement in kafka.cluster.EndPoint.
Author: Ismael Juma <[email protected]>
Reviewers: Gwen Shapira
Closes #1105 from ijuma/kafka-3431-replace-broker-end-point-with-node
(cherry picked from commit 255b5e13863a95cfc327236856db2df188f04d49)
Signed-off-by: Gwen Shapira <[email protected]>
commit e5a1f704d7ed59f9a6a423bbc36510d13a5df1de
Author: Jiangjie Qin <[email protected]>
Date: 2016-03-23T14:15:59Z
KAFKA-3442; Fix FileMessageSet iterator.
Author: Jiangjie Qin <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Jun Rao <[email protected]>
Closes #1112 from becketqin/KAFKA-3442
(cherry picked from commit 7af67ce22aa02121d6b82dc54dad42358282e524)
Signed-off-by: Jun Rao <[email protected]>
commit 2835e73000a44c9586d085275b62eeeb2ce1fcec
Author: Jason Gustafson <[email protected]>
Date: 2016-03-23T16:47:48Z
KAFKA-3409: handle CommitFailedException in MirrorMaker
Author: Jason Gustafson <[email protected]>
Reviewers: Ismael Juma, Ashish Singh, Guozhang Wang
Closes #1115 from hachikuji/KAFKA-3409
(cherry picked from commit 20c313526a0518a51142d3abc5ee2a4d2ef3cb34)
Signed-off-by: Guozhang Wang <[email protected]>
commit f5e1ca625332572923b42a9ae8d660f2452ae92f
Author: Grant Henke <[email protected]>
Date: 2016-03-23T19:54:26Z
KAFKA-3441: 0.10.0 documentation still says "0.9.0"
Author: Grant Henke <[email protected]>
Reviewers: Gwen Shapira
Closes #1122 from granthenke/docs-10
(cherry picked from commit d57847641037823a7306f53251b063869f7affb6)
Signed-off-by: Gwen Shapira <[email protected]>
commit 63e9d246b7978152e12aa4ea8b1ba6bcc40f4498
Author: Ismael Juma <[email protected]>
Date: 2016-03-23T20:53:37Z
KAFKA-3432; Cluster.update() thread-safety
Replace `update` with `withPartitions`, which returns a copy instead of
mutating the instance.
Author: Ismael Juma <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1118 from ijuma/kafka-3432-cluster-update-thread-safety
(cherry picked from commit d4d5920ed40736d21f056188efa8a86c93e22506)
Signed-off-by: Guozhang Wang <[email protected]>
commit b31a2f06c80c2c2cacf34d75998d401bddcca374
Author: Yasuhiro Matsuda <[email protected]>
Date: 2016-03-23T21:25:08Z
HOTFIX: fix NPE in changelogger
Fix NPE in StoreChangeLogger caused by a record out of window retention
period.
guozhangwang
Author: Yasuhiro Matsuda <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1124 from ymatsuda/logger_npe
(cherry picked from commit 80d78f81470f109dc6d221f755b039c7332bb93b)
Signed-off-by: Guozhang Wang <[email protected]>
commit 8525d8279b7798db90f752c329aa3b5936cb5f7f
Author: Yasuhiro Matsuda <[email protected]>
Date: 2016-03-23T21:57:03Z
MINOR: remove streams-smoke-test.sh
guozhangwang
Author: Yasuhiro Matsuda <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1125 from ymatsuda/remove_smoketest_shell_script
(cherry picked from commit de062443381df84ee0d65acc20e44ffca2b2552b)
Signed-off-by: Guozhang Wang <[email protected]>
commit 0cc997ae76010cf85b55d057dea9996fa1931e5b
Author: Gwen Shapira <[email protected]>
Date: 2016-03-23T22:41:07Z
MINOR: Revert 0.10.0 branch to SNAPSHOT per change in release process
Author: Gwen Shapira <[email protected]>
Reviewers: Ewen Cheslack-Postava
Closes #1126 from gwenshap/minor-release-version
commit 4e557f8ef60d46a8870704655c9a35092f74d125
Author: Jason Gustafson <[email protected]>
Date: 2016-03-24T05:36:19Z
KAFKA-3434; add old constructor to ConsumerRecord
Author: Jason Gustafson <[email protected]>
Reviewers: Grant Henke <[email protected]>, Ismael Juma
<[email protected]>, Ewen Cheslack-Postava <[email protected]>
Closes #1123 from hachikuji/KAFKA-3434
(cherry picked from commit cb78223bf90aca4f75699f36c1a82db7661a62f3)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit 4e5653d77359214452b94a65c9db29eaa57a3c31
Author: Grant Henke <[email protected]>
Date: 2016-03-25T17:07:05Z
KAFKA-3460: Remove old 0.7 KafkaMigrationTool
Author: Grant Henke <[email protected]>
Reviewers: Gwen Shapira
Closes #1136 from granthenke/remove-07-migration
(cherry picked from commit d3a66a65365579e9320347663969b30c1148e497)
Signed-off-by: Gwen Shapira <[email protected]>
commit 3f930cd84c1a889f0df2bc5057e073e683860171
Author: Jason Gustafson <[email protected]>
Date: 2016-03-25T19:51:54Z
KAFKA-3463: change default receive buffer size for consumer to 64K
Author: Jason Gustafson <[email protected]>
Reviewers: Gwen Shapira
Closes #1140 from hachikuji/KAFKA-3463
(cherry picked from commit d691faf98cb573c4e92748d95d5c8afc492db806)
Signed-off-by: Gwen Shapira <[email protected]>
commit 1b1b949b7fbd10fe4b08e97769ab024fad8459fb
Author: Andrea Cosentino <[email protected]>
Date: 2016-03-25T22:00:45Z
KAFKA-3449: Rename filterOut() to filterNot() to achieve better terminology
â¦nology
Hi all,
This is my first contribution and I hope it will be good.
The PR is related to this issue:
https://issues.apache.org/jira/browse/KAFKA-3449
Thanks a lot,
Andrea
Author: Andrea Cosentino <[email protected]>
Reviewers: Yasuhiro Matsuda, Guozhang Wang
Closes #1134 from oscerd/KAFKA-3449
(cherry picked from commit c1d8c38345e0a1e04ced143ed07e63fe02ceb8b0)
Signed-off-by: Guozhang Wang <[email protected]>
commit 496bd3fd41c0272b7ef77652b85cbaae748e100e
Author: Guozhang Wang <[email protected]>
Date: 2016-03-25T23:04:58Z
KAFKA-3454: add Kafka Streams web docs
Author: Guozhang Wang <[email protected]>
Reviewers: Gwen Shapira
Closes #1127 from guozhangwang/KStreamsDocs
(cherry picked from commit 23b50093f4100ce7fbff325cdc92ee6cf3c54102)
Signed-off-by: Gwen Shapira <[email protected]>
commit 206757eeb18589952291ce1a5578c66de3669f4c
Author: Liquan Pei <[email protected]>
Date: 2016-03-25T23:46:53Z
KAFKA-3316: Add REST API for listing connector plugins
Author: Liquan Pei <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>
Closes #1090 from Ishiihara/kafka-3316
(cherry picked from commit 78fa20eb58a948abd9ad4e44acfed606400a47f3)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit b45fe77791560a2171e0bbd7df09f648a01b016f
Author: Jiangjie Qin <[email protected]>
Date: 2016-03-26T16:22:59Z
KAFKA-3388; Fix expiration of batches sitting in the accumulator
Author: Jiangjie Qin <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Jun Rao <[email protected]>
Closes #1056 from becketqin/KAFKA-3388
(cherry picked from commit 1fbe445dde71df0023a978c5e54dd229d3d23e1b)
Signed-off-by: Jun Rao <[email protected]>
commit f37b550502f6e3429cb3582db7e1c93614e84c58
Author: Ismael Juma <[email protected]>
Date: 2016-03-28T16:00:03Z
MINOR: Fix typo and tweak wording in `RecordAccumulator` comments
This was recently introduced in:
https://github.com/apache/kafka/commit/1fbe445dde71df0023a978c5e54dd229d3d23e1b
Author: Ismael Juma <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #1152 from ijuma/fix-typos-in-record-accumulator
(cherry picked from commit 4c0660bf3da9879cb405a0f85cf1524511e091e8)
Signed-off-by: Jun Rao <[email protected]>
commit afe82512ba5ec3c22ce0e99acfea9ce4d5b2bb2e
Author: Ismael Juma <[email protected]>
Date: 2016-03-28T21:35:31Z
MINOR: Remove a couple of redundant `CoreUtils.rm` methods
Also:
* Rename remaining `CoreUtils.rm` to `delete` for consistency
* Use `try with resources` in `Utils` to simplify code
* Silence compiler warning due to exception catch clause in `TestUtils`
Author: Ismael Juma <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1153 from ijuma/remove-redundant-core-utils-rm
(cherry picked from commit 43d5078e981bbb25fd81cdc8ba4c339cd2d3f3d2)
Signed-off-by: Guozhang Wang <[email protected]>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
