[
https://issues.apache.org/jira/browse/KAFKA-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804560#comment-14804560
]
Abhishek Nigam commented on KAFKA-2552:
---------------------------------------
Essentially a generic approach to this which would require read and write side
to change would be as follows:
We designate a zookeeper path as scratch:
Ex- /admin/scratch
Write side
When writing json to zookeeper we will chunk it into 1 MB units and store it in
different zookeeper nodes from the sratch all but the first chunk.
The first chunk will live in the original location as we have it today.
Ex- /admin/reassign_partitions
Each chunk will have the following format
"json incompatible header" something other than "{"
length of the zookeeper path to the next json chunk (0 means that this is the
last chunk)
zookeeper path of the next json chunk.
length of chunk of json data blob.
chunk of json data blob.
We will write to this conceptual linked list back to front.
Read side
The zookeeper watch will be fired as before. While reading if we detect there
are more chunks we will do synced read from zookeeper.
> Certain admin commands such as partition assignment fail on large clusters
> --------------------------------------------------------------------------
>
> Key: KAFKA-2552
> URL: https://issues.apache.org/jira/browse/KAFKA-2552
> Project: Kafka
> Issue Type: Improvement
> Reporter: Abhishek Nigam
> Assignee: Abhishek Nigam
>
> This happens because the json generated is greater than 1 MB and exceeds the
> default data limit of zookeeper nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
