I am generating the reassignent.json like below
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --generate --topics-to-move-json-file
/home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
-1 > /home/ubuntu/deploy/kafka/reassignment.json"
Then I am doing the reassignment using the generated file
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --execute --reassignment-json-file
/home/ubuntu/deploy/kafka/reassignment.json
kafka-reassign-partitions.sh helps states
The JSON file with the partition reassignment configurationThe format to
> use is -
> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs" is
> optional. When it is specified, its length must equal the length of the
> replicas list. The value in this list can be either "any" or the absolution
> path of the log directory on the broker. If absolute log directory path is
> specified, it is currently required that the replica has not already been
> created on that broker. The replica will then be created in the specified
> log directory on the broker later.
So it appears reassignment json that is generated by
kafka-reassign-partions.sh is creating an issue with logdirs. Is this some
issue in kafka-reassign-partitions.sh or some misconfiguration from my
side. ?
On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <subharaj.ma...@gmail.com>
wrote:
> Please find the server.properties from one of the broker.
>
> broker.id=0
> port=9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/var/lib/kafka/kafka-logs
> num.recovery.threads.per.data.dir=1
> log.retention.hours=36
> log.retention.bytes=1073741824
> log.segment.bytes=536870912
> log.retention.check.interval.ms=300000
> log.cleaner.enable=false
> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> message.max.bytes=15000000
> replica.fetch.max.bytes=15000000
> auto.create.topics.enable=true
> zookeeper.connection.timeout.ms=6000
> unclean.leader.election.enable=false
> delete.topic.enable=false
> offsets.topic.replication.factor=1
> transaction.state.log.replication.factor=1
> transaction.state.log.min.isr=1
>
> I have placed server.log from a broker at https://gist.github.com/
> debraj-manna/4b4bdae8a1c15c36b313a04f37e8776d
>
> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> Seems to be related to KIP-113.
>>
>> server.properties didn't go thru. Do you mind pastebin'ing its content ?
>>
>> If you can pastebin logs from broker, that should help.
>>
>> Thanks
>>
>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <subharaj.ma...@gmail.com>
>> wrote:
>>
>> > Hi
>> >
>> > I altered a topic like below in kafka 1.1.0
>> >
>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
>> 127.0.0.1:2181
>> > --alter --topic Topic3 --config min.insync.replicas=2
>> >
>> > But whenever I am trying to verify the reassignment it is showing the
>> > below exception
>> >
>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
>> 127.0.0.1:2181 --reassignment-json-file
>> /home/ubuntu/deploy/kafka/reassignment.json
>> --verify
>> >
>> > Partitions reassignment failed due to Size of replicas list Vector(3,
>> 0, 2) is different from size of log dirs list Vector(any) for partition
>> Topic3-7
>> > kafka.common.AdminCommandFailedException: Size of replicas list
>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
>> partition Topic3-7
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.
>> apply(ReassignPartitionsCommand.scala:262)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.
>> apply(ReassignPartitionsCommand.scala:251)
>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitio
>> nsCommand.scala:251)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitio
>> nsCommand.scala:250)
>> > at scala.collection.immutable.List.foreach(List.scala:392)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
>> > at scala.collection.immutable.List.foreach(List.scala:392)
>> > at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassig
>> nmentData(ReassignPartitionsCommand.scala:249)
>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
>> signPartitionsCommand.scala:90)
>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
>> signPartitionsCommand.scala:84)
>> > at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitio
>> nsCommand.scala:58)
>> > at kafka.admin.ReassignPartitionsCommand.main(ReassignPartition
>> sCommand.scala)
>> >
>> >
>> > My reassignment.json & server.properties is attached. Same thing used to
>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
>> > anything changed related to this in kafka 1.1.0 ?
>> >
>>
>
>
