It will be taken as "any" directory for each replica, which means replica will placed on any one of the configured directory on that broker.
Since it is "log_dirs" optional, you can remove from the json. On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <[email protected]> wrote: > It is problem on my side. The code was changing the replicas count but not > the log_dirs. Since I am migrating from 0.10 this part of the code was not > changed. > > I have a follow up question what is the default value of log_dirs if I > don't specify it in reassignment.json ? > > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <[email protected]> > wrote: > > > I am generating the reassignent.json like below > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper > 127.0.0.1:2181 --generate --topics-to-move-json-file > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail -1 > > /home/ubuntu/deploy/kafka/reassignment.json" > > > > Then I am doing the reassignment using the generated file > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper > 127.0.0.1:2181 --execute --reassignment-json-file > /home/ubuntu/deploy/kafka/reassignment.json > > > > kafka-reassign-partitions.sh helps states > > > > The JSON file with the partition reassignment configurationThe format to > >> use is - > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3], > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs" > is > >> optional. When it is specified, its length must equal the length of the > >> replicas list. The value in this list can be either "any" or the > absolution > >> path of the log directory on the broker. If absolute log directory path > is > >> specified, it is currently required that the replica has not already > been > >> created on that broker. The replica will then be created in the > specified > >> log directory on the broker later. > > > > > > So it appears reassignment json that is generated by > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this > > some issue in kafka-reassign-partitions.sh or some misconfiguration from > my > > side. ? > > > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <[email protected] > > > > wrote: > > > >> Please find the server.properties from one of the broker. > >> > >> broker.id=0 > >> port=9092 > >> num.network.threads=3 > >> num.io.threads=8 > >> socket.send.buffer.bytes=102400 > >> socket.receive.buffer.bytes=102400 > >> socket.request.max.bytes=104857600 > >> log.dirs=/var/lib/kafka/kafka-logs > >> num.recovery.threads.per.data.dir=1 > >> log.retention.hours=36 > >> log.retention.bytes=1073741824 > >> log.segment.bytes=536870912 > >> log.retention.check.interval.ms=300000 > >> log.cleaner.enable=false > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181 > >> message.max.bytes=15000000 > >> replica.fetch.max.bytes=15000000 > >> auto.create.topics.enable=true > >> zookeeper.connection.timeout.ms=6000 > >> unclean.leader.election.enable=false > >> delete.topic.enable=false > >> offsets.topic.replication.factor=1 > >> transaction.state.log.replication.factor=1 > >> transaction.state.log.min.isr=1 > >> > >> I have placed server.log from a broker at https://gist.github.com/deb > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d > >> > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <[email protected]> wrote: > >> > >>> Seems to be related to KIP-113. > >>> > >>> server.properties didn't go thru. Do you mind pastebin'ing its content > ? > >>> > >>> If you can pastebin logs from broker, that should help. > >>> > >>> Thanks > >>> > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna < > [email protected] > >>> > > >>> wrote: > >>> > >>> > Hi > >>> > > >>> > I altered a topic like below in kafka 1.1.0 > >>> > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper > >>> 127.0.0.1:2181 > >>> > --alter --topic Topic3 --config min.insync.replicas=2 > >>> > > >>> > But whenever I am trying to verify the reassignment it is showing the > >>> > below exception > >>> > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify > >>> > > >>> > Partitions reassignment failed due to Size of replicas list Vector(3, > >>> 0, 2) is different from size of log dirs list Vector(any) for partition > >>> Topic3-7 > >>> > kafka.common.AdminCommandFailedException: Size of replicas list > >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for > >>> partition Topic3-7 > >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply( > >>> ReassignPartitionsCommand.scala:262) > >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply( > >>> ReassignPartitionsCommand.scala:251) > >>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > >>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:133 > >>> 4) > >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition > >>> sCommand.scala:251) > >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition > >>> sCommand.scala:250) > >>> > at scala.collection.immutable.List.foreach(List.scala:392) > >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250) > >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249) > >>> > at scala.collection.immutable.List.foreach(List.scala:392) > >>> > at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassig > >>> nmentData(ReassignPartitionsCommand.scala:249) > >>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas > >>> signPartitionsCommand.scala:90) > >>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas > >>> signPartitionsCommand.scala:84) > >>> > at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitio > >>> nsCommand.scala:58) > >>> > at kafka.admin.ReassignPartitionsCommand.main(ReassignPartition > >>> sCommand.scala) > >>> > > >>> > > >>> > My reassignment.json & server.properties is attached. Same thing used > >>> to > >>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is > >>> > anything changed related to this in kafka 1.1.0 ? > >>> > > >>> > >> > >> > > >
