Thanks. By default, can you explain me why I received the message in wrong order? Note there are only 9 lines from 1 to 9, but on consumer side their original order becomes messed up.
~~~sent from my cell phone, sorry if there is any typo Hans Jespersen <h...@confluent.io> 于 2018年5月26日周六 上午12:16写道： > If you create a topic with one partition they will be in order. > > Alternatively if you publish with the same key for every message they will > be in the same order even if your topic has more than 1 partition. > > Either way above will work for Kafka. > > -hans > > > On May 25, 2018, at 8:56 PM, Raymond Xie <xie3208...@gmail.com> wrote: > > > > Hello, > > > > I just started learning Kafka and have the environment setup on my > > hortonworks sandbox at home vmware. > > > > test.csv is what I want the producer to send out: > > > > more test1.csv ./kafka-console-producer.sh --broker-list > > sandbox.hortonworks.com:6667 --topic kafka-topic2 > > > > 1, abc > > 2, def > > ... > > 8, vwx > > 9, zzz > > > > What I received are all the content of test.csv, however, not in their > > original order; > > > > kafka-console-consumer.sh --zookeeper 192.168.112.129:2181 --topic > > kafka-topic2 > > > > 2, def > > 1, abc > > ... > > 9, zzz > > 8, vwx > > > > > > I read from google that partition could be the feasible solution, > however, > > my questions are: > > > > 1. for small files like this one, shall I really do the partitioning? how > > small a partition would be acceptable to ensure the sequence? > > 2. for big files, each partition could still contain multiple lines, how > to > > ensure all the lines in each partition won't get messed up on consumer > side? > > > > > > I also want to know what is the best practice to process large volume of > > data through kafka? There should be better way other than console > command. > > > > Thank you very much. > > > > > > > > *------------------------------------------------* > > *Sincerely yours,* > > > > > > *Raymond* >
