Dear Dana, > I would recommend > other tools for bulk transfers.
What tools/languages would you rather recommend then using Python? I could for sure accomplish the same by using the native Java Kafka Producer API, but should this really affect the performance under the assumption that the Kafka configuration stays as is? > On 25 Aug 2016, at 18:43, Dana Powers <dana.pow...@gmail.com> wrote: > > python is generally restricted to a single CPU, and kafka-python will max > out a single CPU well before it maxes a network card. I would recommend > other tools for bulk transfers. Otherwise you may find that partitioning > your data set and running separate python processes for each will increase > the overall CPU available and therefore the throughput. > > One day I will spend time improving the CPU performance of kafka-python, > but probably not in the near term. > > -Dana
