Right now, there is one to one correspondence between kafka partitions and spark partitions. I dont have a requirement of one to one semantics. I need more tasks to be generated in the job so that it can be parallelised and batch can be completed fast. In the previous Receiver based approach number of tasks created were independent of kafka partitions, I need something like that only. Any config available if I dont need one to one semantics? Is there any way I can repartition without incurring any additional cost.
Thanks *VARUN SHARMA*
