Hello, I'm currently testing spark streaming with kafka. I'm creating DirectStream with KafkaUtils and everything's fine. However I would like to use the signature where I can specify my own message handler (to play with partition and offset). In this case, I need to manage offset/partition by myself to fill fromOffsets argument. I have found a Jira on this usecase https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed telling that it's too specific. I'm aware that it can be done using kafka api (TopicMetaDataRequest and OffsetRequest) but what I have to do is almost the same as the KafkaCluster which is private.
is it possible to : - add another signature in KafkaUtils ? - make KafkaCluster public ? or do you have any other srmart solution where I don't need to copy/paste KafkaCluster ? Thanks. Regards, Erwan ALLAIN
