Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > I'd want to see some test cases though that show why the current implementation is wrong from an end-user perspective if it needs to block merging initial kafka support. PR with failing test indicating at least one reason why it's wrong from an end-user perspective: https://github.com/zsxwing/spark/pull/4 > I do not think it is reasonable to suggest we block merging this patch on an overhaul of the DataSource API configuration system. Here's what I actually said: 'if you know your plan down the line is to use json for structured configuration, you should use it now, and provide more convenient ways to construct json later, not use "convenient" non-json hacks now.' No hyperbole about blocking on a complete overhaul, nothing that isn't backwards compatible. I'm just saying that, if the design document already recognizes that json is necessary to work around the string -> string interface... start using structured json strings now, and make it more convenient later. Or do you actually think that stuff like option("assign", "topicA:1:1,topicA:2:2,topicB:3:3") makes it clear what the arguments are? > I think @koeninger made a good suggestion to block accepting certain kafka configurations. In case it wasn't clear, I was not suggesting that preventing users from doing things they could otherwise do with Kafka is actually a good idea. I think it's a bad idea, but if you're going to run with it, you might as well be consistent about it.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org