I tried and failed to do this in https://issues.apache.org/jira/browse/SPARK-22142 because it became clear that the Flume examples would have to be removed to make this work, too. (Well, you can imagine other solutions with extra source dirs or modules for flume examples enabled by a profile, but that doesn't help the docs and is nontrivial complexity for little gain.)
It kind of suggests Flume support should be deprecated if it's put behind a profile. Like with Kafka 0.8. (This is why I'm raising it again to the whole list.) Any preferences among: 1. Put Flume behind a profile, remove examples, deprecate 2. Put Flume behind a profile, remove examples, but don't deprecate 3. Punt until Spark 3.0, when this integration would probably be removed entirely (?) On Tue, Sep 26, 2017 at 10:36 AM Sean Owen <so...@cloudera.com> wrote: > Not a big deal, but I'm wondering whether Flume integration should at > least be opt-in and behind a profile? it still sees some use (at least on > our end) but not applicable to the majority of users. Most other > third-party framework integrations are behind a profile, like YARN, Mesos, > Kinesis, Kafka 0.8, Docker. Just soliciting comments, not arguing for it. > > (Well, actually it annoys me that the Flume integration always fails to > compile in IntelliJ unless you generate the sources manually) >
