Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/15374
Interesting to compare this with Hadoop's annotation, where I have mixed
opinions.
A key advantage Apache spark has is that Scala language lets you really
scope out things as `private[spark]`; in contrast all Hadoop has is package
private, which doesn't scale beyond a single package. By using the
private[spark] option you can implicitly lock out external code *provided that
code doesn't just use the `org.apache.spark` package name. Which, while it's
easy to condemn, is similar to what spark SQL has to do to get at Hive
internals under `org.apache.hive`.
Where Hadoop really falls down is that so many classes are essential to day
to day use of Private/limitedPrivate tags that its devalued, take
`UserGroupInformation` as an example. Want to work in Kerberos: you have to use
that, even though you aren't on the "approved" list. I note that @tgravescs
owns the [homework there|https://issues.apache.org/jira/browse/HADOOP-10506] to
take a sharp knife to that; IMO going to public/unstable is the way to go âat
least for any API tagged as limited private YARN/MAPREDUCE.
The big issue with that LimitedPrivate tag is that it viewed by some on the
Hadoop dev team as giving a green light to any change, without feeling guilty
about breaking things. If I find something downstream breaks against branch-2,
I could walk up the colleague and say "you broke my build" and get the response
"you shouldn't have used it"....I think everyone needs to do some downstream
coding to appreciate the Joy there.
Anyway: LimitedPrivate is essentially worthless, as least as far as I'm
concerned. Probably better to have a documented policy about injected code into
the `org.apache.spark.*` packages with something like "bahir can ask nicely"
and for anyone else "if you have to do this, don't complain when things break".
For public stuff, Evolving, Stable and perhaps something about retirement.
Scala's `deprecated` tag does that & seems pretty well used in Spark. I think
Hadoop should go back and look at it's markup there, maybe add something of its
own which does text+JIRA+since; or at least have a policy of mandating the
javadoc text to go with the method annotation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
