Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/600#issuecomment-35208921
To answer my own question, the reason we do this is probably because we
have no choice when integrating with Java. Additionally, use of ClassTag as in
our API rarely affects correctness, just potentially safety and performance
(e.g., creating a primitive-backed map versus an object one).
This unfortunately adds a burden we must keep in mind as we develop the
Spark Scala APIs:
- ClassTags should be used sparingly and never relied upon for correctness.
- The Java API may be unavoidably slower than the Scala API where ClassTags
are used for perf.
This also means it is very hard, in general, to verify the correctness of a
Java API like the one introduced in this patch, since we're tricking the
compiler and introducing unusual runtime behavior.
In this particular case, I can verify that the API will work because
subtractByKey never actually uses W's ClassTag (I can even remove it from the
signature without issue), but that's not guaranteed to hold in the future.
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastruct...@apache.org or file a JIRA ticket with INFRA.
