Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21537
@gatorsmile I think this is just the partial adoption of what was proposed
[here](https://github.com/apache/spark/pull/19813#issuecomment-354045400) and
implemented in SPARK-24121 (https://github.com/apache/spark/pull/21193).
Despite I agree with you that we should have created a design doc for this,
I think now is a bit late for it. Indeed the base API is already there, since
the design has mostly been done in SPARK-24121 (and not in the scope of this
PR). What is missing now, though, is a throughout adoption. But there are
already many PRs which based on that API: SPARK-22856, SPARK-23951. And the
current codebase already partially adopts it.
So I am not sure whether reverting this PR would be a solution: we should
revert many if we want the change the current design.
I think a design can be created anyway, at least in order to formalize what
was discussed and agreed in the various PRs and have a single source of
information. We can also do some modifications to the current design according
to the comments which could come out discussing the design doc, but I see them
more like further changes than reverting this, as it would mean reverting many
things.
The only last thing I'd suggest is that probably @viirya is the best one
for creating the design doc, since the original proposal and implementation was
done by him.
Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
