Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
Sorry it's taken me a couple of days to respond. I needed the time to
ruminate (and not). I could write voluminously, but I just want to reply to a
couple of points and move on.
> I think that's primarily because the change looks incomplete but the
feature itself sounds good to have. I think that's why people try to take a
look a lot.
It's very incomplete, yes, in comparison to where it started in #16578.
That PR supported pruning in projections, filters, joins and aggregations, and
paved the way for further optimizationsâe.g. window support. This specific PR
started with projections and filters, then removed support for filters, then
removed code that ensured it wouldn't fail under certain scenarios (r.e. the
now ignored tests). I will grant that the latter code definitely falls under
the "gee this seems to fix it" kind of workaround than an actually correct fix,
and I'm not against pursuing the latter. I am uneasy about merging in broken
code.
I have complied with @gatorsmile's requests to remove changes from this PR.
Sometimes in the process I have accidentally left some dangling dead code or
what appears to be some bad design decisions, such as in
https://github.com/apache/spark/pull/21320#issuecomment-407713622. In all of
those cases, this is code that makes more sense in the broader context of
#16578. I could have just kept quiet and complied, or defended myself, but I
didn't have patience for the former nor energy for the latter. At this point,
after about a week of time I am in a better mood to collaborate.
> Also honestly this logic looks convoluted.
I need to point out that I don't know how to respond to comments like
these. I have put my best effort forward. If you want me to change something, I
need more specific guidance. What do you want me to do about this? Don't just
point out a problem, offer a solution or at least a suggestion.
> @ajacques, if you are willing to take over this, please go ahead. I would
appreciate it.
This is probably the best way to get through this PR. I will try to
participate and help. Please post a link to your PR when you open it.
Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
