Hi Ryan, My goal with this email thread is to discuss with the community if there are better ideas (as I was told many other people tried to address this). I'd consider this as a brainstorming email thread. Once we have a good proposal, then we can go ahead with a SPIP.
Thanks, Marco Il giorno mer 12 dic 2018 alle ore 19:13 Ryan Blue <rb...@netflix.com> ha scritto: > Marco, > > I'm actually asking for a design doc that clearly states the problem and > proposes a solution. This is a substantial change and probably should be an > SPIP. > > I think that would be more likely to generate discussion than referring to > PRs or a quick paragraph on the dev list, because the only people that are > looking at it now are the ones already familiar with the problem. > > rb > > On Wed, Dec 12, 2018 at 2:05 AM Marco Gaido <marcogaid...@gmail.com> > wrote: > >> Thank you all for your answers. >> >> @Ryan Blue <rb...@netflix.com> sure, let me state the problem more >> clearly: imagine you have 2 dataframes with a common lineage (for instance >> one is derived from the other by some filtering or anything you prefer). >> And imagine you want to join these 2 dataframes. Currently, there is a fix >> by Reynold which deduplicates the join condition in case the condition is >> an equality one (please notice that in this case, it doesn't matter which >> one is on the left and which one on the right). But if the condition >> involves other comparisons, such as a ">" or a "<", this would result in an >> analysis error, because the attributes on both sides are the same (eg. you >> have the same id#3 attribute on both sides), and you cannot deduplicate >> them blindly as which one is on a specific side matters. >> >> @Reynold Xin <r...@databricks.com> my proposal was to add a dataset id >> in the metadata of each attribute, so that in this case we can distinguish >> from which dataframe the attribute is coming from, ie. having the >> DataFrames `df1` and `df2` where `df2` is derived from `df1`, >> `df1.join(df2, df1("a") > df2("a"))` could be resolved because we would >> know that the first attribute is taken from `df1` and so it has to be >> resolved using it and the same for the other. But I am open to any approach >> to this problem, if other people have better ideas/suggestions. >> >> Thanks, >> Marco >> >> Il giorno mar 11 dic 2018 alle ore 18:31 Jörn Franke < >> jornfra...@gmail.com> ha scritto: >> >>> I don’t know your exact underlying business problem, but maybe a graph >>> solution, such as Spark Graphx meets better your requirements. Usually >>> self-joins are done to address some kind of graph problem (even if you >>> would not describe it as such) and is for these kind of problems much more >>> efficient. >>> >>> Am 11.12.2018 um 12:44 schrieb Marco Gaido <marcogaid...@gmail.com>: >>> >>> Hi all, >>> >>> I'd like to bring to the attention of a more people a problem which has >>> been there for long, ie, self joins. Currently, we have many troubles with >>> them. This has been reported several times to the community and seems to >>> affect many people, but as of now no solution has been accepted for it. >>> >>> I created a PR some time ago in order to address the problem ( >>> https://github.com/apache/spark/pull/21449), but Wenchen mentioned he >>> tried to fix this problem too but so far no attempt was successful because >>> there is no clear semantic ( >>> https://github.com/apache/spark/pull/21449#issuecomment-393554552). >>> >>> So I'd like to propose to discuss here which is the best approach for >>> tackling this issue, which I think would be great to fix for 3.0.0, so if >>> we decide to introduce breaking changes in the design, we can do that. >>> >>> Thoughts on this? >>> >>> Thanks, >>> Marco >>> >>> > > -- > Ryan Blue > Software Engineer > Netflix >