Extending GraphFrames without running into serialization issues

2021-01-05 Thread Michal Monselise
Hi, I am trying to extend GraphFrames and create my own class that has some additional graph functionality. To simplify for this example, I have created a class that doesn't contain any functions. All it does is just extend GraphFrames: import org.apache.spark.sql.DataFrameimport

Re: Join with multiple conditions (In reference to SPARK-7197)

2015-08-26 Thread Michal Monselise
Davies, I created an issue - SPARK-10246 https://issues.apache.org/jira/browse/SPARK-10246 On Tue, Aug 25, 2015 at 12:53 PM, Davies Liu dav...@databricks.com wrote: It's good to support this, could you create a JIRA for it and target for 1.6? On Tue, Aug 25, 2015 at 11:21 AM, Michal

Fwd: Join with multiple conditions (In reference to SPARK-7197)

2015-08-25 Thread Michal Monselise
Hello All, PySpark currently has two ways of performing a join: specifying a join condition or column names. I would like to perform a join using a list of columns that appear in both the left and right DataFrames. I have created an example in this question on Stack Overflow