To add more details to what Reynold mentioned. As you said, there is going to be some slight differences in any case between Pandas and Spark in any case, simply because Spark needs to know the return types of the functions. In your case, you would need to slightly refactor your apply method to the following (in python 3) to add type hints:
``` def f(x) -> float: return x * 3.0 df['col3'] = df['col1'].apply(f) ``` This has the benefit of keeping your code fully compliant with both pandas and pyspark. We will share more information in the future. Tim On Tue, Mar 26, 2019 at 8:08 AM Hyukjin Kwon <gurwls...@gmail.com> wrote: > BTW, I am working on the documentation related with this subject at > https://issues.apache.org/jira/browse/SPARK-26022 to describe the > difference > > 2019년 3월 26일 (화) 오후 3:34, Reynold Xin <r...@databricks.com>님이 작성: > >> We have some early stuff there but not quite ready to talk about it in >> public yet (I hope soon though). Will shoot you a separate email on it. >> >> On Mon, Mar 25, 2019 at 11:32 PM Abdeali Kothari < >> abdealikoth...@gmail.com> wrote: >> >>> Thanks for the reply Reynold - Has this shim project started ? >>> I'd love to contribute to it - as it looks like I have started making a >>> bunch of helper functions to do something similar for my current task and >>> would prefer not doing it in isolation. >>> Was considering making a git repo and pushing stuff there just today >>> morning. But if there's already folks working on it - I'd prefer >>> collaborating. >>> >>> Note - I'm not recommending we make the logical plan mutable (as I am >>> scared of that too!). I think there are other ways of handling that - but >>> we can go into details later. >>> >>> On Tue, Mar 26, 2019 at 11:58 AM Reynold Xin <r...@databricks.com> >>> wrote: >>> >>>> We have been thinking about some of these issues. Some of them are >>>> harder to do, e.g. Spark DataFrames are fundamentally immutable, and making >>>> the logical plan mutable is a significant deviation from the current >>>> paradigm that might confuse the hell out of some users. We are considering >>>> building a shim layer as a separate project on top of Spark (so we can make >>>> rapid releases based on feedback) just to test this out and see how well it >>>> could work in practice. >>>> >>>> On Mon, Mar 25, 2019 at 11:04 PM Abdeali Kothari < >>>> abdealikoth...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> I was doing some spark to pandas (and vice versa) conversion because >>>>> some of the pandas codes we have don't work on huge data. And some spark >>>>> codes work very slow on small data. >>>>> >>>>> It was nice to see that pyspark had some similar syntax for the common >>>>> pandas operations that the python community is used to. >>>>> >>>>> GroupBy aggs: df.groupby(['col2']).agg({'col2': 'count'}).show() >>>>> Column selects: df[['col1', 'col2']] >>>>> Row Filters: df[df['col1'] < 3.0] >>>>> >>>>> I was wondering about a bunch of other functions in pandas which >>>>> seemed common. And thought there must've been a discussion about it in the >>>>> community - hence started this thread. >>>>> >>>>> I was wondering whether there has been discussion on adding the >>>>> following functions: >>>>> >>>>> *Column setters*: >>>>> In Pandas: >>>>> df['col3'] = df['col1'] * 3.0 >>>>> While I do the following in PySpark: >>>>> df = df.withColumn('col3', df['col1'] * 3.0) >>>>> >>>>> *Column apply()*: >>>>> In Pandas: >>>>> df['col3'] = df['col1'].apply(lambda x: x * 3.0) >>>>> While I do the following in PySpark: >>>>> df = df.withColumn('col3', F.udf(lambda x: x * 3.0, 'float')( >>>>> df['col1'])) >>>>> >>>>> I understand that this one cannot be as simple as in pandas due to the >>>>> output-type that's needed here. But could be done like: >>>>> df['col3'] = df['col1'].apply((lambda x: x * 3.0), 'float') >>>>> >>>>> Multi column in pandas is: >>>>> df['col3'] = df[['col1', 'col2']].apply(lambda x: x.col1 * 3.0) >>>>> Maybe this can be done in pyspark as or if we can send a >>>>> pyspark.sql.Row directly it would be similar (?): >>>>> df['col3'] = df[['col1', 'col2']].apply((lambda col1, col2: col1 * >>>>> 3.0), 'float') >>>>> >>>>> *Rename*: >>>>> In Pandas: >>>>> df.rename(columns={...}) >>>>> While I do the following in PySpark: >>>>> df.toDF(*[{'col2': 'col3'}.get(i, i) for i in df.columns]) >>>>> >>>>> *To Dictionary*: >>>>> In Pandas: >>>>> df.to_dict(orient='list') >>>>> While I do the following in PySpark: >>>>> {f.name: [row[i] for row in df.collect()] for i, f in >>>>> enumerate(df.schema.fields)} >>>>> >>>>> I thought I'd start the discussion with these and come back to some of >>>>> the others I see that could be helpful. >>>>> >>>>> *Note*: (with the column functions in mind) I understand the concept >>>>> of the DataFrame cannot be modified. And I am not suggesting we change >>>>> that >>>>> nor any underlying principle. Just trying to add syntactic sugar here. >>>>> >>>>>
