Re: strange behavior of joining dataframes

2018-03-23 Thread Shiyuan
Here is a simple example that reproduces the problem. This code has a missing attribute('kk') error. Is it a bug? Note that if the `select` in line B is removed, this code would run. import pyspark.sql.functions as F df =

strange behavior of joining dataframes

2018-03-20 Thread Shiyuan
Hi Spark-users: I have a dataframe "df_t" which was generated from other dataframes by several transformations. And then I did something very simple, just counting the rows, that is the following code: (A) df_t_1 = df_t.groupby(["Id","key"]).count().withColumnRenamed("count", "cnt1") df_t_2 =