RE: pyspark.sql.functions.last not working as expected

2016-08-18 Thread Alexander Peletz
Is the issue that the default rangeBetween = rangeBetween(-sys.maxsize, 0)? That would explain the behavior below. Is this default documented somewhere? From: Alexander Peletz [mailto:alexand...@slalom.com] Sent: Wednesday, August 17, 2016 8:48 PM To: user <user@spark.apache.org> Subje

RE: pyspark.sql.functions.last not working as expected

2016-08-17 Thread Alexander Peletz
, null, null, null, null, null, null))) + } I would expect the correct results to be as follows instead of what is used above. Shouldn't we always return the first or last value in the partition based on the ordering? It looks something else is going on... can someone explain? +

pyspark.sql.functions.last not working as expected

2016-08-17 Thread Alexander Peletz
this by having a window that is in reverse order and use first() instead of last() but it would be great if last() actually worked. Thanks, Alexander Alexander Peletz Consultant slalom Fortune 100 Best Companies to Work For 2016 Glassdoor Best Places to Work 2016 Consulting Magazine Best Firms