[ https://issues.apache.org/jira/browse/SPARK-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116297#comment-15116297 ]
Emlyn Corrin commented on SPARK-9740: ------------------------------------- Here's the stack trace, it fails during compilation: {code} java.lang.UnsupportedOperationException: 'last('timestamp,true) is not supported in a window operation., compiling:(test.clj:8:1) Exception in thread "main" java.lang.UnsupportedOperationException: 'last('timestamp,true) is not supported in a window operation., compiling:(test.clj:8:1) at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3657) at clojure.lang.Compiler$IfExpr.eval(Compiler.java:2695) at clojure.lang.Compiler.compile1(Compiler.java:7474) at clojure.lang.Compiler.compile(Compiler.java:7541) at clojure.lang.RT.compile(RT.java:406) at clojure.lang.RT.load(RT.java:451) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at sparkalytics.core$loading__5569__auto____7689.invoke(core.clj:1) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3652) at clojure.lang.Compiler.compile1(Compiler.java:7474) at clojure.lang.Compiler.compile1(Compiler.java:7464) at clojure.lang.Compiler.compile(Compiler.java:7541) at clojure.lang.RT.compile(RT.java:406) at clojure.lang.RT.load(RT.java:451) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$compile$fn__5682.invoke(core.clj:5903) at clojure.core$compile.invokeStatic(core.clj:5903) at user$eval13$fn__22.invoke(form-init8921303069087101413.clj:1) at user$eval13.invokeStatic(form-init8921303069087101413.clj:1) at user$eval13.invoke(form-init8921303069087101413.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6917) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.Compiler.loadFile(Compiler.java:7317) at clojure.main$load_script.invokeStatic(main.clj:275) at clojure.main$init_opt.invokeStatic(main.clj:277) at clojure.main$init_opt.invoke(main.clj:277) at clojure.main$initialize.invokeStatic(main.clj:308) at clojure.main$null_opt.invokeStatic(main.clj:342) at clojure.main$null_opt.invoke(main.clj:339) at clojure.main$main.invokeStatic(main.clj:421) at clojure.main$main.doInvoke(main.clj:384) at clojure.lang.RestFn.invoke(RestFn.java:421) at clojure.lang.Var.invoke(Var.java:383) at clojure.lang.AFn.applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var.java:700) at clojure.main.main(main.java:37) Caused by: java.lang.UnsupportedOperationException: 'last('timestamp,true) is not supported in a window operation. at org.apache.spark.sql.expressions.WindowSpec.withAggregate(WindowSpec.scala:191) at org.apache.spark.sql.Column.over(Column.scala:1049) at sparkalytics.jobs.test$fn__8460.invokeStatic(test.clj:10) at sparkalytics.jobs.test$fn__8460.invoke(test.clj:8) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3652) ... 59 more Compilation failed: Subprocess failed {code} (I'm not sure how to get the 59 missing lines from the end, but hopefully this is enough) > first/last aggregate NULL behavior > ---------------------------------- > > Key: SPARK-9740 > URL: https://issues.apache.org/jira/browse/SPARK-9740 > Project: Spark > Issue Type: Sub-task > Components: SQL > Reporter: Herman van Hovell > Assignee: Yin Huai > Labels: releasenotes > Fix For: 1.6.0 > > > The FIRST/LAST aggregates implemented as part of the new UDAF interface, > return the first or last non-null value (if any) found. This is a departure > from the behavior of the old FIRST/LAST aggregates and from the > FIRST_VALUE/LAST_VALUE aggregates in Hive. These would return a null value, > if that happened to be the first/last value seen. SPARK-9592 tries to 'fix' > this behavior for the old UDAF interface. > Hive makes this behavior configurable, by adding a skipNulls flag. I would > suggest to do the same, and make the default behavior compatible with Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org