sandugood commented on PR #5051:
URL:
https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5254380631
> > Tested on Spark4.0 against current branch. Using PySpark Got this error
on the same pipeline that was testing earlier. And I can clearly see a
regression in performance. Same job with `.cache()` on default Spark took <10
min, Comet ran >15min.
> > Here is error:
> > ```
> > Py4JJavaError: An error occurred while calling o12641.createOrReplace.
> > : org.apache.spark.SparkException: Job aborted due to stage failure:
Task 92 in stage 27.0 failed 4 times, most recent failure: Lost task 92.3 in
stage 27.0 (TID 15624) (1.1.1.1 executor 76):
org.apache.comet.CometNativeException: Invalid argument error: column types
must match schema types, expected List(Struct("col_1": Utf8, "col_2": Boolean,
"col_3": Boolean)) but found List(Struct("col_1": Utf8, "col_2": non-null
Boolean, "col_3": Boolean)) at column index 5
> > at org.apache.comet.Native.executePlan(Native Method)
> > at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
> > at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:154)
> > at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:189)
> > at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:154)
> > at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
> > at
org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:152)
> > at
org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:196)
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > (obfuscated column names)
>
> I think the crash may get fixed once #5138 is merged. Looking into perf
regression...
Thanks, @andygrove
Would love to try it out once ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
