Re: [I] Crash when writing an array of struct [incubator-gluten]
clee704 commented on issue #4964: URL: https://github.com/apache/incubator-gluten/issues/4964#issuecomment-2033315974 Confirmed it's fixed in the latest commit. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] Crash when writing an array of struct [incubator-gluten]
clee704 closed issue #4964: Crash when writing an array of struct URL: https://github.com/apache/incubator-gluten/issues/4964 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] Crash when writing an array of struct [incubator-gluten]
JkSelf commented on issue #4964: URL: https://github.com/apache/incubator-gluten/issues/4964#issuecomment-2030926990 @clee704 Sorry for delayed response. It seems this issue has been fixed [here](https://github.com/apache/incubator-gluten/blob/main/cpp/velox/substrait/SubstraitToVeloxExpr.cc#L494). Can you help to verify in your environment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] Crash when writing an array of struct [incubator-gluten]
clee704 commented on issue #4964: URL: https://github.com/apache/incubator-gluten/issues/4964#issuecomment-2021599373 @JkSelf Actually it crashes on Spark 3.4 too. # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7fa4a369e2c5, pid=915090, tid=915112 # # JRE version: OpenJDK Runtime Environment (11.0.22+7) (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1) # Java VM: OpenJDK 64-Bit Server VM (11.0.22+7-post-Ubuntu-0ubuntu222.04.1, mixed mode, tiered, g1 gc, linux-amd64) # Problematic frame: # C [libvelox.so+0x229e2c5] (anonymous namespace)::makeRowVector(std::vector, std::allocator > > const&)+0xa5 # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /ssd/chungmin/repos/spark/core.915090) # # An error report file with more information is saved as: # /ssd/chungmin/repos/spark/hs_err_pid915090.log # # If you would like to submit a bug report, please visit: # https://bugs.launchpad.net/ubuntu/+source/openjdk-lts # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # Spark 3.4.2 Gluten 58a459bf487120208a774d7959f7c7db417f490b -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] Crash when writing an array of struct [incubator-gluten]
JkSelf commented on issue #4964: URL: https://github.com/apache/incubator-gluten/issues/4964#issuecomment-2017458835 @clee704 @zhouyuan The Velox backend lacks support for writing complex types. Spark 34 has already implemented a fallback mechanism. PR (https://github.com/apache/incubator-gluten/pull/5107) has been filed to extend this fallback to Spark versions 32 and 33. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] Crash when writing an array of struct [incubator-gluten]
zhouyuan commented on issue #4964: URL: https://github.com/apache/incubator-gluten/issues/4964#issuecomment-1998834795 It seems the complex data type is not correctly fallback there https://github.com/apache/incubator-gluten/issues/4110 CC: @JkSelf -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
[I] Crash when writing an array of struct [incubator-gluten]
clee704 opened a new issue, #4964:
URL: https://github.com/apache/incubator-gluten/issues/4964
### Backend
VL (Velox)
### Bug description
This write operation crashes:
`spark.sql("select array(struct(1),
null)").write.mode("overwrite").save("X")`
### Spark version
Spark-3.3.x
### Spark configurations
spark.plugins=io.glutenproject.GlutenPlugin
spark.gluten.sql.columnar.backend.lib=velox
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=28g
### System information
Velox System Info v0.0.2
Commit: 58a459bf487120208a774d7959f7c7db417f490b
CMake Version: 3.25.1
System: Linux-6.5.0-1015-azure
Arch: x86_64
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path:
/usr/local;/usr;/;/ssd/linuxbrew/.linuxbrew/Cellar/cmake/3.25.1;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x7fb40369e2c5, pid=281178, tid=0x7fc434d18640
#
# JRE version: OpenJDK Runtime Environment (8.0_392-b08) (build
1.8.0_392-8u392-ga-1~22.04-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.392-b08 mixed mode linux-amd64 )
# Problematic frame:
# C [libvelox.so+0x229e2c5] (anonymous
namespace)::makeRowVector(std::vector,
std::allocator > > const&)+0xa5
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /ssd/chungmin/repos/spark_3.3/hs_err_pid281178.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
