Guoliang Sun created KYLIN-5345:
-----------------------------------
Summary: Setting the noexec attribute on the /tmp directory causes
build tasks to fail
Key: KYLIN-5345
URL: https://issues.apache.org/jira/browse/KYLIN-5345
Project: Kylin
Issue Type: Bug
Affects Versions: 5.0-alpha
Reporter: Guoliang Sun
Fix For: 5.0-alpha
Attachments: image-2022-12-07-16-32-47-365.png,
image-2022-12-07-16-34-34-370.png
The /tmp directory sets the noexec attribute
!image-2022-12-07-16-32-47-365.png|width=386,height=162!
Execution task build fails in getting flat table statistics
!image-2022-12-07-16-34-34-370.png|width=419,height=262!
See log
{code:java}
2022-11-01T15:25:06,703 INFO [logger-thread-0] spark.SparkContext : Starting
job: count at FlatTableAndDictBase.scala:384
2022-11-01T15:25:06,704 INFO [logger-thread-0] scheduler.DAGScheduler : submit
job : 3, executionId is 1
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler : Got
job 3 (count at FlatTableAndDictBase.scala:384) with 1 output partitions
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler : Final
stage: ResultStage 4 (count at FlatTableAndDictBase.scala:384)
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler :
Parents of final stage: List(ShuffleMapStage 3)
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler :
Missing parents: List()
2022-11-01T15:25:06,706 INFO [logger-thread-0] scheduler.DAGScheduler :
Submitting ResultStage 4 (MapPartitionsRDD[12] at count at
FlatTableAndDictBase.scala:384), which has no missing parents
2022-11-01T15:25:06,707 INFO [logger-thread-0] scheduler.DAGScheduler : submit
stage 4 with jobId: 3
2022-11-01T15:25:06,713 INFO [logger-thread-0] memory.MemoryStore : Block
broadcast_5 stored as values in memory (estimated size 11.0 KiB, free 365.4 MiB)
2022-11-01T15:25:06,714 INFO [logger-thread-0] memory.MemoryStore : Block
broadcast_5_piece0 stored as bytes in memory (estimated size 5.5 KiB, free
365.4 MiB)
2022-11-01T15:25:06,715 INFO [logger-thread-0] storage.BlockManagerInfo :
Added broadcast_5_piece0 in memory on client-junhui.kyligence.com:45347 (size:
5.5 KiB, free: 366.2 MiB)
2022-11-01T15:25:06,715 INFO [logger-thread-0] spark.SparkContext : Created
broadcast 5 from broadcast at DAGScheduler.scala:1430
2022-11-01T15:25:06,716 INFO [logger-thread-0] scheduler.DAGScheduler :
Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[12] at count at
FlatTableAndDictBase.scala:384) (first 15 tasks are for partitions Vector(0))
2022-11-01T15:25:06,716 INFO [logger-thread-0] cluster.YarnScheduler : Adding
task set 4.0 with 1 tasks resource profile 0
2022-11-01T15:25:06,718 INFO [logger-thread-0] scheduler.TaskSetManager :
Starting task 0.0 in stage 4.0 (TID 3) (TID 3 , sandbox.hortonworks.com,
executor 1 , partition 0, NODE_LOCAL , 4464 bytes) taskResourceAssignments Map()
2022-11-01T15:25:08,118 INFO [logger-thread-0] storage.BlockManagerInfo :
Added broadcast_5_piece0 in memory on sandbox.hortonworks.com:34590 (size: 5.5
KiB, free: 366.3 MiB)
2022-11-01T15:25:08,142 INFO [logger-thread-0]
spark.MapOutputTrackerMasterEndpoint : Asked to send map output locations for
shuffle 0 to 10.1.2.194:34547
2022-11-01T15:25:08,250 ERROR [logger-thread-0] spark.MapOutputTrackerMaster :
null
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_171]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_171]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_171]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_171]
at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:87)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
org.apache.spark.MapOutputTracker$.serializeOutputStatuses(MapOutputTracker.scala:1367)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
org.apache.spark.ShuffleStatus.$anonfun$serializedMapStatus$2(MapOutputTracker.scala:326)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
~[scala-library-2.12.15.jar:?]
at
org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:77)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:323)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
org.apache.spark.MapOutputTrackerMaster$MessageLoop.handleStatusMessage(MapOutputTracker.scala:699)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:716)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171]
Caused by: java.lang.UnsatisfiedLinkError:
/tmp/libzstd-jni-1.5.0-43051813466323778737.so:
/tmp/libzstd-jni-1.5.0-43051813466323778737.so: failed to map segment from
shared object: Operation not permitted
no zstd-jni-1.5.0-4 in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni-1.5.0-4.so or load
zstd-jni-1.5.0-4 from system libraries. Please try building from source the jar
or providing libzstd-jni-1.5.0-4 in your system.
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
~[?:1.8.0_171]
at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[?:1.8.0_171]
at java.lang.System.loadLibrary(System.java:1122) ~[?:1.8.0_171]
at com.github.luben.zstd.util.Native.load(Native.java:124)
~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at com.github.luben.zstd.util.Native.load(Native.java:55)
~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at
com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<clinit>(ZstdOutputStreamNoFinalizer.java:18)
~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at
com.github.luben.zstd.RecyclingBufferPool.<clinit>(RecyclingBufferPool.java:17)
~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at
org.apache.spark.io.ZStdCompressionCodec.<init>(CompressionCodec.scala:221)
~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
... 15 more {code}
Since this was found during the tests to verify the flame graph function, the
flame graph code will be revert first and the function will be retuned later
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
