zhztheplayer commented on PR #5678: URL: https://github.com/apache/incubator-gluten/pull/5678#issuecomment-2103776291
@xiaoxmeng We got another regression by Velox's change on arbitrator ``` 24/05/10 02:07:04 ERROR ManagedReservationListener: Error reserving memory from target [6492](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6493)java.lang.RuntimeException: Exception: VeloxRuntimeError [6493](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6494)Error Source: RUNTIME [6494](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6495)Error Code: INVALID_STATE [6495](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6496)Retriable: False [6496](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6497)Expression: arbitrationCtx != nullptr [6497](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6498)Context: Operator: HashProbe[2] 1 [6498](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6499)Function: createAsyncMemoryReclaimTask [6499](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6500)File: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/MemoryArbitrator.h [6500](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6501)Line: 464 [6501](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6502)Stack trace: [6502](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6503)# 0 _ZN8facebook5velox7process10StackTraceC1Ei [6503](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6504)# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_ [6504](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6505)# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_ [6505](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6506)# 3 _ZN8facebook5velox4exec9HashProbe11spillOutputERKSt6vectorIPS2_SaIS4_EE [6506](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6507)# 4 _ZN8facebook5velox4exec9HashProbe7reclaimEmRNS0_6memory15MemoryReclaimer5StatsE [6507](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6508)# 5 _ZNSt17_Function_handlerIFmvEZN8facebook5velox4exec8Operator15MemoryReclaimer7reclaimEPNS2_6memory10MemoryPoolEmmRNS6_15MemoryReclaimer5StatsEEUlvE_E9_M_invokeERKSt9_Any_data [6508](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6509)# 6 _ZN8facebook5velox6memory15MemoryReclaimer3runERKSt8functionIFmvEERNS2_5StatsE [6509](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6510)# 7 _ZN8facebook5velox4exec8Operator15MemoryReclaimer7reclaimEPNS0_6memory10MemoryPoolEmmRNS4_15MemoryReclaimer5StatsE [6510](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6511)# 8 _ZNSt17_Function_handlerIFbPN8facebook5velox6memory10MemoryPoolEEZNS1_4exec23HashJoinMemoryReclaimer7reclaimES4_mmRNS2_15MemoryReclaimer5StatsEEUlS4_E_E9_M_invokeERKSt9_Any_dataOS4_ [6511](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6512)# 9 _ZNK8facebook5velox6memory10MemoryPool13visitChildrenERKSt8functionIFbPS2_EE [6512](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6513)# 10 _ZN8facebook5velox4exec23HashJoinMemoryReclaimer7reclaimEPNS0_6memory10MemoryPoolEmmRNS3_15MemoryReclaimer5StatsE [6513](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6514)# 11 _ZN8facebook5velox6memory15MemoryReclaimer7reclaimEPNS1_10MemoryPoolEmmRNS2_5StatsE [6514](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6515)# 12 _ZN8facebook5velox4exec4Task15MemoryReclaimer11reclaimTaskERKSt10shared_ptrIS2_EmmRNS0_6memory15MemoryReclaimer5StatsE [6515](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6516)# 13 _ZN8facebook5velox4exec4Task15MemoryReclaimer7reclaimEPNS0_6memory10MemoryPoolEmmRNS4_15MemoryReclaimer5StatsE [6516](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6517)# 14 _ZN8facebook5velox6memory15MemoryReclaimer7reclaimEPNS1_10MemoryPoolEmmRNS2_5StatsE [6517](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6518)# 15 _ZN6gluten20ListenableArbitrator14shrinkCapacityERKSt6vectorISt10shared_ptrIN8facebook5velox6memory10MemoryPoolEESaIS7_EEmbb [6518](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6519)# 16 _ZN6gluten24WholeStageResultIterator14spillFixedSizeEl [6519](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6520)# 17 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill [6520](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6521)# 18 0x00007f9e29017de7 [6521](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6522) [6522](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6523) at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeSpill(Native Method) [6523](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6524) at org.apache.gluten.vectorized.ColumnarBatchOutIterator.spill(ColumnarBatchOutIterator.java:83) [6524](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6525) at org.apache.gluten.vectorized.NativePlanEvaluator$1.spill(NativePlanEvaluator.java:82) [6525](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6526) at org.apache.gluten.memory.memtarget.Spillers$WithMinSpillSize.spill(Spillers.java:57) [6526](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6527) at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:90) [6527](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6528) at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:61) [6528](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6529) at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:80) [6529](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6530) at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:61) [6530](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6531) at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:80) [6531](https://github.com/apache/incubator-gluten/actions/runs/9025635959/job/24802744893?pr=5678#step:6:6532) at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:61) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
