[
https://issues.apache.org/jira/browse/IMPALA-13165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fang-Yu Rao updated IMPALA-13165:
---------------------------------
Attachment: generate_junitxml.finalize.minidumps.20240616_21_41_14.xml
> Impala daemon crashed with OMException in Ozone build
> -----------------------------------------------------
>
> Key: IMPALA-13165
> URL: https://issues.apache.org/jira/browse/IMPALA-13165
> Project: IMPALA
> Issue Type: Bug
> Reporter: Fang-Yu Rao
> Assignee: Yida Wu
> Priority: Major
> Labels: broken-build
> Attachments:
> generate_junitxml.finalize.minidumps.20240616_21_41_14.xml
>
>
> We found from an internal build that Impala daemon crashed with a lot of
> OMException in an Ozone build.
> For instance, the backend test
> [Multi8RandomSpillToRemoteMix()|https://github.com/apache/impala/blob/master/be/src/runtime/bufferpool/buffer-pool-test.cc#L2065C24-L2070]
> failed with the following stack trace collected from the generated minidump.
> {code}
> Thread 502 (crashed)
> 0 libc.so.6 + 0x36387
> rax = 0x0000000000000000 rdx = 0x0000000000000006
> rcx = 0xffffffffffffffff rbx = 0x000000000607d920
> rsi = 0x0000000000000cfa rdi = 0x00000000000028ec
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0428
> r8 = 0x0000000000000000 r9 = 0x00007fd6662f02e0
> r10 = 0x0000000000000008 r11 = 0x0000000000000202
> r12 = 0x000000000607d920 r13 = 0x000000000607d980
> r14 = 0x0000000000000152 r15 = 0x0000000000000223
> rip = 0x00007fd77dbd1387
> Found by: given as instruction pointer in context
> 1 libc.so.6 + 0x37a78
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0430
> rip = 0x00007fd77dbd2a78
> Found by: stack scanning
> 2 buffer-pool-test!google_breakpad::ExceptionHandler::HandleSignal(int,
> siginfo_t*, void*) + 0x1a0
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f04b8
> rip = 0x0000000003a29e40
> Found by: stack scanning
> 3 buffer-pool-test!tcmalloc::ThreadCache::FetchFromCentralCache(unsigned
> int, int, void* (*)(unsigned long)) + 0x68
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f04f0
> rip = 0x0000000003b6f858
> Found by: stack scanning
> 4 buffer-pool-test!tcmalloc::malloc_oom(unsigned long) + 0xc0
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0500
> rip = 0x0000000003d07f20
> Found by: stack scanning
> 5 buffer-pool-test!google::(anonymous namespace)::FailureSignalHandler(int,
> siginfo_t*, void*) [clone .part.0] + 0xad0
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0558
> rip = 0x00000000039faa00
> Found by: stack scanning
> 6 buffer-pool-test!google::DumpStackTraceAndExit() [clone .cold] + 0x5
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0560
> rip = 0x0000000000f00e4f
> Found by: stack scanning
> 7 libstdc++.so.6 + 0x13aa48
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0570
> rip = 0x00007fd78132ea48
> Found by: stack scanning
> 8 libstdc++.so.6 + 0x13aa48
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0580
> rip = 0x00007fd78132ea48
> Found by: stack scanning
> 9 libstdc++.so.6 + 0x11f8e2
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f05b0
> rip = 0x00007fd7813138e2
> Found by: stack scanning
> 10
> buffer-pool-test!google::LogDestination::WaitForSinks(google::LogMessage::LogMessageData*)
> + 0x110
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f05e0
> rip = 0x00000000039f6460
> Found by: stack scanning
> 11 buffer-pool-test!google::LogMessage::Fail() + 0xd
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0610
> rip = 0x00000000039ef6bd
> Found by: stack scanning
> 12 buffer-pool-test!google::LogMessage::SendToLog() + 0x244
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0620
> rip = 0x00000000039f15f4
> Found by: stack scanning
> 13 libstdc++.so.6 + 0x12cae4
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0640
> rip = 0x00007fd781320ae4
> Found by: stack scanning
> 14 buffer-pool-test!_fini + 0x19b3
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0648
> rip = 0x0000000003d0cb03
> Found by: stack scanning
> 15 buffer-pool-test!_fini + 0xa7c14
> rbp = 0x00007fd6662f06e0 rsp = 0x00007fd6662f0658
> rip = 0x0000000003db2d64
> Found by: stack scanning
> 16 buffer-pool-test!google::LogMessage::Flush() + 0x1ec
> rsp = 0x00007fd6662f06f0 rip = 0x00000000039ef09c
> Found by: stack scanning
> 17 libstdc++.so.6 + 0x12cae4
> rsp = 0x00007fd6662f0730 rip = 0x00007fd781320ae4
> Found by: stack scanning
> 18 buffer-pool-test!google::LogMessageFatal::~LogMessageFatal() + 0x9
> rsp = 0x00007fd6662f0790 rip = 0x00000000039f1b19
> Found by: stack scanning
> 19
> buffer-pool-test!impala::BufferPoolTest::TestRandomInternalImpl(impala::BufferPool*,
> impala::TmpFileGroup*, impala::MemTracker*,
> std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul,
> 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul,
> 18ul, 1812433253ul>*, int, bool) [buffer-pool.h : 338 + 0x8]
> rsp = 0x00007fd6662f07a0 rip = 0x0000000000f8721f
> Found by: stack scanning
> {code}
> During the crash we also saw quite a few OMException from the console output.
> {code}
> 08:46:11
> hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/a44cc3c871369491_8dcaa671747530a3_0000000000000000_0000000000000000/impala-scratch-ae339172-59d6-41ef-9a6a-249c4d9ff537):
>
> FileSystem#create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/a44cc3c871369491_8dcaa671747530a3_0000000000000000_0000000000000000/impala-scratch-b305d4f9-8e61-4b96-afd4-9940bd8f48b6):
>
> FileSystem#create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)
> error:
> 08:46:11
> hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/a44cc3c871369491_8dcaa671747530a3_0000000000000000_0000000000000000/impala-scratch-45a6a781-55f7-44aa-9a06-2ed6a6242e92):
>
> FileSystem#create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/a44cc3c871369491_8dcaa671747530a3_0000000000000000_0000000000000000/impala-scratch-0f52e72b-2cc0-4697-a1aa-838891422844):
>
> FileSystem#create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/a44cc3c871369491_8dcaa671747530a3_0000000000000000_0000000000000000/impala-scratch-4d43c8b9-e062-474a-945c-9e20e4d50998):
>
> FileSystem#create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)
> error:
> 08:46:11
> hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/a44cc3c871369491_8dcaa671747530a3_0000000000000000_0000000000000000/impala-scratch-40a6653e-3b88-4c62-8d4e-1f23e2e5eb9c):
>
> FileSystem#create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)
> error:
> 08:46:11 error:
> 08:46:11 error:
> 08:46:11 OMException: Allocated 0 blocks. Requested 1 blocksINTERNAL_ERROR
> org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks.
> Requested 1 blocks
> 08:46:11 at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:728)
> 08:46:11 at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createFile(OzoneManagerProtocolClientSideTranslatorPB.java:2133)
> 08:46:11 at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createFile(RpcClient.java:2001)
> 08:46:11 at
> org.apache.hadoop.ozone.client.OzoneBucket.createFile(OzoneBucket.java:822)
> 08:46:11 at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.createFile(BasicRootedOzoneClientAdapterImpl.java:389)
> 08:46:11 at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.createOutputStream(BasicRootedOzoneFileSystem.java:299)
> 08:46:11 at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.create(BasicRootedOzoneFileSystem.java:261)
> 08:46:11 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1177)
> 08:46:11 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1157)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
