[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8294/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 22:04:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. 2. Created a large table which has ~270K partitions and reduced the memory of the catalogd to 16GB. This configuration throws bad_alloc exception in the ThriftDebugString method and crashes the catalogd. After the patch the crash is averted and we see a error message on the debug UI instead. I also looped around the catalog web UI call for more than an hour to see if there are any other stability issues. I could not see any problems. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Reviewed-on: http://gerrit.cloudera.org:8080/17110 Reviewed-by: Vihang Karajgaonkar Tested-by: Vihang Karajgaonkar --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 121 insertions(+), 6 deletions(-) Approvals: Vihang Karajgaonkar: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 9: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 21:44:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. 2. Created a large table which has ~270K partitions and reduced the memory of the catalogd to 16GB. This configuration throws bad_alloc exception in the ThriftDebugString method and crashes the catalogd. After the patch the crash is averted and we see a error message on the debug UI instead. I also looped around the catalog web UI call for more than an hour to see if there are any other stability issues. I could not see any problems. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 121 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/9 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 06:52:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8275/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 01:20:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6930/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 01:01:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 8: Code-Review+2 (1 comment) Carrying forward previous +2 from Joe and Quanlong. http://gerrit.cloudera.org:8080/#/c/17110/7/tests/custom_cluster/test_thrift_debug_string_exception.py File tests/custom_cluster/test_thrift_debug_string_exception.py: http://gerrit.cloudera.org:8080/#/c/17110/7/tests/custom_cluster/test_thrift_debug_string_exception.py@42 PS7, Line 42: est_thrift_debug_str(self): : """Sanity test which executes API call to get a catalog object and make sure th > Nit: Could you note in this comment that this is a positive test where it s Thanks for the comment. I update the documentation for the test methods as suggested. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 01:00:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 121 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/8 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 7: Code-Review+2 (1 comment) This makes sense to me. One small nit on a comment. Since Quanlong has already +1'd, I'm going to take this to +2. http://gerrit.cloudera.org:8080/#/c/17110/7/tests/custom_cluster/test_thrift_debug_string_exception.py File tests/custom_cluster/test_thrift_debug_string_exception.py: http://gerrit.cloudera.org:8080/#/c/17110/7/tests/custom_cluster/test_thrift_debug_string_exception.py@42 PS7, Line 42: The test executes a API call to get a catalog object from the debug UI and makes : sure that catalogd does not crash if the ThriftDebugString throws an exception. Nit: Could you note in this comment that this is a positive test where it should not throw an exception? -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 02 Mar 2021 00:53:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 7: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/17110/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17110/6//COMMIT_MSG@13 PS6, Line 13: caught on the catalogd side and it crashes the catalogd. Thanks for digging into this! The codes look safe for me. > One thing which I tried is I updated the thrift library to throw a exception > and I looped around on the catalogd web API call for an hour. I observed the > memory usage of catalogd and it was stable. That's cool! I'm ok if we can't reproduce the exception. I think this is enough. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 01 Mar 2021 06:12:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8248/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Feb 2021 01:20:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/17110/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17110/6//COMMIT_MSG@13 PS6, Line 13: caught on the catalogd side and it crashes the catalogd. > Will the thrift library has memory leak or corrupt memory after this? It'd the code in thrift which expands the memory buffer tries to allocate first than then assign to the pointer as seen here: https://github.com/apache/thrift/blob/53dd39833a08ce33582e5ff31fa18bb4735d6731/lib/cpp/src/thrift/transport/TBufferTransports.cpp#L371 Based on my understanding, realloc will only reassign the pointer if it is able to allocate the memory otherwise it returns a nullptr and doesn't modify the original pointer (ref: https://en.cppreference.com/w/cpp/memory/c/realloc) Since the TMemoryBuffer is defined as a local variable in ThriftDebugString here (https://github.com/apache/thrift/blob/6140fb27e0b15de2ba042401073435b049482389/lib/cpp/src/thrift/protocol/TDebugProtocol.h#L162) when a exception is thrown, the memory buffer is freed up by the destructor as defined here https://github.com/apache/thrift/blob/6140fb27e0b15de2ba042401073435b049482389/lib/cpp/src/thrift/transport/TBufferTransports.h#L565. So think we should not have a leak or corrupt memory. I will see if I can reproduce this somehow. If you have a any ideas, I can try them out. One thing which I tried is I updated the thrift library to throw a exception and I looped around on the catalogd web API call for an hour. I observed the memory usage of catalogd and it was stable. http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/debug-util.h File be/src/util/debug-util.h: http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/debug-util.h@196 PS6, Line 196: EXCEPTION_STR_MAP > nit: EXCEPTION_STR_MAP Done http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/thrift-debug-util.h File be/src/util/thrift-debug-util.h: http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/thrift-debug-util.h@33 PS6, Line 33: > nit: I think we can add 'noexcept' here Done -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Feb 2021 01:01:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 120 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/7 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 6: Code-Review+1 (3 comments) http://gerrit.cloudera.org:8080/#/c/17110/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17110/6//COMMIT_MSG@13 PS6, Line 13: caught on the catalogd side and it crashes the catalogd. Will the thrift library has memory leak or corrupt memory after this? It'd be good if we can reproduce the exception in a stress test (manually) and see if catalogd still works normally after catching the exception. http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/debug-util.h File be/src/util/debug-util.h: http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/debug-util.h@196 PS6, Line 196: exception_str_map nit: EXCEPTION_STR_MAP http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/thrift-debug-util.h File be/src/util/thrift-debug-util.h: http://gerrit.cloudera.org:8080/#/c/17110/6/be/src/util/thrift-debug-util.h@33 PS6, Line 33: nit: I think we can add 'noexcept' here -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 25 Feb 2021 02:28:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 22:07:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 120 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/6 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8228/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 21:35:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/17110/5/tests/custom_cluster/test_thrift_debug_string_exception.py File tests/custom_cluster/test_thrift_debug_string_exception.py: http://gerrit.cloudera.org:8080/#/c/17110/5/tests/custom_cluster/test_thrift_debug_string_exception.py@19 PS5, Line 19: from tests.common.skip import SkipIfS3, SkipIfABFS, SkipIfADLS, SkipIfIsilon, SkipIfLocal flake8: F401 'tests.common.skip.SkipIfIsilon' imported but unused http://gerrit.cloudera.org:8080/#/c/17110/5/tests/custom_cluster/test_thrift_debug_string_exception.py@19 PS5, Line 19: from tests.common.skip import SkipIfS3, SkipIfABFS, SkipIfADLS, SkipIfIsilon, SkipIfLocal flake8: F401 'tests.common.skip.SkipIfS3' imported but unused http://gerrit.cloudera.org:8080/#/c/17110/5/tests/custom_cluster/test_thrift_debug_string_exception.py@19 PS5, Line 19: from tests.common.skip import SkipIfS3, SkipIfABFS, SkipIfADLS, SkipIfIsilon, SkipIfLocal flake8: F401 'tests.common.skip.SkipIfABFS' imported but unused http://gerrit.cloudera.org:8080/#/c/17110/5/tests/custom_cluster/test_thrift_debug_string_exception.py@19 PS5, Line 19: from tests.common.skip import SkipIfS3, SkipIfABFS, SkipIfADLS, SkipIfIsilon, SkipIfLocal flake8: F401 'tests.common.skip.SkipIfLocal' imported but unused http://gerrit.cloudera.org:8080/#/c/17110/5/tests/custom_cluster/test_thrift_debug_string_exception.py@19 PS5, Line 19: from tests.common.skip import SkipIfS3, SkipIfABFS, SkipIfADLS, SkipIfIsilon, SkipIfLocal flake8: F401 'tests.common.skip.SkipIfADLS' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 21:16:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. In order to simulate this crash a new debug action called EXCEPTION was added. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 121 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/5 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8224/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 19:17:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8223/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 19:17:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/17110/3/tests/custom_cluster/test_thrift_debug_string_exception.py File tests/custom_cluster/test_thrift_debug_string_exception.py: http://gerrit.cloudera.org:8080/#/c/17110/3/tests/custom_cluster/test_thrift_debug_string_exception.py@21 PS3, Line 21: @SkipIfS3.hive flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/17110/3/tests/custom_cluster/test_thrift_debug_string_exception.py@55 PS3, Line 55: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 18:59:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/17110/2/tests/custom_cluster/test_thrift_debug_string_exception.py File tests/custom_cluster/test_thrift_debug_string_exception.py: http://gerrit.cloudera.org:8080/#/c/17110/2/tests/custom_cluster/test_thrift_debug_string_exception.py@21 PS2, Line 21: @SkipIfS3.hive flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/17110/2/tests/custom_cluster/test_thrift_debug_string_exception.py@55 PS2, Line 55: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 24 Feb 2021 18:58:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17110 ) Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. A new custom cluster test was added which simulates a exception thrown in this method and makes sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 125 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/3 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17110 Change subject: IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString .. IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString This patch adds a wrapper around ThriftDebugString method provided in the Thrift library. The thrift's method can throw exceptions like (bad_alloc or TProtocolException) when the object cannot be serialized into a string representation. This exception is not caught on the catalogd side and it crashes the catalogd. The error was specifically seen in the catalogd's debug UI which provides a way to display a Table object. An exception thrown when rendering the table on the UI would have crashed the catalogd before the patch. A new custom cluster test was added which simulates a exception thrown in this method and making sure that fetching the table from catalogd's debug UI does not crash the catalogd. Tests: 1. Added a new custom cluster test which reproduces the crash. Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 --- M be/src/catalog/catalog-server.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h A be/src/util/thrift-debug-util.h A tests/custom_cluster/test_thrift_debug_string_exception.py 5 files changed, 125 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/17110/2 -- To view, visit http://gerrit.cloudera.org:8080/17110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66 Gerrit-Change-Number: 17110 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
