[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 14: I have resolved the JIRA and changed it's status to resolved. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 14 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 14: Thanks for your contribution! Please go ahead and resolve the JIRA when you have a chance. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 14 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Reviewed-on: http://gerrit.cloudera.org:8080/4971 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 33 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 14 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 13 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 13: Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/136/ -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 13 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 13: Code-Review+2 Needed a rebase -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 13 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 12: Verified-1 Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/135/ -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 12: Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/135/ -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 12: Code-Review+2 This seems like it should be uncontroversial, particularly since the default behaviour is unchanged. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 12: (2 comments) Thank you. I have modified the code according to your opinion. http://gerrit.cloudera.org:8080/#/c/4971/11/be/src/common/logging.cc File be/src/common/logging.cc: PS11, Line 181: max_log_files <= > nit: rewrite as <= 0 to be consistent with other places Done http://gerrit.cloudera.org:8080/#/c/4971/11/be/src/service/impala-server.h File be/src/service/impala-server.h: Line 272: static const string AUDIT_EVENT_LOG_FILE_PREFIX; > Let's remove this function and simplify things by making AUDIT_EVENT_LOG_FI Done -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Hello Lars Volker, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4971 to look at the new patch set (#12). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 33 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/12 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: (2 comments) Looks good - couple of minor comments http://gerrit.cloudera.org:8080/#/c/4971/11/be/src/common/logging.cc File be/src/common/logging.cc: PS11, Line 181: max_log_files < 1 nit: rewrite as <= 0 to be consistent with other places http://gerrit.cloudera.org:8080/#/c/4971/11/be/src/service/impala-server.h File be/src/service/impala-server.h: Line 272: static std::string GetAuditEventLogFilePrefix(); Let's remove this function and simplify things by making AUDIT_EVENT_LOG_FILE_PREFIX a public member of ImpalaServer. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: Is there any expert review my code? Thanks. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: Code-Review+1 Let's someone else have a look. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: Audit event log has no link file, so there is no link file switch problem. we don't need to limit the minimum number of log file to 2. It can be 1. It is 0 means no limit. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: > I have tested that set max_audit_event_log_files to 1. That worked > as expected. > > I have researched CheckAndRotateLogFiles and associated glog codes. > Each severity of the log creates a soft link file always point to > the latest log file with the severity. The log switch process > includes creating a new log file, delete the old link file, create > new link file, so involved in the old and new log files. I think > that restrict the minimum log file number greater than 1(>=2) is to > ensure the link files to switch correctly while deleting old log > files. Yes, that makes sense to me. I think in this case we should keep the <= 1 check, to prevent this race condition. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: I have tested that set max_audit_event_log_files to 1. That worked as expected. I have researched CheckAndRotateLogFiles and associated glog codes. Each severity of the log creates a soft link file always point to the latest log file with the severity. The log switch process includes creating a new log file, delete the old link file, create new link file, so involved in the old and new log files. I think that restrict the minimum log file number greater than 1(>=2) is to ensure the link files to switch correctly while deleting old log files. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: Here's the commit that added CheckAndRotateLogFiles(). Please have a look at it and consider whether the original reason for having the <=1 check makes sense. I couldn't see why ==1 wouldn't work, but I haven't had much time to look at it. tree c3d35c8393c3b1384749d141d2b2b3fc01c677b4 parent 07efc0cb17682b2e3218d8b94389f13366f8eca7 author Martin Grund Mon Jan 5 14:52:49 2015 -0800 committer jenkins Fri Jan 9 16:51:15 2015 -0800 IMPALA-377: Log Rotation for Impala Processes This patch enables a simple form of log rotation for Impala. Using the new command line flag: -max_log_files=XX the specified number of log files is kept for each severity level and all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files will be retained and log rotation is effectively disabled. The rotation is checked with every interval of "logbufsecs" that is set to a default value of 5s. Testing is limited as logging can only be initialized once per process. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: Here's the commit that added CheckAndRotateLogFiles(). Please have a look at it and consider whether the original reason for having the <=1 check makes sense. I couldn't see why ==1 wouldn't work, but I haven't had much time to look at it. tree c3d35c8393c3b1384749d141d2b2b3fc01c677b4 parent 07efc0cb17682b2e3218d8b94389f13366f8eca7 author Martin Grund Mon Jan 5 14:52:49 2015 -0800 committer jenkins Fri Jan 9 16:51:15 2015 -0800 IMPALA-377: Log Rotation for Impala Processes This patch enables a simple form of log rotation for Impala. Using the new command line flag: -max_log_files=XX the specified number of log files is kept for each severity level and all older log files are removed. The number has to be greater than 1 to keep at least the current log file open. If set to 0, all log files will be retained and log rotation is effectively disabled. The rotation is checked with every interval of "logbufsecs" that is set to a default value of 5s. Testing is limited as logging can only be initialized once per process. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: > (3 comments) > > I have modified the code according to your opinion. The minimum > number of audit event log file should be 1 and 0 represents no > limit. Have you tested that specifying 1 actually works as expected? The other method CheckAndRotateLogFiles() still has the <=1 check, and I'm not sure why it was implemented that way. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 11: (3 comments) I have modified the code according to your opinion. The minimum number of audit event log file should be 1 and 0 represents no limit. http://gerrit.cloudera.org:8080/#/c/4971/10/be/src/common/logging.cc File be/src/common/logging.cc: Line 181: if (max_log_files < 1) return; > Why do we not allow 1 here? Is there a particular reason why it wouldn't wo Done PS10, Line 182: Check audit event log files : // Build glob pattern for input e.g. /tmp/impala_audit_event_log_1.0-* : string fname = strings::Substitute( > Maybe some of this could go on fewer lines. Done Line 188: } > nit: only one newline here. Done -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has uploaded a new patch set (#11). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 36 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/11 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 10: (3 comments) Thanks for applying the fixes. I noticed that there a <=1 where I would expect something like <1. It'd be great if you could have a look. http://gerrit.cloudera.org:8080/#/c/4971/10/be/src/common/logging.cc File be/src/common/logging.cc: Line 181: if (max_log_files <= 1) return; Why do we not allow 1 here? Is there a particular reason why it wouldn't work to retain only one logfile? PS10, Line 182: Check audit event log files : // Build glob pattern for input : // e.g. /tmp/impala_audit_event_log_1.0-* Maybe some of this could go on fewer lines. Line 188: nit: only one newline here. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 10 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 10: (1 comment) Thanks. I have modified my code. http://gerrit.cloudera.org:8080/#/c/4971/9/be/src/common/init.cc File be/src/common/init.cc: Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > You will have to adjust the wrapping of string literals manually, since git Done -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 10 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has uploaded a new patch set (#10). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 38 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/10 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 10 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/4971/9/be/src/common/init.cc File be/src/common/init.cc: Line 69: "Maximum number of audit event log files " You will have to adjust the wrapping of string literals manually, since git-clang-format does not do this automatically. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 9: Use clang-format-diff.py to correct code format. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has uploaded a new patch set (#9). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 39 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/9 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. Patch Set 8: (3 comments) Thank you for reviewing my low level format errors so many time. I will learn to use clang-format tool.:) http://gerrit.cloudera.org:8080/#/c/4971/7//COMMIT_MSG Commit Message: Line 11: event log is enabled the growing number of log files will fill up the > nit: comma Done http://gerrit.cloudera.org:8080/#/c/4971/7/be/src/common/logging.cc File be/src/common/logging.cc: Line 17: > Please move this to the block of impala includes, starting in L34. Please k Done Line 57: mutex logging_mutex; > Please remove this new line. Done -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow
Donghui Xu has uploaded a new patch set (#8). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 37 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/8 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 7: (3 comments) Only minor formatting issues left. Please see here for a description on how to use clang-format: https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala It's a good tool to become familiar with. :) http://gerrit.cloudera.org:8080/#/c/4971/7//COMMIT_MSG Commit Message: Line 11: event log is enabled,the growing number of log files will fill up the nit: comma http://gerrit.cloudera.org:8080/#/c/4971/7/be/src/common/logging.cc File be/src/common/logging.cc: Line 17: #include "service/impala-server.h" Please move this to the block of impala includes, starting in L34. Please keep that block lexicographically sorted. Line 57: Please remove this new line. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 7: Sorry. I'm not familiar with the use of git-clang-format tools.:) I have compiled and tested the newly added codes in my environment through the following cases: First, I set max_audit_event_log_files to 0 or 1, the check and rotation mechanism didn't work. Second, I set max_audit_event_log_files to 2, the check and rotation mechanism worked. I also have tested setting max_audit_event_log_files and max_log_files to different values, they all worked Correctly. The theck and rotation function LoggingSupport::DeleteOldLogs has been tested by unit test cases in be/src/util/Logging-support-test.cc. So No unit test cases need to be added. Would you tell me about your advice please? -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 7: (6 comments) Sorry. I'm not familiar with the use of git-clang-format tools.:) I have compiled and tested the newly added codes in my environment through the following cases: First, I set max_audit_event_log_files to 0 or 1, the check and rotation mechanism didn't work. Second, I set max_audit_event_log_files to 2, the check and rotation mechanism worked. I also have tested setting max_audit_event_log_files and max_log_files to different values, they all worked Correctly. The theck and rotation function LoggingSupport::DeleteOldLogs has been tested by unit test cases in be/src/util/Logging-support-test.cc. So No unit test cases need to be added. Would you tell me about your advice please? http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/init.cc File be/src/common/init.cc: Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > Done Done Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > This still needs to be done. Done Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > Please follow the indentation of the surrounding code Done PS3, Line 69: . > nit: space missing Done http://gerrit.cloudera.org:8080/#/c/4971/6/be/src/service/impala-server.h File be/src/service/impala-server.h: Line 271: /// Get the prefix of audit event log filename. > nit: dot. Done Line 272: static std::string GetAuditEventLogFilePrefix(); > Please remove trailing whitespace. Also check other files. Done -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has uploaded a new patch set (#7). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled,the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 38 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/7 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 6: (4 comments) There are still some minor formatting issues left. Have you actually run git-clang-format on your change? That should fix them automatically. Also, have you thought about how we can test this new behaviour, both to make sure it works as intended and does not break other features? http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/init.cc File be/src/common/init.cc: Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > Please follow the indentation of the surrounding code This still needs to be done. http://gerrit.cloudera.org:8080/#/c/4971/6/be/src/common/logging.cc File be/src/common/logging.cc: Line 186: ImpalaServer::GetAuditEventLogFilePrefix()); much better :) http://gerrit.cloudera.org:8080/#/c/4971/6/be/src/service/impala-server.h File be/src/service/impala-server.h: Line 271: /// Get the prefix of audit event log filename nit: dot. Line 272: static std::string GetAuditEventLogFilePrefix(); Please remove trailing whitespace. Also check other files. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 6: I have added static method GetAuditEventLogFilePrefix in class ImpalaServer. Thus reduplicate definition of AUDIT_EVENT_LOG_FILE_PREFIX in logging.cc can be deleted, function CheckAndRotateAuditEventLogFiles can use GetAuditEventLogFilePrefix to get AUDIT_EVENT_LOG_FILE_PREFIX defined in impala-server.cc -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has uploaded a new patch set (#6). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled,the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/service/impala-server.cc M be/src/service/impala-server.h 5 files changed, 38 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/6 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 5: (9 comments) logging.cc Line 58: I can't use it by include impala-server.h, because this variable is const type, while impala-server.h only contains methods and member variables. It does not include this variable. http://gerrit.cloudera.org:8080/#/c/4971/1//COMMIT_MSG Commit Message: PS1, Line 2: davidxdh > nit: you might want to configure git to include your name here Done PS1, Line 10: dit e > fill up Done http://gerrit.cloudera.org:8080/#/c/4971/1/be/src/common/init.cc File be/src/common/init.cc: PS1, Line 68: 0, > I don't think we should set a default here, because audit information might Done Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > Use spaces instead of tabs, wrap lines at 90 characters. You can use git-cl Done http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/init.cc File be/src/common/init.cc: Line 69: "to retain. The most recent audit event log files are retained. If set to 0, " > Please follow the indentation of the surrounding code Done http://gerrit.cloudera.org:8080/#/c/4971/1/be/src/common/logging.cc File be/src/common/logging.cc: Line 58: // Prefix of audit event log filename. The version number is > Where in the code do we make sure that event log files are always written w This variable is referenced from AUDIT_EVENT_LOG_FILE_PREFIX at line 195 in be/src/service/impala-server.cc. I can't use it by include impala-server.h, because this variable is const while impala-server.h only contains methods and member variables. http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/logging.cc File be/src/common/logging.cc: Line 185: if (max_log_files <= 1) return; > This line looks wrong, please update comment. Done http://gerrit.cloudera.org:8080/#/c/4971/4/be/src/common/logging.cc File be/src/common/logging.cc: PS4, Line 61: AUDIT_EVENT_LOG_FILE_PREFIX I can't use it by include impala-server.h, because this variable is const type, while impala-server.h only contains methods and member variables. It does not include this variable. http://gerrit.cloudera.org:8080/#/c/4971/4/be/src/common/logging.h File be/src/common/logging.h: PS4, Line 85: /// Helper function that checks for the number of logfiles in the log directory and > please wrap this paragraph and the one below at 90 characters. Done -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has uploaded a new patch set (#5). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled,the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h 3 files changed, 34 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/5 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 4: (2 comments) Please address the questions in my previous comments and mark them as "Done" once they are addressed. http://gerrit.cloudera.org:8080/#/c/4971/1/be/src/common/logging.cc File be/src/common/logging.cc: Line 58: // Prefix of audit event log filename. The version number is > Please address this comment. This variable is defined in impala-server.cc:195, too. Can you reuse that definition here instead of duplicating it? http://gerrit.cloudera.org:8080/#/c/4971/4/be/src/common/logging.h File be/src/common/logging.h: PS4, Line 85: /// Helper function that checks for the number of logfiles in the log directory and removes please wrap this paragraph and the one below at 90 characters. -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has uploaded a new patch set (#4). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled,the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h 3 files changed, 31 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/4 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 3: (5 comments) Please mark comments as done and post replies here when you're done. http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/init.cc File be/src/common/init.cc: Line 69: "to retain.The most recent audit event log files are retained. If set to 0, " Please follow the indentation of the surrounding code PS3, Line 69: .T nit: space missing http://gerrit.cloudera.org:8080/#/c/4971/1/be/src/common/logging.cc File be/src/common/logging.cc: Line 58: const string AUDIT_EVENT_LOG_FILE_PREFIX = "impala_audit_event_log_1.0-"; > Where in the code do we make sure that event log files are always written w Please address this comment. http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/logging.cc File be/src/common/logging.cc: Line 185: // e.g. /tmp/impalad.*.INFO.* This line looks wrong, please update comment. http://gerrit.cloudera.org:8080/#/c/4971/3/be/src/common/logging.h File be/src/common/logging.h: Line 90: /// and removes line wrapping -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Donghui Xu has uploaded a new patch set (#3). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled,the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h 3 files changed, 29 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/3 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Donghui Xu Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
davy...@163.com has uploaded a new patch set (#2). Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled,the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h 3 files changed, 28 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/2 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: davy...@163.com Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
Lars Volker has posted comments on this change. Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. Patch Set 1: (6 comments) Thank you for working on this. Please make sure your change follows the general style guidelines of impala, especially with regards to whitespace. http://gerrit.cloudera.org:8080/#/c/4971/1//COMMIT_MSG Commit Message: Please re-format the commit message to wrap lines at 80 characters PS1, Line 2: davidxdh nit: you might want to configure git to include your name here PS1, Line 10: spill fill up http://gerrit.cloudera.org:8080/#/c/4971/1/be/src/common/init.cc File be/src/common/init.cc: PS1, Line 68: 100 I don't think we should set a default here, because audit information might be sensitive to retention policies we don't control. I'll asked this as a question in the Jira. Line 69: "level. The most recent audit event log files are retained. If set to 0, all audit event log files are " Use spaces instead of tabs, wrap lines at 90 characters. You can use git-clang-format to format whitespace. This page contains more information: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868536 http://gerrit.cloudera.org:8080/#/c/4971/1/be/src/common/logging.cc File be/src/common/logging.cc: Line 58: const string AUDIT_EVENT_LOG_FILE_PREFIX = "impala_audit_event_log_1.0-"; Where in the code do we make sure that event log files are always written with this name prefix? Can you take that information and re-use it here? -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: davy...@163.com Gerrit-Reviewer: Lars Volker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4431: Add audit event log control mechanism to prevent disk overflow.
davy...@163.com has uploaded a new change for review. http://gerrit.cloudera.org:8080/4971 Change subject: IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. .. IMPALA-4431: Add audit event log control mechanism to prevent disk overflow. There is no limit to the number of audit event log files. When audit event log is enabled, the growing number of log files will spill the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 --- M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h 3 files changed, 27 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/4971/1 -- To view, visit http://gerrit.cloudera.org:8080/4971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: davy...@163.com