[ https://issues.apache.org/jira/browse/HDDS-2042?focusedWorklogId=302183&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302183 ]
ASF GitHub Bot logged work on HDDS-2042: ---------------------------------------- Author: ASF GitHub Bot Created on: 27/Aug/19 16:36 Start Date: 27/Aug/19 16:36 Worklog Time Spent: 10m Work Description: adoroszlai commented on pull request #1357: HDDS-2042. Avoid log on console with Ozone shell URL: https://github.com/apache/hadoop/pull/1357 ## What changes were proposed in this pull request? Sample Docker Compose-based clusters are currently [configured](https://github.com/apache/hadoop/blob/3329257d99d2808e66ae6c2fe87a9c4f8877026f/hadoop-ozone/dist/src/main/compose/ozone/docker-config#L32-L33) to log almost everything to the console. This is useful because logs can be accessed via `docker logs` or `docker-compose logs`. Further, [default logs settings](https://github.com/apache/hadoop/blob/3329257d99d2808e66ae6c2fe87a9c4f8877026f/hadoop-ozone/dist/src/main/conf/log4j.properties#L137-L138) for non-Docker Compose clusters also direct INFO and higher level messages to the console. However, it does not work well with interactive clients, eg. `ozone sh`, since regular console output and log messages are mixed. Previously there [were](https://issues.apache.org/jira/browse/HDDS-1489) [attempts](https://issues.apache.org/jira/browse/HDDS-465) to work around this by setting specific class log levels to `WARN` or `ERROR`. This approach has two problems: 1. it applies to Ozone server processes, too 2. new INFO or higher level messages may be introduced any time (eg. MetricsSystem startup) This pull request proposes to collect logs for `ozone` subcommands `sh` and `freon` to files instead of the console. It eliminates the need for per-class log level settings (though current ones are kept). https://issues.apache.org/jira/browse/HDDS-2042 ## How was this patch tested? Tested commands in sample docker cluster. Output is clean (except pesky warning (omitted here) about illegal access for key operations). ``` $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozones3 $ docker-compose exec scm bash bash-4.2$ ozone sh volume create vol1 bash-4.2$ ozone sh bucket create vol1/buck1 bash-4.2$ ozone sh key put vol1/buck1/key1 /etc/passwd ``` Verified that log file contains the messages previously sent to console: ``` bash-4.2$ tail /var/log/hadoop/ozone-shell.log 2019-08-27 14:45:02,695 [main] INFO RpcClient:293 - Creating Volume: vol1, with hadoop as owner. 2019-08-27 14:45:13,099 [main] INFO RpcClient:432 - Creating Bucket: vol1/buck1, with Versioning false and Storage Type set to DISK and Encryption set to false 2019-08-27 14:45:24,011 [main] INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties 2019-08-27 14:45:24,160 [main] INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). 2019-08-27 14:45:24,160 [main] INFO MetricsSystemImpl:191 - XceiverClientMetrics metrics system started ``` Verbose mode still works: ``` bash-4.2$ ozone sh --verbose key put vol1/buck1/key2 /etc/passwd Volume Name : vol1 Bucket Name : buck1 Key Name : key2 File Hash : b01f053617ddfeb782a3e757d9c08912 ``` Freon: ``` bash-4.2$ ozone freon rk --numOfVolumes=1 --numOfBuckets=1 --numOfKeys=20 --numOfThreads=1 100.00% |?????????????????????????????????????????????????????????????????????????????????????????????????????| 20/20 Time: 0:00:03 *************************************************** Status: Success Git Base Revision: e97acb3bd8f3befd27418996fa5d4b50bf2e17bf Number of Volumes created: 1 Number of Buckets created: 1 Number of Keys added: 20 Ratis replication factor: ONE Ratis replication type: STAND_ALONE Average Time spent in volume creation: 00:00:00,080 Average Time spent in bucket creation: 00:00:00,013 Average Time spent in key creation: 00:00:00,234 Average Time spent in key write: 00:00:02,310 Total bytes written: 204800 Total Execution time: 00:00:06,284 *************************************************** ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 302183) Remaining Estimate: 0h Time Spent: 10m > Avoid log on console with Ozone shell > ------------------------------------- > > Key: HDDS-2042 > URL: https://issues.apache.org/jira/browse/HDDS-2042 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: docker > Reporter: Doroszlai, Attila > Assignee: Doroszlai, Attila > Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HDDS-1489 fixed several sample docker compose configs to avoid unnecessary > messages on console when running eg. {{ozone sh key put}}. The goal of this > task is to fix the remaining ones. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org