hddong commented on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823742124
@HyukjinKwon : I think there some different with `-S`. 1. `-S` cannot change log to `INFO`(or other level) 2. with log4j we can change `spark-sql` init log level. 3. And they can worrk together. With default: ``` > spark-sql 21/04/21 10:52:50 WARN Utils: Your hostname, hongddMac.local resolves to a loopback address: 127.0.0.1; using 172.20.30.177 instead (on interface en0) 21/04/21 10:52:50 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 21/04/21 10:52:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/04/21 10:52:53 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 21/04/21 10:52:53 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 21/04/21 10:52:54 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0 21/04/21 10:52:54 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore hongdd@127.0.0.1 Spark master: local[*], Application Id: local-1618973572072 spark-sql> show databases; default Time taken: 1.719 seconds, Fetched 1 row(s) ``` And `-S` not out line `Time taken: 1.719 seconds, Fetched 1 row(s)` With log4j changed `INFO`: ``` >bin/spark-sql 21/04/21 10:56:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/04/21 10:56:15 INFO HiveConf: Found configuration file null 21/04/21 10:56:15 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir. 21/04/21 10:56:15 INFO SharedState: Warehouse path is 'file:/user/hive/warehouse'. 21/04/21 10:56:15 INFO SessionState: Created HDFS directory: /tmp/hive/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd 21/04/21 10:56:15 INFO SessionState: Created local directory: /var/folders/xl/jv1jvw6s6hv42ht1ty6gnk100000gn/T/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd 21/04/21 10:56:15 INFO SessionState: Created HDFS directory: /tmp/hive/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd/_tmp_space.db 21/04/21 10:56:15 INFO SparkContext: Running Spark version 3.2.0-SNAPSHOT 21/04/21 10:56:15 INFO ResourceUtils: ============================================================== 21/04/21 10:56:15 INFO ResourceUtils: No custom resources configured for spark.driver. 21/04/21 10:56:15 INFO ResourceUtils: ============================================================== 21/04/21 10:56:15 INFO SparkContext: Submitted application: SparkSQL::172.20.30.177 .... spark-sql> show databases; 21/04/21 10:56:52 INFO HiveMetaStore: 0: get_databases: * 21/04/21 10:56:52 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_databases: * 21/04/21 10:56:53 INFO CodeGenerator: Code generated in 129.111772 ms 21/04/21 10:56:53 INFO CodeGenerator: Code generated in 5.292995 ms default Time taken: 1.735 seconds, Fetched 1 row(s) 21/04/21 10:56:53 INFO SparkSQLCLIDriver: Time taken: 1.735 seconds, Fetched 1 row(s) ``` And `-S` effect when you query but not init. ``` 1/04/21 10:57:40 INFO HiveMetaStore: 0: get_all_functions 21/04/21 10:57:40 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_all_functions 21/04/21 10:57:40 INFO HiveMetaStore: 0: get_database: default 21/04/21 10:57:40 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_database: default 21/04/21 10:57:40 INFO SparkSQLCLIDriver: Spark master: local[*], Application Id: local-1618973857876 spark-sql> show databases; default 21/04/21 10:57:46 INFO SparkSQLCLIDriver: Time taken: 1.698 seconds, Fetched 1 row(s) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
