[jira] [Commented] (HIVE-13790) log4j2 syslog appender not taking "LoggerFields" and "KeyValuePair" options

2016-06-07 Thread Alexandre Linte (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318630#comment-15318630
 ] 

Alexandre Linte commented on HIVE-13790:


Hi [~prasanth_j],

I tried to set the keys in MDC without a better result. Hive seems to ignore 
the MDC configuration.

> log4j2 syslog appender not taking "LoggerFields" and "KeyValuePair" options
> ---
>
> Key: HIVE-13790
> URL: https://issues.apache.org/jira/browse/HIVE-13790
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore
>Affects Versions: 2.0.0
> Environment: Hive 2.0.0, Hadoop 2.7.2, Spark 1.6.1, HBase 1.1.2
>Reporter: Alexandre Linte
>
> I'm trying to use the Syslog appender with log4j2 in Hive 2.0.0. The syslog 
> appender is configured on my hiveserver2 and my metastore.
> With a simple configuration, the logs are well written in the logfile with a 
> generic pattern layout:
> {noformat}
> May 19 10:12:16 myhiveserver2.fr Starting HiveServer2
> May 19 10:12:18 myhiveserver2.fr Connected to metastore.
> May 19 10:12:20 myhiveserver2.fr Service: CLIService is inited.
> May 19 10:12:20 myhiveserver2.fr Service: ThriftBinaryCLIService is inited.
> {noformat}
> I tried to customize this pattern layout by adding the loggerFields parameter 
> in my hive-log4j2.properties. At the end, the configuration file is:
> {noformat}
> status = TRACE
> name = HiveLog4j2
> packages = org.apache.hadoop.hive.ql.log
> property.hive.log.level = INFO
> property.hive.root.logger = SYSLOG
> property.hive.query.id = hadoop
> property.hive.log.dir = /var/log/bigdata
> property.hive.log.file = bigdata.log
> appenders = console, SYSLOG
> appender.console.type = Console
> appender.console.name = console
> appender.console.target = SYSTEM_ERR
> appender.console.layout.type = PatternLayout
> appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n
> appender.SYSLOG.type = Syslog
> appender.SYSLOG.name = SYSLOG
> appender.SYSLOG.host = 127.0.0.1
> appender.SYSLOG.port = 514
> appender.SYSLOG.protocol = UDP
> appender.SYSLOG.facility = LOCAL1
> appender.SYSLOG.layout.type = loggerFields
> appender.SYSLOG.layout.sdId = test
> appender.SYSLOG.layout.enterpriseId = 18060
> appender.SYSLOG.layout.pairs.type = KeyValuePair
> appender.SYSLOG.layout.pairs.key = service
> appender.SYSLOG.layout.pairs.value = hiveserver2
> appender.SYSLOG.layout.pairs.key = loglevel
> appender.SYSLOG.layout.pairs.value = %p
> appender.SYSLOG.layout.pairs.key = message
> appender.SYSLOG.layout.pairs.value = %c%m%n
> loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX
> logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
> logger.NIOServerCnxn.level = WARN
> logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
> logger.ClientCnxnSocketNIO.level = WARN
> logger.DataNucleus.name = DataNucleus
> logger.DataNucleus.level = ERROR
> logger.Datastore.name = Datastore
> logger.Datastore.level = ERROR
> logger.JPOX.name = JPOX
> logger.JPOX.level = ERROR
> rootLogger.level = ${sys:hive.log.level}
> rootLogger.appenderRefs = root
> rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
> {noformat}
> Unfortunately, the logs are still written in a generic pattern layout. The 
> KeyValuePairs are not used. The log4j logs are:
> {noformat}
> 2016-05-19 10:36:14,866 main DEBUG Initializing configuration 
> org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@5433a329
> 2016-05-19 10:36:16,575 main DEBUG Took 1.706004 seconds to load 3 plugins 
> from package org.apache.hadoop.hive.ql.log
> 2016-05-19 10:36:16,575 main DEBUG PluginManager 'Core' found 80 plugins
> 2016-05-19 10:36:16,576 main DEBUG PluginManager 'Level' found 0 plugins
> 2016-05-19 10:36:16,578 main DEBUG Building Plugin[name=property, 
> class=org.apache.logging.log4j.core.config.Property]. Searching for builder 
> factory method...
> 2016-05-19 10:36:16,583 main DEBUG No builder factory method found in class 
> org.apache.logging.log4j.core.config.Property. Going to try finding a factory 
> method instead.
> 2016-05-19 10:36:16,583 main DEBUG Still building Plugin[name=property, 
> class=org.apache.logging.log4j.core.config.Property]. Searching for factory 
> method...
> 2016-05-19 10:36:16,584 main DEBUG Found factory method [createProperty]: 
> public static org.apache.logging.log4j.core.config.Property 
> org.apache.logging.log4j.core.config.Property.createProperty(java.lang.String,java.lang.String).
> 2016-05-19 10:36:16,611 main DEBUG TypeConverterRegistry initializing.
> 2016-05-19 10:36:16,611 main DEBUG PluginManager 'TypeConverter' found 21 
> plugins
> 2016-05-19 10:36:16,636 main DEBUG Calling createProperty on class 
> org.apache.logging.log4j.core.config.Property for

[jira] [Commented] (HIVE-13790) log4j2 syslog appender not taking "LoggerFields" and "KeyValuePair" options

2016-06-06 Thread Prasanth Jayachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317218#comment-15317218
 ] 

Prasanth Jayachandran commented on HIVE-13790:
--

Shouldn't the loggerFields need the keys to be set in MDC? AFAIK hive does not 
set any keys to MDC.

> log4j2 syslog appender not taking "LoggerFields" and "KeyValuePair" options
> ---
>
> Key: HIVE-13790
> URL: https://issues.apache.org/jira/browse/HIVE-13790
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore
>Affects Versions: 2.0.0
> Environment: Hive 2.0.0, Hadoop 2.7.2, Spark 1.6.1, HBase 1.1.2
>Reporter: Alexandre Linte
>
> I'm trying to use the Syslog appender with log4j2 in Hive 2.0.0. The syslog 
> appender is configured on my hiveserver2 and my metastore.
> With a simple configuration, the logs are well written in the logfile with a 
> generic pattern layout:
> {noformat}
> May 19 10:12:16 myhiveserver2.fr Starting HiveServer2
> May 19 10:12:18 myhiveserver2.fr Connected to metastore.
> May 19 10:12:20 myhiveserver2.fr Service: CLIService is inited.
> May 19 10:12:20 myhiveserver2.fr Service: ThriftBinaryCLIService is inited.
> {noformat}
> I tried to customize this pattern layout by adding the loggerFields parameter 
> in my hive-log4j2.properties. At the end, the configuration file is:
> {noformat}
> status = TRACE
> name = HiveLog4j2
> packages = org.apache.hadoop.hive.ql.log
> property.hive.log.level = INFO
> property.hive.root.logger = SYSLOG
> property.hive.query.id = hadoop
> property.hive.log.dir = /var/log/bigdata
> property.hive.log.file = bigdata.log
> appenders = console, SYSLOG
> appender.console.type = Console
> appender.console.name = console
> appender.console.target = SYSTEM_ERR
> appender.console.layout.type = PatternLayout
> appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n
> appender.SYSLOG.type = Syslog
> appender.SYSLOG.name = SYSLOG
> appender.SYSLOG.host = 127.0.0.1
> appender.SYSLOG.port = 514
> appender.SYSLOG.protocol = UDP
> appender.SYSLOG.facility = LOCAL1
> appender.SYSLOG.layout.type = loggerFields
> appender.SYSLOG.layout.sdId = test
> appender.SYSLOG.layout.enterpriseId = 18060
> appender.SYSLOG.layout.pairs.type = KeyValuePair
> appender.SYSLOG.layout.pairs.key = service
> appender.SYSLOG.layout.pairs.value = hiveserver2
> appender.SYSLOG.layout.pairs.key = loglevel
> appender.SYSLOG.layout.pairs.value = %p
> appender.SYSLOG.layout.pairs.key = message
> appender.SYSLOG.layout.pairs.value = %c%m%n
> loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX
> logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
> logger.NIOServerCnxn.level = WARN
> logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
> logger.ClientCnxnSocketNIO.level = WARN
> logger.DataNucleus.name = DataNucleus
> logger.DataNucleus.level = ERROR
> logger.Datastore.name = Datastore
> logger.Datastore.level = ERROR
> logger.JPOX.name = JPOX
> logger.JPOX.level = ERROR
> rootLogger.level = ${sys:hive.log.level}
> rootLogger.appenderRefs = root
> rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
> {noformat}
> Unfortunately, the logs are still written in a generic pattern layout. The 
> KeyValuePairs are not used. The log4j logs are:
> {noformat}
> 2016-05-19 10:36:14,866 main DEBUG Initializing configuration 
> org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@5433a329
> 2016-05-19 10:36:16,575 main DEBUG Took 1.706004 seconds to load 3 plugins 
> from package org.apache.hadoop.hive.ql.log
> 2016-05-19 10:36:16,575 main DEBUG PluginManager 'Core' found 80 plugins
> 2016-05-19 10:36:16,576 main DEBUG PluginManager 'Level' found 0 plugins
> 2016-05-19 10:36:16,578 main DEBUG Building Plugin[name=property, 
> class=org.apache.logging.log4j.core.config.Property]. Searching for builder 
> factory method...
> 2016-05-19 10:36:16,583 main DEBUG No builder factory method found in class 
> org.apache.logging.log4j.core.config.Property. Going to try finding a factory 
> method instead.
> 2016-05-19 10:36:16,583 main DEBUG Still building Plugin[name=property, 
> class=org.apache.logging.log4j.core.config.Property]. Searching for factory 
> method...
> 2016-05-19 10:36:16,584 main DEBUG Found factory method [createProperty]: 
> public static org.apache.logging.log4j.core.config.Property 
> org.apache.logging.log4j.core.config.Property.createProperty(java.lang.String,java.lang.String).
> 2016-05-19 10:36:16,611 main DEBUG TypeConverterRegistry initializing.
> 2016-05-19 10:36:16,611 main DEBUG PluginManager 'TypeConverter' found 21 
> plugins
> 2016-05-19 10:36:16,636 main DEBUG Calling createProperty on class 
> org.apache.logging.log4j.core.config.Property for element

[jira] [Commented] (HIVE-13790) log4j2 syslog appender not taking "LoggerFields" and "KeyValuePair" options

2016-06-06 Thread Alexandre Linte (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316429#comment-15316429
 ] 

Alexandre Linte commented on HIVE-13790:


Nothing new about this JIRA? Nobody tried to use a SYSLOG appender with Hive 
2.0.0?

> log4j2 syslog appender not taking "LoggerFields" and "KeyValuePair" options
> ---
>
> Key: HIVE-13790
> URL: https://issues.apache.org/jira/browse/HIVE-13790
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore
>Affects Versions: 2.0.0
> Environment: Hive 2.0.0, Hadoop 2.7.2, Spark 1.6.1, HBase 1.1.2
>Reporter: Alexandre Linte
>
> I'm trying to use the Syslog appender with log4j2 in Hive 2.0.0. The syslog 
> appender is configured on my hiveserver2 and my metastore.
> With a simple configuration, the logs are well written in the logfile with a 
> generic pattern layout:
> {noformat}
> May 19 10:12:16 myhiveserver2.fr Starting HiveServer2
> May 19 10:12:18 myhiveserver2.fr Connected to metastore.
> May 19 10:12:20 myhiveserver2.fr Service: CLIService is inited.
> May 19 10:12:20 myhiveserver2.fr Service: ThriftBinaryCLIService is inited.
> {noformat}
> I tried to customize this pattern layout by adding the loggerFields parameter 
> in my hive-log4j2.properties. At the end, the configuration file is:
> {noformat}
> status = TRACE
> name = HiveLog4j2
> packages = org.apache.hadoop.hive.ql.log
> property.hive.log.level = INFO
> property.hive.root.logger = SYSLOG
> property.hive.query.id = hadoop
> property.hive.log.dir = /var/log/bigdata
> property.hive.log.file = bigdata.log
> appenders = console, SYSLOG
> appender.console.type = Console
> appender.console.name = console
> appender.console.target = SYSTEM_ERR
> appender.console.layout.type = PatternLayout
> appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n
> appender.SYSLOG.type = Syslog
> appender.SYSLOG.name = SYSLOG
> appender.SYSLOG.host = 127.0.0.1
> appender.SYSLOG.port = 514
> appender.SYSLOG.protocol = UDP
> appender.SYSLOG.facility = LOCAL1
> appender.SYSLOG.layout.type = loggerFields
> appender.SYSLOG.layout.sdId = test
> appender.SYSLOG.layout.enterpriseId = 18060
> appender.SYSLOG.layout.pairs.type = KeyValuePair
> appender.SYSLOG.layout.pairs.key = service
> appender.SYSLOG.layout.pairs.value = hiveserver2
> appender.SYSLOG.layout.pairs.key = loglevel
> appender.SYSLOG.layout.pairs.value = %p
> appender.SYSLOG.layout.pairs.key = message
> appender.SYSLOG.layout.pairs.value = %c%m%n
> loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX
> logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
> logger.NIOServerCnxn.level = WARN
> logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
> logger.ClientCnxnSocketNIO.level = WARN
> logger.DataNucleus.name = DataNucleus
> logger.DataNucleus.level = ERROR
> logger.Datastore.name = Datastore
> logger.Datastore.level = ERROR
> logger.JPOX.name = JPOX
> logger.JPOX.level = ERROR
> rootLogger.level = ${sys:hive.log.level}
> rootLogger.appenderRefs = root
> rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
> {noformat}
> Unfortunately, the logs are still written in a generic pattern layout. The 
> KeyValuePairs are not used. The log4j logs are:
> {noformat}
> 2016-05-19 10:36:14,866 main DEBUG Initializing configuration 
> org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@5433a329
> 2016-05-19 10:36:16,575 main DEBUG Took 1.706004 seconds to load 3 plugins 
> from package org.apache.hadoop.hive.ql.log
> 2016-05-19 10:36:16,575 main DEBUG PluginManager 'Core' found 80 plugins
> 2016-05-19 10:36:16,576 main DEBUG PluginManager 'Level' found 0 plugins
> 2016-05-19 10:36:16,578 main DEBUG Building Plugin[name=property, 
> class=org.apache.logging.log4j.core.config.Property]. Searching for builder 
> factory method...
> 2016-05-19 10:36:16,583 main DEBUG No builder factory method found in class 
> org.apache.logging.log4j.core.config.Property. Going to try finding a factory 
> method instead.
> 2016-05-19 10:36:16,583 main DEBUG Still building Plugin[name=property, 
> class=org.apache.logging.log4j.core.config.Property]. Searching for factory 
> method...
> 2016-05-19 10:36:16,584 main DEBUG Found factory method [createProperty]: 
> public static org.apache.logging.log4j.core.config.Property 
> org.apache.logging.log4j.core.config.Property.createProperty(java.lang.String,java.lang.String).
> 2016-05-19 10:36:16,611 main DEBUG TypeConverterRegistry initializing.
> 2016-05-19 10:36:16,611 main DEBUG PluginManager 'TypeConverter' found 21 
> plugins
> 2016-05-19 10:36:16,636 main DEBUG Calling createProperty on class 
> org.apache.logging.log4j.core.config.Property for element Property with 
>