Miklos Szurap created HIVE-21455:
------------------------------------
Summary: Too verbose logging in AvroGenericRecordReader
Key: HIVE-21455
URL: https://issues.apache.org/jira/browse/HIVE-21455
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Miklos Szurap
{{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
verbose, likely we don't need to log that on INFO level.
For example a table:
{noformat}
create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
{noformat}
and querying it with a select star - with 3 datafiles HiveServer2 logs the
following:
{noformat}
2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
[HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
2019-03-15 09:18:35,999 INFO
org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
[HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
{"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
2019-03-15 09:18:36,004 INFO
org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
[HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
{"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
2019-03-15 09:18:36,010 INFO
org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
[HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
{"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
{noformat}
This has a huge performance and storage penalty on a table with big schema and
thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
