Hi,
I am pretty perplexed at this time, trying to get hive to store the stats
after a query runs.
hive.stats.autogather is set to *true* by default
This is in our hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/aux-jars/mysql-connector-java-5.1.34.jar</value>
</property>
<property>
<name>hive.stats.dbclass</name>
<value>jdbc:mysql</value>
</property>
<property>
<name>hive.stats.dbconnectionstring</name>
<value>jdbc:mysql://*<hostname>*:3306/*<database>*
?useUnicode=true&characterEncoding=UTF-8&user=*<username>*
&password=*<password>*</value>
</property>
<property>
<name>hive.stats.jdbcdriver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
After a query runs, I see this error in the hive shell
*[Error 30017]: Skipping stats aggregation by error
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 30000]:
StatsPublisher cannot be obtained. There was a error to retrieve the
StatsPublisher, and retrying might help. If you dont want the query to fail
because accurate statistics could not be collected, set
hive.stats.reliable=false*
I see this in the hive logs
*2017-09-08T17:11:41,625 ERROR [Thread-140]: ERROR
org.apache.hadoop.hive.ql.stats.StatsFactory
(StatsFactory.java:initialize(76)) - jdbc:mysql Publisher/Aggregator
classes cannot be loaded.*
*2017-09-08T17:11:51,405 ERROR [779c2413-28ee-4fb0-8bda-0dd8f8bebe04 main]:
ERROR org.apache.hadoop.hive.ql.stats.StatsFactory
(StatsFactory.java:initialize(76)) - jdbc:mysql Publisher/Aggregator
classes cannot be loaded.*
*2017-09-08T17:11:51,406 ERROR [779c2413-28ee-4fb0-8bda-0dd8f8bebe04 main]:
ERROR org.apache.hadoop.hive.ql.exec.Task
(SessionState.java:printError(1038)) - [Error 30017]: Skipping stats
aggregation by error org.apache.hadoop.hive.ql.metadata.HiveException:
[Error 30000]: StatsPublisher cannot be obtained. There was a error to
retrieve the StatsPublisher, and retrying might help. If you dont want the
query to fail because accurate statistics could not be collected, set
hive.stats.reliable=false*
I looked at the documentation on
https://cwiki.apache.org/confluence/display/Hive/StatsDev.
It says that mysql and hbase stats publishers are already implemented in
the hive code and they implement the *IStatsPublisher *and *IStatsAggregator
*interface. I am looking at the hive codebase and I don't see those
interfaces defined in any classes. However, I found the *StatsPublisher*
and *StatsAggregator *interfaces. When looking at which classes have
implemented the *StatsPublisher *interface, I found only 2 usages.
*public class FSStatsPublisher implements StatsPublisher*
*public static class TFSOStatsPublisher implements StatsPublisher*
The second one is a test class and no one else extends the first one.
So, I am not sure, if the code is missing or I am supposed to implement
something or the configuration is wrong or missing something.
Also, on this page https://cwiki.apache.org/confluence/display/Hive/StatsDev,
I see this
The user can also specify the implementation to be used for the storage of
temporary statistics setting the variable hive.stats.dbclass. For example,
to set HBase as the implementation of *temporary* statistics storage (the
default is jdbc:derby or fs, depending on the Hive version) the user should
issue the following command:
What does *temporary* mean in this context?
What is more confusing is that when I run
*DESC EXTENDED <tablename>;*
I see this
*parameters:{transient_lastDdlTime=1504891564, totalSize=391013,
COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=6}*
Then I ran, *ANALYZE TABLE <tablename> COMPUTE STATISTICS FOR COLUMNS *and
ran* DESC EXTENDED* again.
*parameters:{transient_lastDdlTime=1504891564, totalSize=391013,
COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"id":"true","col2":"true","*
*col3**":"true","**col4**":"true"}}, numFiles=6}*
Does this mean some stats are stored?
Any help is appreciated.
Thanx.
--
Regards,
Premal Shah.
