Hi, I am writing a application which needs the metastore about hive tables. I have used webhcat to get the information about tables and process them. But a simple request takes over eight seconds to respond on localhost. Why is this so slow, and how can I fix it or is there other way I can extract the metadata in C?
$ time curl -s 'http://localhost:50111/templeton/v1/ddl/database/default/table/haha?user.name=ctdean <http://localhost:50111/templeton/v1/ddl/database/default/table/haha?user.name=ctdean>' {"columns": [{"name":"id","type":"int"}], "database":"default", "table":"haha"} real 0m8.400s user 0m0.053s sys 0m0.019s it seems to run a hcat.py, and it create a bunch of things then clear them, it takes very long time, does anyone have some ideas about it?? Any suggestions will be very appreciated! $hcat.py -e "use default; desc haha; " SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/04/21 16:38:13 INFO conf.HiveConf: Found configuration file file:/usr/local/hive/conf/hive-site.xml 18/04/21 16:38:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/04/21 16:38:16 INFO session.SessionState: Created HDFS directory: /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668 18/04/21 16:38:16 INFO session.SessionState: Created local directory: /tmp/hive/java/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668 18/04/21 16:38:16 INFO session.SessionState: Created HDFS directory: /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668/_tmp_space.db 18/04/21 16:38:16 INFO ql.Driver: Compiling command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62): use default 18/04/21 16:38:17 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 18/04/21 16:38:17 INFO metastore.ObjectStore: ObjectStore, initialize called 18/04/21 16:38:18 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 18/04/21 16:38:18 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 18/04/21 16:38:18 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 18/04/21 16:38:20 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 18/04/21 16:38:20 INFO metastore.ObjectStore: Initialized ObjectStore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: Added admin role in metastore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: Added public role in metastore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_all_functions 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_all_functions 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_database: default 18/04/21 16:38:20 INFO ql.Driver: Semantic Analysis Completed 18/04/21 16:38:20 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) 18/04/21 16:38:20 INFO ql.Driver: Completed compiling command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62); Time taken: 3.936 seconds 18/04/21 16:38:20 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 18/04/21 16:38:20 INFO ql.Driver: Executing command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62): use default 18/04/21 16:38:20 INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=05096382-f9b6-4dae-aee2-dfa6750c0668, clientType=HIVECLI] 18/04/21 16:38:20 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. 18/04/21 16:38:20 INFO hive.metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=Cleaning up thread local RawStore... 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore 18/04/21 16:38:20 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_database: default 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 18/04/21 16:38:20 INFO metastore.ObjectStore: ObjectStore, initialize called 18/04/21 16:38:20 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 18/04/21 16:38:20 INFO metastore.ObjectStore: Initialized ObjectStore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_database: default 18/04/21 16:38:20 INFO ql.Driver: Completed executing command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62); Time taken: 0.202 seconds OK
