Thank you very much for your reply, I am wondering whether I use the webhcat
rightly, I don’t think it is normal to create all the directories and objects
to get a table describ and take 8 seconds. The webhcat should not be so slow,
Or it is because I forget to start some server which can respond immediately?
> 在 2018年4月23日，下午3:06，Peter Vary <pv...@cloudera.com> 写道：
>
> Hi,
>
> Alexander Kolbasov has a project which might interest you (keeping in mind,
> that this is not production ready - more like a proof of concept):
> https://github.com/akolb1/gometastore/blob/master/hmstool/doc/hmstool.md
>
> Also you can use HMS thrift API directly to access the MetaStore, or if you
> can/want write java code, you can use HiveMetastoreClient class to do it in
> java.
>
> I am not sure about the performance gains compared to HCat, but currently
> there are no faster interfaces for HMS that I know of.
>
> Regards,
> Peter
>
>
> 侯宗田 <zongtian...@icloud.com> ezt írta (időpont: 2018. ápr. 23., Hét 2:40):
>
>> Can anyone give me some suggestions? I have been stuck in this problem for
>> several days. Need help!!
>>> 在 2018年4月22日，下午9:38，侯宗田 <zongtian...@icloud.com> 写道：
>>>
>>>
>>> Hi,
>>>
>>> I am writing a application which needs the metastore about hive tables.
>> I have used webhcat to get the information about tables and process them.
>> But a simple request takes over eight seconds to respond on localhost. Why
>> is this so slow, and how can I fix it or is there other way I can extract
>> the metadata in C?
>>>
>>> $ time curl -s '
>> http://localhost:50111/templeton/v1/ddl/database/default/table/haha?user.name=ctdean
>> <
>> http://localhost:50111/templeton/v1/ddl/database/default/table/haha?user.name=ctdean
>>> '
>>> {"columns":
>>> [{"name":"id","type":"int"}],
>>> "database":"default",
>>> "table":"haha"}
>>>
>>> real 0m8.400s
>>> user 0m0.053s
>>> sys 0m0.019s
>>> it seems to run a hcat.py, and it create a bunch of things then clear
>> them, it takes very long time, does anyone have some ideas about it?? Any
>> suggestions will be very appreciated!
>>>
>>> $hcat.py -e "use default; desc haha; "
>>> SLF4J: Class path contains multiple SLF4J bindings.
>>> SLF4J: Found binding in
>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings <
>> http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>> 18/04/21 16:38:13 INFO conf.HiveConf: Found configuration file
>> file:/usr/local/hive/conf/hive-site.xml
>>> 18/04/21 16:38:15 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>>> 18/04/21 16:38:16 INFO session.SessionState: Created HDFS directory:
>> /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668
>>> 18/04/21 16:38:16 INFO session.SessionState: Created local directory:
>> /tmp/hive/java/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668
>>> 18/04/21 16:38:16 INFO session.SessionState: Created HDFS directory:
>> /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668/_tmp_space.db
>>> 18/04/21 16:38:16 INFO ql.Driver: Compiling
>> command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62):
>> use default
>>> 18/04/21 16:38:17 INFO metastore.HiveMetaStore: 0: Opening raw store
>> with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
>>> 18/04/21 16:38:17 INFO metastore.ObjectStore: ObjectStore, initialize
>> called
>>> 18/04/21 16:38:18 INFO DataNucleus.Persistence: Property
>> hive.metastore.integral.jdo.pushdown unknown - will be ignored
>>> 18/04/21 16:38:18 INFO DataNucleus.Persistence: Property
>> datanucleus.cache.level2 unknown - will be ignored
>>> 18/04/21 16:38:18 INFO metastore.ObjectStore: Setting MetaStore object
>> pin classes with
>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
>>> 18/04/21 16:38:20 INFO metastore.MetaStoreDirectSql: Using direct SQL,
>> underlying DB is MYSQL
>>> 18/04/21 16:38:20 INFO metastore.ObjectStore: Initialized ObjectStore
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: Added admin role in
>> metastore
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: Added public role in
>> metastore
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: No user is added in
>> admin role, since config is empty
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_all_functions
>>> 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda
>> ip=unknown-ip-addr cmd=get_all_functions
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default
>>> 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda
>> ip=unknown-ip-addr cmd=get_database: default
>>> 18/04/21 16:38:20 INFO ql.Driver: Semantic Analysis Completed
>>> 18/04/21 16:38:20 INFO ql.Driver: Returning Hive schema:
>> Schema(fieldSchemas:null, properties:null)
>>> 18/04/21 16:38:20 INFO ql.Driver: Completed compiling
>> command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62);
>> Time taken: 3.936 seconds
>>> 18/04/21 16:38:20 INFO ql.Driver: Concurrency mode is disabled, not
>> creating a lock manager
>>> 18/04/21 16:38:20 INFO ql.Driver: Executing
>> command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62):
>> use default
>>> 18/04/21 16:38:20 INFO sqlstd.SQLStdHiveAccessController: Created
>> SQLStdHiveAccessController for session context : HiveAuthzSessionContext
>> [sessionString=05096382-f9b6-4dae-aee2-dfa6750c0668, clientType=HIVECLI]
>>> 18/04/21 16:38:20 WARN session.SessionState: METASTORE_FILTER_HOOK will
>> be ignored, since hive.security.authorization.manager is set to instance of
>> HiveAuthorizerFactory.
>>> 18/04/21 16:38:20 INFO hive.metastore: Mestastore configuration
>> hive.metastore.filter.hook changed from
>> org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to
>> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Cleaning up thread
>> local RawStore...
>>> 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda
>> ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Done cleaning up
>> thread local RawStore
>>> 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda
>> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
>>> 18/04/21 16:38:20 INFO ql.Driver: Starting task [Stage-0:DDL] in serial
>> mode
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default
>>> 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda
>> ip=unknown-ip-addr cmd=get_database: default
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Opening raw store
>> with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
>>> 18/04/21 16:38:20 INFO metastore.ObjectStore: ObjectStore, initialize
>> called
>>> 18/04/21 16:38:20 INFO metastore.MetaStoreDirectSql: Using direct SQL,
>> underlying DB is MYSQL
>>> 18/04/21 16:38:20 INFO metastore.ObjectStore: Initialized ObjectStore
>>> 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default
>>> 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda
>> ip=unknown-ip-addr cmd=get_database: default
>>> 18/04/21 16:38:20 INFO ql.Driver: Completed executing
>> command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62);
>> Time taken: 0.202 seconds
>>> OK
>>
>>
