Tried the workaround for HDB 2.0.0 and it does not seem to be working.
I have a job which running every hour to run a COUNT on native table and
external hdfs pxf table. I have intermittently checked the Hive table via
HCatalog in the 24 hours period (close to 3 time roughly).
However, after the expiration of 24 hours since the last PXF restart the
hcatalog based SELECT queries start to fail. The external hdfs pxf table
and native table continue to work with the workaround in place.
The error remains the same. Looks like the workaround does not fix this. Am
I missing something here?
This is what I get on the psql prompt:
postgres=# SELECT * FROM hcatalog.default.hive_table;
ERROR: remote component error (500) from '127.0.0.1:51200': type
Exception report message java.lang.Exception:
java.lang.RuntimeException: Failed connecting to Hive MetaStore service:
Could not connect to meta store using any of the URIs provided. Most recent
failure: org.apache.thrift.transport.TTransportException: GSS initiate
failed description The server encountered an internal error that
prevented it from fulfilling this request. exception
javax.servlet.ServletException: java.lang.Exception:
java.lang.RuntimeException: Failed connecting to Hive MetaStore service:
Could not connect to meta store using any of the URIs provided. Most recent
failure: org.apache.thrift.transport.TTransportException: GSS initiate
failed (libchurl.c:878)
LINE 1: SELECT * FROM hcatalog.default.hive_table;
And the following is what I see in hivemetastore.log:
2017-01-28 22:46:01,713 ERROR [pool-6-thread-69]: server.TThreadPoolServer
(TThreadPoolServer.java:run(296)) - Error occurred during processing of
message.
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
GSS initiate failed
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated
failure: GSS initiate failed
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Regards,
Gagan Brahmi
On Mon, Jan 23, 2017 at 10:54 PM, Gagan Brahmi <gaganbra...@gmail.com>
wrote:
> Thank you Vineet. I've setup a job similar to the workaround for HAWQ-940.
>
> Waiting for the results till tomorrow.
>
>
> Regards,
> Gagan Brahmi
>
> On Jan 23, 2017 22:52, "Vineet Goel" <vvin...@apache.org> wrote:
>
>> Hi Gagan,
>>
>> Check the recent Pull Request by Kavinder, which should solve this issue:
>> https://github.com/apache/incubator-hawq/pull/1075
>>
>> Hi Kavinder - should JIRA HAWQ-762 be set to resolved?
>>
>> The fix will get included in the next (or next+1) Apache HAWQ release as
>> well as the next HDB release.
>>
>> Your workaround for now may include running a cron job to call a PSQL
>> script which queries a simple or dummy HAWQ PXF external table every 12
>> hours or so.
>>
>> Thanks
>> Vineet
>>
>>
>> On Mon, Jan 23, 2017 at 9:07 PM Gagan Brahmi <gaganbra...@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > I am seeing issues with secure HAWQ (HDB 2.0.0 and 2.0.1) cluster on
>> top of
>> > HDP. Querying Hive tables using hcatalog.hive_db.hive_table returns
>> errors
>> > after every 24 hours (token expiration for pxf user). The solution
>> looks to
>> > be a restart pxf for now.
>> >
>> > I see a jira mentioning this behavior but not sure if this is actively
>> > worked on right now. The jira in question is
>> > https://issues.apache.org/jira/browse/HAWQ-762
>> >
>> > While executing the query using PXF I encounter the following error:
>> >
>> > -----------
>> >
>> > ERROR: remote component error (500) from '127.0.0.1:51200': type
>> > Exception report message java.lang.Exception:
>> > java.lang.RuntimeException: Failed connecting to Hive MetaStore service:
>> > Could not connect to meta store using any of the URIs provided. Most
>> recent
>> > failure: org.apache.thrift.transport.TTransportException: GSS initiate
>> > failed description The server encountered an internal error that
>> > prevented it from fulfilling this request. exception
>> > javax.servlet.ServletException: java.lang.Exception:
>> > java.lang.RuntimeException: Failed connecting to Hive MetaStore service:
>> > Could not connect to meta store using any of the URIs provided. Most
>> recent
>> > failure: org.apache.thrift.transport.TTransportException: GSS initiate
>> > failed (libchurl.c:884)
>> >
>> > -----------
>> >
>> >
>> > While verifying Hive metastore I found the following error stack.
>> >
>> > -----------
>> >
>> > 2017-01-23 07:50:16,969 ERROR [pool-6-thread-200]:
>> server.TThreadPoolServer
>> > (TThreadPoolServer.java:run(296)) - Error occurred during processing of
>> > message.
>> > java.lang.RuntimeException:
>> > org.apache.thrift.transport.TTransportException: Peer indicated
>> failure:
>> > GSS initiate failed
>> > at
>> >
>> > org.apache.thrift.transport.TSaslServerTransport$Factory.get
>> Transport(TSaslServerTransport.java:219)
>> > at
>> >
>> > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$
>> TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
>> > at
>> >
>> > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$
>> TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:356)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1704)
>> > at
>> >
>> > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$
>> TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBr
>> idge.java:606)
>> > at
>> >
>> > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.
>> run(TThreadPoolServer.java:268)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: org.apache.thrift.transport.TTransportException: Peer
>> indicated
>> > failure: GSS initiate failed
>> > at
>> >
>> > org.apache.thrift.transport.TSaslTransport.receiveSaslMessag
>> e(TSaslTransport.java:199)
>> > at
>> >
>> > org.apache.thrift.transport.TSaslServerTransport.handleSaslS
>> tartMessage(TSaslServerTransport.java:125)
>> > at
>> > org.apache.thrift.transport.TSaslTransport.open(TSaslTranspo
>> rt.java:271)
>> > at
>> >
>> > org.apache.thrift.transport.TSaslServerTransport.open(TSaslS
>> erverTransport.java:41)
>> > at
>> >
>> > org.apache.thrift.transport.TSaslServerTransport$Factory.get
>> Transport(TSaslServerTransport.java:216)
>> > ... 10 more
>> >
>> > -----------
>> >
>> > Is there any workaround for this problem for now besides restarting PXF
>> > every 24 hours (or the kerberos ticket expiration).
>> >
>> >
>> > Regards,
>> > Gagan Brahmi
>> >
>>
>
