Can you please check the regionserver log on datanode-001.fqdn.com that has hbase:meta region after enabling DEBUG log? That will show the exact reason why the connections are failing.
On Wed, May 29, 2019 at 10:48 AM Aleksandr Saraseka <asaras...@eztexting.com> wrote: > Thank you Josh, that helps a lot. > We have Query Server on a dedicated server and none of existing guides > have an information that we need to have core-site.xml > with hadoop.security.authentication option set to kerberos. > > On Tue, May 28, 2019 at 11:59 PM Josh Elser <els...@apache.org> wrote: > >> Make sure you have authorization set up correctly between PQS and HBase. >> >> Specifically, you must have the appropriate Hadoop proxyuser rules set >> up in core-site.xml so that HBase will allow PQS to impersonate the PQS >> end-user. >> >> On 5/14/19 11:04 AM, Aleksandr Saraseka wrote: >> > Hello, I have HBase + PQS 4.14.1 >> > If I'm trying to connect by think client - everything works, but if I'm >> > using thin client in PQS logs I can see continuous INFO messages >> > 2019-05-14 13:53:58,701 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=10, retries=35, started=48292 ms ago, cancelled=false, msg= >> > ... >> > 2019-05-14 14:18:41,446 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=33, retries=35, started=510325 ms ago, cancelled=false, msg= >> > 2019-05-14 14:19:01,489 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=34, retries=35, started=530368 ms ago, cancelled=false, msg= >> > ... >> > 2019-05-14 14:18:41,446 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=33, retries=35, started=510325 ms ago, cancelled=false, msg= >> > 2019-05-14 14:19:01,489 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=34, retries=35, started=530368 ms ago, cancelled=false, msg= >> > 2019-05-14 14:19:50,139 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=10, retries=35, started=48480 ms ago, cancelled=false, msg=row >> > 'SYSTEM:CATALOG,,' on table 'hbase:meta' at >> > region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com >> > <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0 >> > 2019-05-14 14:20:10,333 INFO >> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, >> > tries=11, retries=35, started=68676 ms ago, cancelled=false, msg=row >> > 'SYSTEM:CATALOG,,' on table 'hbase:meta' at >> > region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com >> > <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0 >> > >> > *Hbase security logs:* >> > 2019-05-14 14:42:19,524 INFO >> > SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for >> > HTTP/phoenix-queryserver-fqdn....@realm.com >> > <mailto:phoenix-queryserver-fqdn....@realm.com> (auth:KERBEROS) >> > 2019-05-14 14:42:19,524 INFO >> > SecurityLogger.org.apache.hadoop.hbase.Server: Connection from >> > 10.252.16.253 port: 41040 with version info: version: "1.2.0-cdh5.14.2" >> > url: >> > >> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2" >> >> > revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT >> 2018" >> > src_checksum: "05e6e90e06dd7796f56067208a9bf2aa" >> > 2019-05-14 14:42:29,634 INFO >> > SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for >> > HTTP/phoenix-queryserver-fqdn....@realm.com >> > <mailto:phoenix-queryserver-fqdn....@realm.com> (auth:KERBEROS) >> > 2019-05-14 14:42:29,635 INFO >> > SecurityLogger.org.apache.hadoop.hbase.Server: Connection from >> > 10.252.16.253 port: 41046 with version info: version: "1.2.0-cdh5.14.2" >> > url: >> > >> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2" >> >> > revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT >> 2018" >> > src_checksum: "05e6e90e06dd7796f56067208a9bf2aa" >> > >> > >> > *thin client logs:* >> > 19/05/14 14:10:08 DEBUG execchain.MainClientExec: Proxy auth state: >> > UNCHALLENGED >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> POST / HTTP/1.1 >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> >> Content-Length: 137 >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Type: >> > application/octet-stream >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Host: >> > host-fqdn.com:8765 <http://host-fqdn.com:8765> >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Connection: >> > Keep-Alive >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> User-Agent: >> > Apache-HttpClient/4.5.2 (Java/1.8.0_161) >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> >> > Accept-Encoding: gzip,deflate >> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Authorization: >> > Negotiate >> > >> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg== >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "POST / >> > HTTP/1.1[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Length: >> > 137[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Type: >> > application/octet-stream[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Host: >> > host-fqdn.com:8765[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Connection: >> > Keep-Alive[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "User-Agent: >> > Apache-HttpClient/4.5.2 (Java/1.8.0_161)[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Accept-Encoding: >> > gzip,deflate[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Authorization: >> > Negotiate >> > >> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> >> > >> "?org.apache.calcite.avatica.proto.Requests$OpenConnectionRequest[0x12]F[\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> >> > "$5de75f3c-d53d-4a53-b78c-4167156a6b67[0x12][0x10][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> >> > "[0x8]password[0x12][0x4]none[0x12][0xc][\n]" >> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> >> > "[0x4]user[0x12][0x4]none" >> > >> > *and thin client fails with:* >> > Tue May 14 14:59:43 UTC 2019, >> > RpcRetryingCaller{globalStartTime=1557845452306, pause=100, >> retries=35}, >> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to >> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020> >> > failed on local exception: >> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: >> > Connection to datasys-secure-hbase-data001- >> > stg.c.cf-stage.internal/10.252.20.182:60020 <http://10.252.20.182:60020> >> >> > is closing. Call id=69, waitTime=15 >> > >> > at >> > >> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157) >> > at >> > >> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > ... 1 more >> > Caused by: >> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to >> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020> >> > failed on local exception: org.apac >> > he.hadoop.hbase.exceptions.ConnectionClosingException: Connection to >> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020> >> > is closing. Call id=69, waitTime=15 >> > >> > Firewall is widely open from PQS to all HBase/Hadoop nodes. >> > Also can someone provide impersonal config for working PQS with >> Kerberos >> > ? Maybe I missed something. >> > >> > -- >> > >> > >> > Aleksandr Saraseka >> > DBA at EZ Texting >> > >> > M 380997600401 <tel:380997600401> >> > >> > E asaras...@eztexting.com <mailto:asaras...@eztexting.com> >> > >> > W http://www.eztexting.com >> > < >> http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >> >> > >> > >> > < >> http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >> >> > < >> http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >> >> > < >> http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >> >> > < >> https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >> >> > < >> https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature >> > >> > >> > > > -- > Aleksandr Saraseka > DBA at EZ Texting > M 380997600401 > E asaras...@eztexting.com > W http://www.eztexting.com > <http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > > > <http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > <http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > <http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > <https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > <https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >
