[ https://issues.apache.org/jira/browse/SENTRY-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Gitline updated SENTRY-2506: ---------------------------------- Description: Sentry clients (e.g., HMS) show errors like: 2019-02-26 14:04:13,451 ERROR org.apache.sentry.core.common.transport.SentryTransportPool: [pool-9-thread-131]: Failed to obtain transport for XXXX.net:8038: null 2019-02-26 14:04:40,815 ERROR org.apache.sentry.core.common.transport.SentryTransportPool: [pool-9-thread-130]: Failed to obtain transport for XXXX.net:8038: null 2019-02-26 14:04:43,816 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-9-thread-130]: Encountered failure while notifying notification for CREATE_DATABASE (id: 19) org.apache.sentry.core.common.exception.SentryUserException: java.net.SocketException: Connection reset at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] at com.sun.proxy.$Proxy34.notifyHmsEvent(Unknown Source) ~[?:?] :::::: Sentry stderr log shows: Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.Exception: Error starting server at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:596) at org.apache.sentry.SentryMain.main(SentryMain.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:313) at org.apache.hadoop.util.RunJar.main(RunJar.java:227) Caused by: java.lang.Exception: Error starting server at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:207) at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.PrivilegedActionException: java.net.BindException: Address already in use at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:195) ... 5 more Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:401) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.sentry.api.service.thrift.SentryWebServer.start(SentryWebServer.java:190) at org.apache.sentry.service.thrift.SentryService.startSentryWebServer(SentryService.java:428) at org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:273) at org.apache.sentry.service.thrift.SentryService.access$000(SentryService.java:81) at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:198) at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:195) ... 8 more port 29000 turns out to be taken by another application: $ netstat -tulpn | grep 29000 tcp 0 0 0.0.0.0:29000 0.0.0.0:* LISTEN 9510/BLAH . But Sentry still starts as if normal although the clients (Hive, Impala, etc.) cannot run queries. That is extremely confusing. Sentry startup should be corrected to fail to start in those situations with a clear error pointing at the cause. was: Sentry clients (e.g., HMS) show errors like: 2019-02-26 14:04:13,451 ERROR org.apache.sentry.core.common.transport.SentryTransportPool: [pool-9-thread-131]: Failed to obtain transport for asdf.svr.us.XXXX.net:8038: null 2019-02-26 14:04:40,815 ERROR org.apache.sentry.core.common.transport.SentryTransportPool: [pool-9-thread-130]: Failed to obtain transport for asdf.svr.us.XXXX.net:8038: null 2019-02-26 14:04:43,816 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-9-thread-130]: Encountered failure while notifying notification for CREATE_DATABASE (id: 19) org.apache.sentry.core.common.exception.SentryUserException: java.net.SocketException: Connection reset at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] at com.sun.proxy.$Proxy34.notifyHmsEvent(Unknown Source) ~[?:?] :::::: Sentry stderr log shows: Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.Exception: Error starting server at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:596) at org.apache.sentry.SentryMain.main(SentryMain.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:313) at org.apache.hadoop.util.RunJar.main(RunJar.java:227) Caused by: java.lang.Exception: Error starting server at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:207) at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.PrivilegedActionException: java.net.BindException: Address already in use at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:195) ... 5 more Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:401) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.sentry.api.service.thrift.SentryWebServer.start(SentryWebServer.java:190) at org.apache.sentry.service.thrift.SentryService.startSentryWebServer(SentryService.java:428) at org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:273) at org.apache.sentry.service.thrift.SentryService.access$000(SentryService.java:81) at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:198) at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:195) ... 8 more port 29000 turns out to be taken by another application: $ netstat -tulpn | grep 29000 tcp 0 0 0.0.0.0:29000 0.0.0.0:* LISTEN 9510/aim-agent . But Sentry still starts as if normal although the clients (Hive, Impala, etc.) cannot run queries. That is extremely confusing. Sentry startup should be corrected to fail to start in those situations with a clear error pointing at the cause. > Sentry starts but dysfunctional when Web Port is unavailable > ------------------------------------------------------------ > > Key: SENTRY-2506 > URL: https://issues.apache.org/jira/browse/SENTRY-2506 > Project: Sentry > Issue Type: Bug > Components: Sentry > Reporter: Boris Gitline > Priority: Major > > Sentry clients (e.g., HMS) show errors like: > 2019-02-26 14:04:13,451 ERROR > org.apache.sentry.core.common.transport.SentryTransportPool: > [pool-9-thread-131]: Failed to obtain transport for XXXX.net:8038: null > 2019-02-26 14:04:40,815 ERROR > org.apache.sentry.core.common.transport.SentryTransportPool: > [pool-9-thread-130]: Failed to obtain transport for XXXX.net:8038: null > 2019-02-26 14:04:43,816 ERROR > org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: > [pool-9-thread-130]: Encountered failure while notifying notification for > CREATE_DATABASE (id: 19) > org.apache.sentry.core.common.exception.SentryUserException: > java.net.SocketException: Connection reset > at > org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166) > ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] > at > org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90) > ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] > at > org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) > ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?] > at com.sun.proxy.$Proxy34.notifyHmsEvent(Unknown Source) ~[?:?] > :::::: > Sentry stderr log shows: > Exception in thread "main" java.util.concurrent.ExecutionException: > java.lang.Exception: Error starting server > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:596) > at org.apache.sentry.SentryMain.main(SentryMain.java:120) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:313) > at org.apache.hadoop.util.RunJar.main(RunJar.java:227) > Caused by: java.lang.Exception: Error starting server > at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:207) > at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.security.PrivilegedActionException: java.net.BindException: > Address already in use > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:195) > ... 5 more > Caused by: java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:433) > at sun.nio.ch.Net.bind(Net.java:425) > at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317) > at > org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) > at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at org.eclipse.jetty.server.Server.doStart(Server.java:401) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > org.apache.sentry.api.service.thrift.SentryWebServer.start(SentryWebServer.java:190) > at > org.apache.sentry.service.thrift.SentryService.startSentryWebServer(SentryService.java:428) > at > org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:273) > at > org.apache.sentry.service.thrift.SentryService.access$000(SentryService.java:81) > at > org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:198) > at > org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:195) > ... 8 more > port 29000 turns out to be taken by another application: > $ netstat -tulpn | grep 29000 > tcp 0 0 0.0.0.0:29000 0.0.0.0:* > LISTEN 9510/BLAH > . But Sentry still starts as if normal although the clients (Hive, Impala, > etc.) cannot run queries. That is extremely confusing. Sentry startup should > be corrected to fail to start in those situations with a clear error pointing > at the cause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)