Ahh, looks like it was the ambari-agent process, part of HDP. Since that runs on the same machine, it wasn't in my tcpdump.
Second time ambari-agent has done something unexpected for me! (first was renewing a keytab behind my back) On Sun, 8 Dec 2019 at 12:39, James Srinivasan <[email protected]> wrote: > > I'm running Accumulo 1.7 (HDP3) on a Kerberized cluster. When trying > to debug some client libthrift issues, I noticed errors like this > every minute (pretty much on the minute) in my tserver logs: > > 2019-12-08 12:35:01,255 [server.TThreadPoolServer] ERROR: Error > occurred during processing of message. > java.lang.RuntimeException: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > at > org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51) > at > org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > at > org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > at > org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) > at > org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > ... 11 more > > I get them even if there are no clients running and the tserver is the > only accumulo process running in the cluster (no master, no other > tservers etc.) and curiously I don't see any network traffic on port > 9997. Any idea how to debug further? > > Many thanks, > > James
