[ 
https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638634#comment-14638634
 ] 

Rodolfo Grave commented on SAMZA-727:
-------------------------------------

When running hello-samza on a CDH 5.4.0 Hadoop cluster secured with Kerberos, I 
get the following error:

java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: "xxxx/xx.xx.xx.xx"; destination host is: 
"xxxxx":8020;

Running klist before and after running the Samza job returns:

-bash-4.1$ klist
Ticket cache: FILE:/tmp/krb5cc_3073905
Default principal: myu...@intranet.mycompany.com

Valid starting     Expires            Service principal
07/23/15 10:00:22  07/23/15 20:00:22  
krbtgt/intranet.mycompany....@intranet.mycompany.com

The full stack trace is:

2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got container id: 
container_e105_1437556818770_0108_01_000001
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got app attempt id: 
appattempt_1437556818770_0108_000001
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got node manager host: 
myserver2660.intranet.mycompany.com
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got node manager port: 8041
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got node manager http port: 8042
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got config: 
{task.inputs=kafka.wikipedia-raw, 
metrics.reporter.jmx.class=org.apache.samza.metrics.reporter.JmxReporterFactory,
 job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, 
serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory,
 
metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory,
 task.checkpoint.replication.factor=1, job.name=wikipedia-parser, 
task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory,
 systems.kafka.consumer.zookeeper.connect=myserver2649:2181/, 
systems.kafka.consumer.auto.offset.reset=largest, 
systems.kafka.samza.msg.serde=json, 
serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory, 
metrics.reporters=snapshot,jmx, task.checkpoint.system=kafka, 
systems.kafka.streams.metrics.samza.msg.serde=metrics, 
systems.kafka.producer.bootstrap.servers=myserver2663:9092,myserver2664:9092,myserver2665:9092,
 task.class=samza.examples.wikipedia.task.WikipediaParserStreamTask, 
yarn.package.path=hdfs://nameservice1/bigdata/projects/myprojectplayground/samza-hello-samza-master-0.9.1-dist.tar.gz,
 metrics.reporter.snapshot.stream=kafka.metrics, 
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory}
2015-07-23 10:00:57 ClientHelper [INFO] trying to connect to RM 0.0.0.0:8032
2015-07-23 10:00:58 JmxServer [INFO] According to 
InetAddress.getLocalHost.getHostName we are myserver2660.intranet.mycompany.com
2015-07-23 10:00:58 JmxServer [INFO] Started JmxServer registry port=46994 
server port=46995 
url=service:jmx:rmi://localhost:46995/jndi/rmi://localhost:46994/jmxrmi
2015-07-23 10:00:58 JmxServer [INFO] If you are tunneling, you might want to 
try JmxServer registry port=46994 server port=46995 
url=service:jmx:rmi://myserver2660.intranet.mycompany.com:46995/jndi/rmi://myserver2660.intranet.mycompany.com:46994/jmxrmi
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Creating new metrics 
snapshot reporter.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [WARN] Unable to find 
implementation version in jar's meta info. Defaulting to 0.0.1.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got system stream 
SystemStream [system=kafka, stream=metrics].
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got system factory 
org.apache.samza.system.kafka.KafkaSystemFactory@2ad6b556.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got producer 
org.apache.samza.system.kafka.KafkaSystemProducer@32e3e90f.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got serde 
org.apache.samza.serializers.MetricsSnapshotSerde@2475986a.
2015-07-23 10:00:58 MetricsSnapshotReporter [INFO] got metrics snapshot 
reporter properties [job name: wikipedia-parser, job id: 1, containerName: 
ApplicationMaster, version: 0.0.1, samzaVersion: 0.9.1, host: 
myserver2660.intranet.mycompany.com]
2015-07-23 10:00:58 MetricsSnapshotReporter [INFO] Registering 
MetricsSnapshotReporterFactory with producer.
2015-07-23 10:00:58 MetricsSnapshotReporter [INFO] Registering 
ApplicationMaster with producer.
2015-07-23 10:00:58 JmxReporterFactory [INFO] Creating JMX reporter with  name 
jmx.
2015-07-23 10:00:58 SamzaAppMasterTaskManager [INFO] No yarn.container.count 
specified. Defaulting to one container.
2015-07-23 10:00:58 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:58 VerifiableProperties [INFO] Property auto.offset.reset is 
overridden to largest
2015-07-23 10:00:58 VerifiableProperties [INFO] Property client.id is 
overridden to samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1
2015-07-23 10:00:58 VerifiableProperties [INFO] Property group.id is overridden 
to undefined-samza-consumer-group-ce2ba112-ecf1-4304-bf70-bc5dda3aea3b
2015-07-23 10:00:58 VerifiableProperties [INFO] Property zookeeper.connect is 
overridden to myserver2649:2181/
2015-07-23 10:00:58 KafkaCheckpointManager [INFO] Creating 
KafkaCheckpointManager with: 
clientId=samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1, 
checkpointTopic=__samza_checkpoint_ver_1_for_wikipedia-parser_1, 
systemName=kafka
2015-07-23 10:00:58 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:58 VerifiableProperties [INFO] Property auto.offset.reset is 
overridden to largest
2015-07-23 10:00:58 VerifiableProperties [INFO] Property client.id is 
overridden to samza_admin-wikipedia_parser-1-1437642058864-2
2015-07-23 10:00:58 VerifiableProperties [INFO] Property group.id is overridden 
to undefined-samza-consumer-group-bb834e1a-1861-459e-a310-108667ccf2a0
2015-07-23 10:00:58 VerifiableProperties [INFO] Property zookeeper.connect is 
overridden to myserver2649:2181/
2015-07-23 10:00:58 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:58 VerifiableProperties [INFO] Property client.id is 
overridden to samza_admin-wikipedia_parser-1-1437642058864-2
2015-07-23 10:00:58 VerifiableProperties [INFO] Property metadata.broker.list 
is overridden to myserver2663:9092,myserver2664:9092,myserver2665:9092
2015-07-23 10:00:58 VerifiableProperties [INFO] Property request.timeout.ms is 
overridden to 30000
2015-07-23 10:00:58 ClientUtils$ [INFO] Fetching metadata from broker 
id:0,host:myserver2663,port:9092 with correlation id 0 for 1 topic(s) 
Set(wikipedia-raw)
2015-07-23 10:00:58 SyncProducer [INFO] Connected to myserver2663:9092 for 
producing
2015-07-23 10:00:59 SyncProducer [INFO] Disconnecting from myserver2663:9092
2015-07-23 10:00:59 KafkaSystemAdmin$ [INFO] Got metadata: Map(wikipedia-raw -> 
SystemStreamMetadata [streamName=wikipedia-raw, partitionMetadata={Partition 
[partition=0]=SystemStreamPartitionMetadata [oldestOffset=null, 
newestOffset=null, upcomingOffset=0]}])
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Attempting to create 
checkpoint topic __samza_checkpoint_ver_1_for_wikipedia-parser_1.
2015-07-23 10:00:59 ZkEventThread [INFO] Starting ZkClient event thread.
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 19:12 GMT
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:host.name=myserver2660.intranet.mycompany.com
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.version=1.7.0_67
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.vendor=Oracle 
Corporation
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:java.class.path=/var/run/cloudera-scm-agent/process/7124-yarn-NODEMANAGER:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/activation-1.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/akka-actor_2.10-2.1.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/aopalliance-1.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/apacheds-i18n-2.0.0-M15.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/api-asn1-api-1.0.0-M20.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/api-util-1.0.0-M20.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/asm-3.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/avro-1.7.6-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/cglib-2.2.1-v20090111.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-beanutils-1.7.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-beanutils-core-1.8.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-cli-1.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-codec-1.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-collections-3.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-compress-1.4.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-configuration-1.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-daemon-1.0.13.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-digester-1.8.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-el-1.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-httpclient-3.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-io-2.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-lang-2.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-logging-1.1.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-math3-3.1.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-net-3.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/config-1.0.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/curator-client-2.7.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/curator-framework-2.7.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/curator-recipes-2.7.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/grizzled-slf4j_2.10-1.0.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/gson-2.2.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/guava-16.0.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/guice-3.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/guice-servlet-3.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-annotations-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-auth-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-common-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-hdfs-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-yarn-api-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-yarn-client-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-yarn-common-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/htrace-core-3.0.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/httpclient-4.2.5.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/httpcore-4.2.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/irclib-1.10.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-core-asl-1.9.13.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-jaxrs-1.8.8.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-mapper-asl-1.9.13.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-xc-1.8.8.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jasper-compiler-5.5.23.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jasper-runtime-5.5.23.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/java-xmlbuilder-0.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/javax.inject-1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/javax.servlet-3.0.0.v201112011016.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jaxb-api-2.2.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jaxb-impl-2.2.3-1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-client-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-core-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-guice-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-json-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-server-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jets3t-0.9.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jettison-1.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-6.1.26.cloudera.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-continuation-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-http-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-io-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-security-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-server-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-servlet-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-util-6.1.26.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-util-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-webapp-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-xml-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jline-0.9.94.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/joda-convert-1.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/joda-time-2.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jopt-simple-3.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jsch-0.1.42.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jsp-api-2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jsr305-3.0.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/junit-3.8.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/juniversalchardet-1.0.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/kafka-clients-0.8.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/kafka_2.10-0.8.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/log4j-1.2.17.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/lz4-1.2.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/metrics-core-2.2.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/mime-util-2.1.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/netty-3.7.0.Final.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/paranamer-2.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/protobuf-java-2.5.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/rl_2.10-0.4.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/rocksdbjni-3.5.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-api-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-core_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-hello-samza-master-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-kafka_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-kv-rocksdb_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-kv_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-log4j-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-shell-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-yarn_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scala-compiler-2.10.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scala-library-2.10.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scala-reflect-2.10.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalate-core_2.10-1.6.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalate-util_2.10-1.6.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalatra-common_2.10-2.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalatra-scalate_2.10-2.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalatra_2.10-2.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/servlet-api-2.5.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/slf4j-api-1.7.7.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/slf4j-log4j12-1.7.7.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/snappy-java-1.1.1.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/stax-api-1.0-2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/xmlenc-0.52.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/xz-1.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/zkclient-0.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/zookeeper-3.4.5-cdh5.4.0.jar
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:java.io.tmpdir=/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/tmp
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:os.name=Linux
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:os.arch=amd64
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:os.version=2.6.32-504.16.2.el6.x86_64
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:user.name=myuser
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:user.home=/home/myuser
2015-07-23 10:00:59 ZooKeeper [INFO] Client 
environment:user.dir=/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001
2015-07-23 10:00:59 ZooKeeper [INFO] Initiating client connection, 
connectString=myserver2649:2181/ sessionTimeout=6000 
watcher=org.I0Itec.zkclient.ZkClient@7dc94405
2015-07-23 10:00:59 ClientCnxn [INFO] Opening socket connection to server 
myserver2649.intranet.mycompany.com/10.xx.xx.1:2181. Will not attempt to 
authenticate using SASL (unknown error)
2015-07-23 10:00:59 ClientCnxn [INFO] Socket connection established to 
myserver2649.intranet.mycompany.com/10.xx.xx.1:2181, initiating session
2015-07-23 10:00:59 ClientCnxn [INFO] Session establishment complete on server 
myserver2649.intranet.mycompany.com/10.xx.xx.1:2181, sessionid = 
0x34e4ea7b9cba938, negotiated timeout = 6000
2015-07-23 10:00:59 ZkClient [INFO] zookeeper state changed (SyncConnected)
2015-07-23 10:00:59 ZkEventThread [INFO] Terminate ZkClient event thread.
2015-07-23 10:00:59 ZooKeeper [INFO] Session: 0x34e4ea7b9cba938 closed
2015-07-23 10:00:59 ClientCnxn [INFO] EventThread shut down
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Checkpoint topic 
__samza_checkpoint_ver_1_for_wikipedia-parser_1 already exists.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Validating checkpoint topic 
__samza_checkpoint_ver_1_for_wikipedia-parser_1.
2015-07-23 10:00:59 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:59 VerifiableProperties [INFO] Property client.id is 
overridden to samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1
2015-07-23 10:00:59 VerifiableProperties [INFO] Property metadata.broker.list 
is overridden to myserver2663:9092,myserver2664:9092,myserver2665:9092
2015-07-23 10:00:59 VerifiableProperties [INFO] Property request.timeout.ms is 
overridden to 30000
2015-07-23 10:00:59 ClientUtils$ [INFO] Fetching metadata from broker 
id:0,host:myserver2663,port:9092 with correlation id 0 for 1 topic(s) 
Set(__samza_checkpoint_ver_1_for_wikipedia-parser_1)
2015-07-23 10:00:59 SyncProducer [INFO] Connected to myserver2663:9092 for 
producing
2015-07-23 10:00:59 SyncProducer [INFO] Disconnecting from myserver2663:9092
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Successfully validated 
checkpoint topic __samza_checkpoint_ver_1_for_wikipedia-parser_1.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Connecting to leader 
myserver2663.intranet.mycompany.com:9092 for topic 
__samza_checkpoint_ver_1_for_wikipedia-parser_1 and to fetch all checkpoint 
messages.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Got offset 0 for topic 
__samza_checkpoint_ver_1_for_wikipedia-parser_1 and partition 0. Attempting to 
fetch messages for changelog partition mapping.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Get latest offset 1 for topic 
__samza_checkpoint_ver_1_for_wikipedia-parser_1 and partition 0.
2015-07-23 10:00:59 JobCoordinator$ [INFO] SystemStreamPartitionGrouper 
org.apache.samza.container.grouper.stream.GroupByPartition@3022338b has grouped 
the SystemStreamPartitions into the following taskNames:
2015-07-23 10:00:59 JobCoordinator$ [INFO] Saving task-to-changelog partition 
mapping: Map(Partition 0 -> 0)
2015-07-23 10:00:59 ProducerConfig [INFO] ProducerConfig values: 
        value.serializer = class 
org.apache.kafka.common.serialization.ByteArraySerializer
        key.serializer = class 
org.apache.kafka.common.serialization.ByteArraySerializer
        block.on.buffer.full = true
        retry.backoff.ms = 100
        buffer.memory = 33554432
        batch.size = 16384
        metrics.sample.window.ms = 30000
        metadata.max.age.ms = 300000
        receive.buffer.bytes = 32768
        timeout.ms = 30000
        max.in.flight.requests.per.connection = 1
        bootstrap.servers = [myserver2663:9092, myserver2664:9092, 
myserver2665:9092]
        metric.reporters = []
        client.id = samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1
        compression.type = none
        retries = 2147483647
        max.request.size = 1048576
        send.buffer.bytes = 131072
        acks = all
        reconnect.backoff.ms = 10
        linger.ms = 0
        metrics.num.samples = 2
        metadata.fetch.timeout.ms = 60000

2015-07-23 10:00:59 SamzaAppMasterService [INFO] Starting webapp at a random 
rpc and tracking port
2015-07-23 10:00:59 ClientHelper [INFO] trying to connect to RM 0.0.0.0:8032
2015-07-23 10:00:59 Server [INFO] jetty-8.1.8.v20121106
2015-07-23 10:00:59 ServletTemplateEngine [INFO] Scalate template engine using 
working directory: 
/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/tmp/scalate-4241117790539801997-workdir
2015-07-23 10:00:59 AbstractConnector [INFO] Started 
SelectChannelConnector@0.0.0.0:58211
2015-07-23 10:00:59 HttpServer [INFO] Started HttpServer on: 
http://10.xx.xx.14:58211/
2015-07-23 10:00:59 Server [INFO] jetty-8.1.8.v20121106
2015-07-23 10:01:00 ServletTemplateEngine [INFO] Scalate template engine using 
working directory: 
/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/tmp/scalate-4463253233581688239-workdir
2015-07-23 10:01:00 AbstractConnector [INFO] Started 
SelectChannelConnector@0.0.0.0:32842
2015-07-23 10:01:00 HttpServer [INFO] Started HttpServer on: 
http://10.xx.xx.14:32842/
2015-07-23 10:01:00 Server [INFO] jetty-8.1.8.v20121106
2015-07-23 10:01:00 AbstractConnector [INFO] Started 
SelectChannelConnector@0.0.0.0:42237
2015-07-23 10:01:00 HttpServer [INFO] Started HttpServer on: 
http://10.xx.xx.14:42237/
2015-07-23 10:01:00 JobCoordinator [INFO] Startd HTTP server: 
http://10.xx.xx.14:42237/
2015-07-23 10:01:00 SamzaAppMasterService [INFO] Webapp is started at (rpc 
http://10.xx.xx.14:58211/, tracking http://10.xx.xx.14:32842/, coordinator 
http://10.xx.xx.14:42237/)
2015-07-23 10:01:03 SamzaAppMasterLifecycle [INFO] Got AM register response. 
The YARN RM supports container requests with max-mem: 307200, max-cpu: 48
2015-07-23 10:01:03 MetricsSnapshotReporter [INFO] Starting producer.
2015-07-23 10:01:03 MetricsSnapshotReporter [INFO] Starting reporter timer.
2015-07-23 10:01:03 ContainerManagementProtocolProxy [INFO] 
yarn.client.max-cached-nodemanagers-proxies : 0
2015-07-23 10:01:03 SamzaAppMasterTaskManager [INFO] Requesting 1 containers
2015-07-23 10:01:03 SamzaAppMasterTaskManager [INFO] Requesting 1 container(s) 
with 1024mb of memory
2015-07-23 10:01:03 KafkaSystemProducer [INFO] Creating a new producer for 
system kafka.
2015-07-23 10:01:03 ProducerConfig [INFO] ProducerConfig values: 
        value.serializer = class 
org.apache.kafka.common.serialization.ByteArraySerializer
        key.serializer = class 
org.apache.kafka.common.serialization.ByteArraySerializer
        block.on.buffer.full = true
        retry.backoff.ms = 100
        buffer.memory = 33554432
        batch.size = 16384
        metrics.sample.window.ms = 30000
        metadata.max.age.ms = 300000
        receive.buffer.bytes = 32768
        timeout.ms = 30000
        max.in.flight.requests.per.connection = 1
        bootstrap.servers = [myserver2663:9092, myserver2664:9092, 
myserver2665:9092]
        metric.reporters = []
        client.id = samza_producer-wikipedia_parser-1-1437642058692-0
        compression.type = none
        retries = 2147483647
        max.request.size = 1048576
        send.buffer.bytes = 131072
        acks = 1
        reconnect.backoff.ms = 10
        linger.ms = 0
        metrics.num.samples = 2
        metadata.fetch.timeout.ms = 60000

2015-07-23 10:01:05 AMRMClientImpl [INFO] Received new token for : 
myserver2665.intranet.mycompany.com:8041
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Got a container from YARN 
ResourceManager: Container: [ContainerId: 
container_e105_1437556818770_0108_01_000002, NodeId: 
myserver2665.intranet.mycompany.com:8041, NodeHttpAddress: 
myserver2665.intranet.mycompany.com:8042, Resource: <memory:17408, vCores:1>, 
Priority: 0, Token: Token { kind: ContainerToken, service: 10.xx.xx.19:8041 }, ]
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Got available task id (0) 
for container: Container: [ContainerId: 
container_e105_1437556818770_0108_01_000002, NodeId: 
myserver2665.intranet.mycompany.com:8041, NodeHttpAddress: 
myserver2665.intranet.mycompany.com:8042, Resource: <memory:17408, vCores:1>, 
Priority: 0, Token: Token { kind: ContainerToken, service: 10.xx.xx.19:8041 }, ]
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Claimed SSP taskNames 
ContainerModel [containerId=0, tasks={Partition 0=TaskModel [taskName=Partition 
0, systemStreamPartitions=[SystemStreamPartition [kafka, wikipedia-raw, 0]], 
changeLogPartition=Partition [partition=0]]}] for container ID 0
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Task ID 0 using command 
bin/run-container.sh
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Task ID 0 using env 
Map(JAVA_OPTS -> , SAMZA_CONTAINER_ID -> 0, SAMZA_COORDINATOR_URL -> 
http://10.xx.xx.14:42237/)
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Starting task ID 0 using 
package path 
hdfs://nameservice1/bigdata/projects/myprojectplayground/samza-hello-samza-master-0.9.1-dist.tar.gz
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] starting container 
hdfs://nameservice1/bigdata/projects/myprojectplayground/samza-hello-samza-master-0.9.1-dist.tar.gz
 Container: [ContainerId: container_e105_1437556818770_0108_01_000002, NodeId: 
myserver2665.intranet.mycompany.com:8041, NodeHttpAddress: 
myserver2665.intranet.mycompany.com:8042, Resource: <memory:17408, vCores:1>, 
Priority: 0, Token: Token { kind: ContainerToken, service: 10.xx.xx.19:8041 }, 
] Map(JAVA_OPTS -> , SAMZA_CONTAINER_ID -> 0, SAMZA_COORDINATOR_URL -> 
http://10.xx.xx.14:42237/) WrappedArray(export SAMZA_LOG_DIR=<LOG_DIR> && ln 
-sfn <LOG_DIR> logs && exec ./__package/bin/run-container.sh 1>logs/stdout 
2>logs/stderr)
2015-07-23 10:01:05 NativeCodeLoader [WARN] Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 1 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 2 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 3 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 4 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 5 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 6 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 7 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 8 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 9 fail over attempts. 
Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 10 fail over 
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 11 fail over 
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 12 fail over 
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 13 fail over 
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking 
getFileInfo of class ClientNamenodeProtocolTranslatorPB over 
myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 14 fail over 
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2650.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the 
server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException 
as:myuser (auth:KERBEROS) cause:java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [WARN] Exception while invoking 
class 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo
 over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020. Not retrying because 
failovers (15) exceeded maximum allowed (15)
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:05 SamzaAppMaster$ [ERROR] Error occured in amClient's callback
java.io.IOException: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: 
"myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: 
"myserver2649.intranet.mycompany.com":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
        at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
        at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
        at 
org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at 
org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:1438)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed 
to find any Kerberos tgt)
        at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
2015-07-23 10:01:06 ContextHandler [INFO] stopped 
o.e.j.s.ServletContextHandler{/,jar:file:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/filecache/10/samza-hello-samza-master-0.9.1-dist.tar.gz/lib/samza-yarn_2.10-0.9.1.jar!/scalate}
2015-07-23 10:01:07 HttpServer [INFO] Stopped server
2015-07-23 10:01:07 ContextHandler [INFO] stopped 
o.e.j.s.ServletContextHandler{/,jar:file:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/filecache/10/samza-hello-samza-master-0.9.1-dist.tar.gz/lib/samza-yarn_2.10-0.9.1.jar!/scalate}
2015-07-23 10:01:07 HttpServer [INFO] Stopped server
2015-07-23 10:01:07 ContextHandler [INFO] stopped 
o.e.j.s.ServletContextHandler{/,null}
2015-07-23 10:01:07 HttpServer [INFO] Stopped server
2015-07-23 10:01:07 JobCoordinator [INFO] Stopped HTTP server.
2015-07-23 10:01:07 SamzaAppMasterLifecycle [INFO] Shutting down.
2015-07-23 10:01:07 AMRMClientImpl [INFO] Waiting for application to be 
successfully unregistered.
2015-07-23 10:01:07 MetricsSnapshotReporter [INFO] Stopping producer.
2015-07-23 10:01:07 MetricsSnapshotReporter [INFO] Stopping reporter timer.

> Support for Kerberos
> --------------------
>
>                 Key: SAMZA-727
>                 URL: https://issues.apache.org/jira/browse/SAMZA-727
>             Project: Samza
>          Issue Type: New Feature
>          Components: yarn
>    Affects Versions: 0.9.0
>         Environment: YARN with Kerberos
>            Reporter: Qi FU
>
> Samza doesn't support Kerberos, which is very common for YARN cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to