[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096615#comment-16096615 ]
Apache Spark commented on SPARK-21494: -------------------------------------- User 'vanzin' has created a pull request for this issue: https://github.com/apache/spark/pull/18706 > Spark 2.2.0 AES encryption not working with External shuffle > ------------------------------------------------------------ > > Key: SPARK-21494 > URL: https://issues.apache.org/jira/browse/SPARK-21494 > Project: Spark > Issue Type: Bug > Components: Block Manager, Shuffle > Affects Versions: 2.2.0 > Environment: AWS EMR > Reporter: Udit Mehrotra > Attachments: logs.zip > > > Spark’s new AES based authentication mechanism does not seem to work when > configured with external shuffle service on YARN. > Here is the stack trace for the error we see in the driver logs: > ERROR YarnScheduler: Lost executor 40 on ip-10-167-104-125.ec2.internal: > Unable to create executor due to Unable to register with external shuffle > server due to: java.lang.IllegalArgumentException: Authentication failed. > at > org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125) > at > org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157) > at > org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105) > at > org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) > at > org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) > at > org.spark_project.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) > > Here are the settings we are configuring in ‘spark-defaults’ and ‘yarn-site’: > spark.network.crypto.enabled true > spark.network.crypto.saslFallback false > spark.authenticate true > > Turning on DEBUG logs for class ‘org.apache.spark.network.crypto’ on both > Spark and YARN side is not giving much information either about why > authentication fails. The driver and node manager logs have been attached to > the JIRA. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org