Esteban Avendaño created HADOOP-18054:
-----------------------------------------
Summary: Unable to load AWS credentials from any provider in the
chain
Key: HADOOP-18054
URL: https://issues.apache.org/jira/browse/HADOOP-18054
Project: Hadoop Common
Issue Type: Bug
Components: auth, fs, fs/s3, security
Affects Versions: 3.3.1
Environment: From top to down.
Kubernetes version 1.18.20
Spark Version: 2.4.4
Kubernetes Setup: Pod with serviceAccountName that binds with IAM Role using
IRSA (EKS Feature).
{code:java}
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn:
arn:aws:iam::999999999999:role/EKSDefaultPolicyFor-Spark
name: spark
namespace: spark {code}
AWS Setup:
IAM Role with permissions over the S3 Bucket
Bucket with permissions granted over the IAM Role.
Code:
{code:java}
def run_etl():
sc =
SparkSession.builder.appName("TXD-PYSPARK-ORACLE-SIEBEL-CASOS").getOrCreate()
sqlContext = SQLContext(sc)
args = sys.argv
load_date = args[1] # Ej: "2019-05-21"
output_path = args[2] # Ej: s3://mybucket/myfolder
print(args, "load_date", load_date, "output_path", output_path)
sc._jsc.hadoopConfiguration().set(
"fs.s3a.aws.credentials.provider",
"com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
)
sc._jsc.hadoopConfiguration().set("com.amazonaws.services.s3.enableV4",
"true")
sc._jsc.hadoopConfiguration().set("fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem")
# sc._jsc.hadoopConfiguration().set("fs.s3.impl",
"org.apache.hadoop.fs.s3native.NativeS3FileSystem")
sc._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.s3a.impl",
"org.apache.hadoop.fs.s3a.S3A")
session = boto3.session.Session()
client = session.client(service_name='secretsmanager',
region_name="us-east-1")
get_secret_value_response = client.get_secret_value(
SecretId="Siebel_Connection_Info"
)
secret = get_secret_value_response["SecretString"]
secret = json.loads(secret)
db_username = secret.get("db_username")
db_password = secret.get("db_password")
db_host = secret.get("db_host")
db_port = secret.get("db_port")
db_name = secret.get("db_name")
db_url = "jdbc:oracle:thin:@{}:{}/{}".format(db_host, db_port, db_name)
jdbc_driver_name = "oracle.jdbc.OracleDriver"
dbtable = """(SELECT * FROM SIEBEL.REPORTE_DE_CASOS WHERE JOB_ID IN (SELECT
JOB_ID FROM SIEBEL.SERVICE_CONSUMED_STATUS WHERE PUBLISH_INFORMATION_DT BETWEEN
TO_DATE('{} 00:00:00', 'YYYY-MM-DD HH24:MI:SS') AND TO_DATE('{} 23:59:59',
'YYYY-MM-DD HH24:MI:SS')))""".format(load_date, load_date)
df = sqlContext.read\
.format("jdbc")\
.option("charset", "utf8")\
.option("driver", jdbc_driver_name)\
.option("url",db_url)\
.option("dbtable", dbtable)\
.option("user", db_username)\
.option("password", db_password)\
.option("oracle.jdbc.timezoneAsRegion", "false")\
.load()
# Particionado
a_load_date = load_date.split('-')
df = df.withColumn("year", lit(a_load_date[0]))
df = df.withColumn("month", lit(a_load_date[1]))
df = df.withColumn("day", lit(a_load_date[2]))
df.write.mode("append").partitionBy(["year", "month",
"day"]).csv(output_path, header=True)
# Es importante cerrar la conexion para evitar problemas como el reportado
en
#
https://stackoverflow.com/questions/40830638/cannot-load-main-class-from-jar-file
sc.stop()
if __name__ == '__main__':
run_etl() {code}
Log's
{code:java}
+ '[' -z s3://mybucket.spark.jobs/siebel-casos-actividades ']'
+ aws s3 cp s3://mybucket.spark.jobs/siebel-casos-actividades /opt/ --recursive
--include '*'
download:
s3://mybucket.spark.jobs/siebel-casos-actividades/txd-pyspark-siebel-casos.py
to ../../txd-pyspark-siebel-casos.py
download:
s3://mybucket.spark.jobs/siebel-casos-actividades/txd-pyspark-siebel-actividades.py
to ../../txd-pyspark-siebel-actividades.py
download: s3://mybucket.jobs/siebel-casos-actividades/hadoop-aws-3.3.1.jar to
../../hadoop-aws-3.3.1.jar
download: s3://mybucket.spark.jobs/siebel-casos-actividades/ojdbc8.jar to
../../ojdbc8.jar
download:
s3://mybucket.spark.jobs/siebel-casos-actividades/aws-java-sdk-bundle-1.11.901.jar
to ../../aws-java-sdk-bundle-1.11.901.jar
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver-py
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '2021-12-18 s3a://mybucket.raw/siebel/casos/' ']'
+ PYSPARK_ARGS='2021-12-18 s3a://mybucket.raw/siebel/casos/'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' 3 == 2 ']'
+ '[' 3 == 3 ']'
++ python3 -V
+ pyv3='Python 3.6.9'
+ export PYTHON_VERSION=3.6.9
+ PYTHON_VERSION=3.6.9
+ export PYSPARK_PYTHON=python3
+ PYSPARK_PYTHON=python3
+ export PYSPARK_DRIVER_PYTHON=python3
+ PYSPARK_DRIVER_PYTHON=python3
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf
"spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@"
$PYSPARK_PRIMARY $PYSPARK_ARGS)
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
spark.driver.bindAddress=0.0.0.0/0 --deploy-mode client --properties-file
/opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner
file:/opt/txd-pyspark-siebel-casos.py 2021-12-18
s3a://mybucket.raw/siebel/casos/
21/12/21 18:37:43 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/12/21 18:37:45 INFO SparkContext: Running Spark version 2.4.4
21/12/21 18:37:45 INFO SparkContext: Submitted application:
TXD-PYSPARK-ORACLE-SIEBEL-CASOS
21/12/21 18:37:45 INFO SecurityManager: Changing view acls to: root
21/12/21 18:37:45 INFO SecurityManager: Changing modify acls to: root
21/12/21 18:37:45 INFO SecurityManager: Changing view acls groups to:
21/12/21 18:37:45 INFO SecurityManager: Changing modify acls groups to:
21/12/21 18:37:45 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); groups
with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()
21/12/21 18:37:45 INFO Utils: Successfully started service 'sparkDriver' on
port 7078.
21/12/21 18:37:45 INFO SparkEnv: Registering MapOutputTracker
21/12/21 18:37:45 INFO SparkEnv: Registering BlockManagerMaster
21/12/21 18:37:45 INFO BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/12/21 18:37:45 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/12/21 18:37:45 INFO DiskBlockManager: Created local directory at
/var/data/spark-458585a1-50f9-45c6-a4cf-d552c04a97dc/blockmgr-6c240735-3731-487a-a592-5c9a4d687020
21/12/21 18:37:45 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
21/12/21 18:37:45 INFO SparkEnv: Registering OutputCommitCoordinator
21/12/21 18:37:46 INFO Utils: Successfully started service 'SparkUI' on port
4040.
21/12/21 18:37:46 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://spark-siebel-casos-1640111855179-driver-svc.spark.svc:4040
21/12/21 18:37:46 INFO SparkContext: Added JAR file:///opt/ojdbc8.jar at
spark://spark-siebel-casos-1640111855179-driver-svc.spark.svc:7078/jars/ojdbc8.jar
with timestamp 1640111866249
21/12/21 18:37:46 INFO SparkContext: Added JAR
file:///opt/aws-java-sdk-bundle-1.11.901.jar at
spark://spark-siebel-casos-1640111855179-driver-svc.spark.svc:7078/jars/aws-java-sdk-bundle-1.11.901.jar
with timestamp 1640111866249
21/12/21 18:37:46 INFO SparkContext: Added JAR file:///opt/hadoop-aws-3.3.1.jar
at
spark://spark-siebel-casos-1640111855179-driver-svc.spark.svc:7078/jars/hadoop-aws-3.3.1.jar
with timestamp 1640111866249
21/12/21 18:37:46 INFO SparkContext: Added file
file:///opt/txd-pyspark-siebel-casos.py at
spark://spark-siebel-casos-1640111855179-driver-svc.spark.svc:7078/files/txd-pyspark-siebel-casos.py
with timestamp 1640111866266
21/12/21 18:37:46 INFO Utils: Copying /opt/txd-pyspark-siebel-casos.py to
/var/data/spark-458585a1-50f9-45c6-a4cf-d552c04a97dc/spark-f99cee68-d203-4a2a-8335-9743eeac5350/userFiles-32cbe539-22db-4547-8d22-d98f85354418/txd-pyspark-siebel-casos.py
21/12/21 18:37:48 INFO ExecutorPodsAllocator: Going to request 2 executors from
Kubernetes.
21/12/21 18:37:48 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
21/12/21 18:37:48 INFO NettyBlockTransferService: Server created on
spark-siebel-casos-1640111855179-driver-svc.spark.svc:7079
21/12/21 18:37:48 INFO BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
21/12/21 18:37:48 INFO BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, spark-siebel-casos-1640111855179-driver-svc.spark.svc,
7079, None)
21/12/21 18:37:48 INFO BlockManagerMasterEndpoint: Registering block manager
spark-siebel-casos-1640111855179-driver-svc.spark.svc:7079 with 413.9 MB RAM,
BlockManagerId(driver, spark-siebel-casos-1640111855179-driver-svc.spark.svc,
7079, None)
21/12/21 18:37:48 INFO BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, spark-siebel-casos-1640111855179-driver-svc.spark.svc,
7079, None)
21/12/21 18:37:48 INFO BlockManager: Initialized BlockManager:
BlockManagerId(driver, spark-siebel-casos-1640111855179-driver-svc.spark.svc,
7079, None)
21/12/21 18:37:53 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor
NettyRpcEndpointRef(spark-client://Executor) (10.3.170.156:58300) with ID 1
21/12/21 18:37:53 INFO BlockManagerMasterEndpoint: Registering block manager
10.3.170.156:34671 with 413.9 MB RAM, BlockManagerId(1, 10.3.170.156, 34671,
None)
21/12/21 18:37:54 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor
NettyRpcEndpointRef(spark-client://Executor) (10.3.170.184:52960) with ID 2
21/12/21 18:37:54 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is
ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
21/12/21 18:37:54 INFO SharedState: Setting hive.metastore.warehouse.dir
('null') to the value of spark.sql.warehouse.dir
('file:/opt/spark/work-dir/spark-warehouse').
21/12/21 18:37:54 INFO SharedState: Warehouse path is
'file:/opt/spark/work-dir/spark-warehouse'.
21/12/21 18:37:54 INFO BlockManagerMasterEndpoint: Registering block manager
10.3.170.184:46293 with 413.9 MB RAM, BlockManagerId(2, 10.3.170.184, 46293,
None)
21/12/21 18:37:54 INFO StateStoreCoordinatorRef: Registered
StateStoreCoordinator endpoint
['/opt/txd-pyspark-siebel-casos.py', '2021-12-18',
's3a://mybucket.raw/siebel/casos/'] load_date 2021-12-18 output_path
s3a://mybucket.raw/siebel/casos/
Traceback (most recent call last):
File "/opt/txd-pyspark-siebel-casos.py", line 68, in <module>
run_etl()
File "/opt/txd-pyspark-siebel-casos.py", line 60, in run_etl
df.write.mode("append").partitionBy(["year", "month",
"day"]).csv(output_path, header=True)
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 931,
in csv
File "/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line
1257, in __call__
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in
deco
File "/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328,
in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o83.csv.
: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any
provider in the chain
at
com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at
org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:424)
at
org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:524)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:290)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:664)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)21/12/21 18:38:00 INFO
SparkContext: Invoking stop() from shutdown hook
21/12/21 18:38:00 INFO SparkUI: Stopped Spark web UI at
http://spark-siebel-casos-1640111855179-driver-svc.spark.svc:4040
21/12/21 18:38:00 INFO KubernetesClusterSchedulerBackend: Shutting down all
executors
21/12/21 18:38:00 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
executor to shut down
21/12/21 18:38:00 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has
been closed (this is expected if the application is shutting down.)
21/12/21 18:38:01 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
21/12/21 18:38:01 INFO MemoryStore: MemoryStore cleared
21/12/21 18:38:01 INFO BlockManager: BlockManager stopped
21/12/21 18:38:01 INFO BlockManagerMaster: BlockManagerMaster stopped
21/12/21 18:38:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
21/12/21 18:38:01 INFO SparkContext: Successfully stopped SparkContext
21/12/21 18:38:01 INFO ShutdownHookManager: Shutdown hook called
21/12/21 18:38:01 INFO ShutdownHookManager: Deleting directory
/var/data/spark-458585a1-50f9-45c6-a4cf-d552c04a97dc/spark-f99cee68-d203-4a2a-8335-9743eeac5350
21/12/21 18:38:01 INFO ShutdownHookManager: Deleting directory
/tmp/spark-245a2532-04c0-4309-9f43-00fbca06d435
21/12/21 18:38:01 INFO ShutdownHookManager: Deleting directory
/var/data/spark-458585a1-50f9-45c6-a4cf-d552c04a97dc/spark-f99cee68-d203-4a2a-8335-9743eeac5350/pyspark-83a9a9c2-0b44-4bcf-8d86-6fb388ba275e
{code}
Pod describe:
{code:java}
Containers:
spark-kubernetes-driver:
Container ID:
docker://3606841142e7dc76f3a5b29f7df87da8159a0d0c53897f96444670a04134a2ff
Image: registry.example.com/myorg/ata/spark/spark-k8s/spark-py:2.4.4
Image ID:
docker-pullable://registry.example.com/myorg/ata/spark/spark-k8s/spark-py@sha256:744eae637693e0c6f2195ed1e4e2bab9def5b9c7507518c5d4b61b7933c63e10
Ports: 7078/TCP, 7079/TCP, 4040/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
driver-py
--properties-file
/opt/spark/conf/spark.properties
--class
org.apache.spark.deploy.PythonRunner
State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 21 Dec 2021 12:44:59 -0300
Finished: Tue, 21 Dec 2021 12:45:29 -0300
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 1433Mi
Requests:
cpu: 1
memory: 1433Mi
Environment:
JOB_PATH:
s3://mybucket.spark.jobs/siebel-casos-actividades
SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
SPARK_LOCAL_DIRS:
/var/data/spark-9ff6233c-1660-4be4-94b7-2e961412f958
PYSPARK_PRIMARY: file:/opt/txd-pyspark-siebel-casos.py
PYSPARK_MAJOR_PYTHON_VERSION: 3
PYSPARK_APP_ARGS: 2021-12-18 s3a://mybucket.raw/siebel/casos/
PYSPARK_FILES:
SPARK_CONF_DIR: /opt/spark/conf
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN:
arn:aws:iam::999999999999:role/EKSDefaultPolicyFor-Spark
AWS_WEB_IDENTITY_TOKEN_FILE:
/var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/opt/spark/conf from spark-conf-volume (rw)
/var/data/spark-9ff6233c-1660-4be4-94b7-2e961412f958 from
spark-local-dir-1 (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from spark-token-r6p46 (ro)
{code}
Reporter: Esteban Avendaño
Hello everybody, please help with this issue. I have a job running with spark
over kubernetes (AWS EKS) and I get this error:
{code:java}
py4j.protocol.Py4JJavaError: An error occurred while calling o83.csv.
: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any
provider in the chain
at
com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
