Hi, someone can help I'm trying to use parquet in IBM Block Storage at Spark but when I try to load get this error:
using this config credentials = { "name": "keystone", *"auth_url": "https://identity.open.softlayer.com <https://identity.open.softlayer.com>",* "project": "object_storage_23f274c1_d11XXXXXXXXXXXXXXXe634", "projectId": "XXXXXXd9c4aa39b7c7eCCCCCCCCb", "region": "dallas", "userId": "XXXXX64087180b40XXXXX2b909", "username": "admin_XXXX9dd810f8901d48778XXXXXX", "password": "chXXXXXXXXXXXXX6_", "domainId": "c1ddad17cfcXXXXXXXXX41", "domainName": "10XXXXXX", "role": "admin" } def set_hadoop_config(credentials): """This function sets the Hadoop configuration with given credentials, so it is possible to access data using SparkContext""" prefix = "fs.swift.service." + credentials['name'] hconf = sc._jsc.hadoopConfiguration() *hconf.set(prefix + ".auth.url", credentials['auth_url']+'/v3/auth/tokens')* hconf.set(prefix + ".auth.endpoint.prefix", "endpoints") hconf.set(prefix + ".tenant", credentials['projectId']) hconf.set(prefix + ".username", credentials['userId']) hconf.set(prefix + ".password", credentials['password']) hconf.setInt(prefix + ".http.port", 8080) hconf.set(prefix + ".region", credentials['region']) hconf.setBoolean(prefix + ".public", True) set_hadoop_config(credentials) ------------------------------------------------- Py4JJavaErrorTraceback (most recent call last) <ipython-input-55-5a14928215eb> in <module>() ----> 1 train.groupby('Acordo').count().show() *Py4JJavaError: An error occurred while calling* o406.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 60 in stage 30.0 failed 10 times, most recent failure: Lost task 60.9 in stage 30.0 (TID 2556, yp-spark-dal09-env5-0039): org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException:* Missing mandatory configuration option: fs.swift.service.keystone.auth.url* at org.apache.hadoop.fs.swift.http.RestClientBindings.copy(RestClientBindings.java:223) at org.apache.hadoop.fs.swift.http.RestClientBindings.bind(RestClientBindings.java:147) *Daniel Lopes* Chief Data and Analytics Officer | OneMatch c: +55 (18) 99764-2733 | https://www.linkedin.com/in/dslopes www.onematch.com.br <http://www.onematch.com.br/?utm_source=EmailSignature&utm_term=daniel-lopes>