soumilshah1995 commented on issue #11391: URL: https://github.com/apache/hudi/issues/11391#issuecomment-2149868601
Added following packages ``` HUDI_VERSION = '0.14.0' SPARK_VERSION = '3.4' os.environ["JAVA_HOME"] = "/opt/homebrew/opt/openjdk@11" SUBMIT_ARGS = f"--packages org.apache.hudi:hudi-spark{SPARK_VERSION}-bundle_2.12:{HUDI_VERSION},com.amazonaws:dynamodb-lock-client:1.2.0,com.amazonaws:aws-java-sdk-dynamodb:1.12.735,com.amazonaws:aws-java-sdk-core:1.12.735,org.apache.hudi:hudi-aws-bundle:{HUDI_VERSION},org.apache.hudi:hudi-aws:{HUDI_VERSION} pyspark-shell" os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS os.environ['PYSPARK_PYTHON'] = sys.executable spark = SparkSession.builder \ .config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \ .config('spark.sql.extensions', 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension') \ .config('className', 'org.apache.hudi') \ .config('spark.sql.hive.convertMetastoreParquet', 'false') \ .getOrCreate() ``` # Error : org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider ``` g.apache.hudi#hudi-aws-bundle added as a dependency org.apache.hudi#hudi-aws added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-aa8d9c29-7056-4201-b20a-c5f73fac7ea9;1.0 confs: [default] found org.apache.hudi#hudi-spark3.4-bundle_2.12;0.14.0 in spark-list found com.amazonaws#dynamodb-lock-client;1.2.0 in central found software.amazon.awssdk#dynamodb;2.20.8 in central found software.amazon.awssdk#aws-json-protocol;2.20.8 in central found software.amazon.awssdk#aws-core;2.20.8 in central found software.amazon.awssdk#annotations;2.20.8 in central found software.amazon.awssdk#regions;2.20.8 in central found software.amazon.awssdk#utils;2.20.8 in central found org.reactivestreams#reactive-streams;1.0.2 in central found org.slf4j#slf4j-api;1.7.30 in local-m2-cache found software.amazon.awssdk#sdk-core;2.20.8 in central found software.amazon.awssdk#http-client-spi;2.20.8 in central found software.amazon.awssdk#metrics-spi;2.20.8 in central found software.amazon.awssdk#endpoints-spi;2.20.8 in central found software.amazon.awssdk#profiles;2.20.8 in central found software.amazon.awssdk#json-utils;2.20.8 in central found software.amazon.awssdk#third-party-jackson-core;2.20.8 in central found software.amazon.awssdk#auth;2.20.8 in central found software.amazon.eventstream#eventstream;1.0.1 in central found software.amazon.awssdk#protocol-core;2.20.8 in central found software.amazon.awssdk#apache-client;2.20.8 in central found org.apache.httpcomponents#httpclient;4.5.13 in local-m2-cache found org.apache.httpcomponents#httpcore;4.4.13 in local-m2-cache found commons-logging#commons-logging;1.2 in local-m2-cache found software.amazon.awssdk#netty-nio-client;2.20.8 in central found io.netty#netty-codec-http2;4.1.86.Final in central found io.netty#netty-common;4.1.86.Final in central found io.netty#netty-buffer;4.1.86.Final in central found io.netty#netty-transport;4.1.86.Final in central found io.netty#netty-resolver;4.1.86.Final in central found io.netty#netty-codec;4.1.86.Final in central found io.netty#netty-transport-classes-epoll;4.1.86.Final in central found io.netty#netty-transport-native-unix-common;4.1.86.Final in central found com.amazonaws#aws-java-sdk-dynamodb;1.12.735 in central found com.amazonaws#aws-java-sdk-s3;1.12.735 in central found com.amazonaws#aws-java-sdk-kms;1.12.735 in central found com.amazonaws#aws-java-sdk-core;1.12.735 in central found commons-codec#commons-codec;1.15 in local-m2-cache found com.fasterxml.jackson.core#jackson-databind;2.12.7.2 in central found com.fasterxml.jackson.core#jackson-annotations;2.12.7 in local-m2-cache found com.fasterxml.jackson.core#jackson-core;2.12.7 in local-m2-cache found com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.12.6 in central found joda-time#joda-time;2.12.7 in central found com.amazonaws#jmespath-java;1.12.735 in central found org.apache.hudi#hudi-aws-bundle;0.14.0 in central found org.apache.hudi#hudi-common;0.14.0 in central found org.openjdk.jol#jol-core;0.16 in local-m2-cache found com.fasterxml.jackson.datatype#jackson-datatype-jsr310;2.10.0 in local-m2-cache found com.github.ben-manes.caffeine#caffeine;2.9.1 in local-m2-cache found org.checkerframework#checker-qual;3.10.0 in local-m2-cache found com.google.errorprone#error_prone_annotations;2.5.1 in local-m2-cache found org.apache.orc#orc-core;1.6.0 in local-m2-cache found org.apache.orc#orc-shims;1.6.0 in local-m2-cache found org.slf4j#slf4j-api;1.7.36 in local-m2-cache found com.google.protobuf#protobuf-java;3.21.7 in local-m2-cache found commons-lang#commons-lang;2.6 in local-m2-cache found io.airlift#aircompressor;0.15 in local-m2-cache found javax.xml.bind#jaxb-api;2.2.11 in local-m2-cache found org.apache.hive#hive-storage-api;2.6.0 in local-m2-cache found org.jetbrains#annotations;17.0.0 in local-m2-cache found org.roaringbitmap#RoaringBitmap;0.9.47 in local-m2-cache found org.apache.httpcomponents#fluent-hc;4.4.1 in local-m2-cache found org.rocksdb#rocksdbjni;7.5.3 in local-m2-cache found org.apache.hbase#hbase-client;2.4.9 in local-m2-cache found org.apache.hbase.thirdparty#hbase-shaded-protobuf;3.5.1 in local-m2-cache found org.apache.hbase#hbase-protocol-shaded;2.4.9 in local-m2-cache found org.apache.yetus#audience-annotations;0.5.0 in local-m2-cache found org.apache.hbase#hbase-protocol;2.4.9 in local-m2-cache found javax.annotation#javax.annotation-api;1.2 in local-m2-cache found commons-io#commons-io;2.11.0 in local-m2-cache found org.apache.commons#commons-lang3;3.9 in local-m2-cache found org.apache.hbase.thirdparty#hbase-shaded-miscellaneous;3.5.1 in local-m2-cache found com.google.errorprone#error_prone_annotations;2.7.1 in local-m2-cache found org.apache.hbase.thirdparty#hbase-shaded-netty;3.5.1 in local-m2-cache found org.apache.zookeeper#zookeeper;3.5.7 in local-m2-cache found org.apache.zookeeper#zookeeper-jute;3.5.7 in local-m2-cache found io.netty#netty-handler;4.1.45.Final in local-m2-cache found io.netty#netty-transport-native-epoll;4.1.45.Final in local-m2-cache found org.apache.htrace#htrace-core4;4.2.0-incubating in local-m2-cache found org.jruby.jcodings#jcodings;1.0.55 in local-m2-cache found org.jruby.joni#joni;2.1.31 in local-m2-cache found io.dropwizard.metrics#metrics-core;4.1.1 in local-m2-cache found org.apache.commons#commons-crypto;1.0.0 in local-m2-cache found org.apache.hbase#hbase-server;2.4.9 in local-m2-cache found org.apache.hbase#hbase-procedure;2.4.9 in local-m2-cache found org.apache.hbase#hbase-replication;2.4.9 in local-m2-cache found org.glassfish.web#javax.servlet.jsp;2.3.2 in local-m2-cache found org.glassfish#javax.el;3.0.1-b12 in local-m2-cache found javax.servlet.jsp#javax.servlet.jsp-api;2.3.1 in local-m2-cache found org.apache.commons#commons-math3;3.6.1 in local-m2-cache found org.jamon#jamon-runtime;2.4.1 in local-m2-cache found com.lmax#disruptor;3.4.2 in local-m2-cache found org.lz4#lz4-java;1.8.0 in local-m2-cache found org.roaringbitmap#shims;0.9.47 in local-m2-cache found org.apache.hudi#hudi-hive-sync;0.14.0 in central found org.apache.hudi#hudi-hadoop-mr;0.14.0 in central found org.apache.hudi#hudi-sync-common;0.14.0 in central found com.beust#jcommander;1.78 in central found org.apache.hudi#hudi-aws;0.14.0 in central found software.amazon.awssdk#cloudwatch;2.18.40 in central found software.amazon.awssdk#aws-query-protocol;2.18.40 in central found software.amazon.awssdk#glue;2.18.40 in central found software.amazon.awssdk#sqs;2.18.40 in central found org.apache.parquet#parquet-avro;1.10.1 in local-m2-cache found org.apache.parquet#parquet-column;1.10.1 in local-m2-cache found org.apache.parquet#parquet-common;1.10.1 in local-m2-cache found org.apache.parquet#parquet-format;2.4.0 in local-m2-cache found org.apache.parquet#parquet-encoding;1.10.1 in local-m2-cache found org.apache.parquet#parquet-hadoop;1.10.1 in local-m2-cache found org.apache.parquet#parquet-jackson;1.10.1 in local-m2-cache found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in local-m2-cache found org.codehaus.jackson#jackson-core-asl;1.9.13 in local-m2-cache found org.xerial.snappy#snappy-java;1.1.8.3 in local-m2-cache found commons-pool#commons-pool;1.6 in local-m2-cache found org.apache.avro#avro;1.8.2 in local-m2-cache found com.thoughtworks.paranamer#paranamer;2.7 in local-m2-cache found org.apache.commons#commons-compress;1.8.1 in local-m2-cache found org.tukaani#xz;1.5 in local-m2-cache found it.unimi.dsi#fastutil;7.0.13 in local-m2-cache :: resolution report :: resolve 734ms :: artifacts dl 22ms :: modules in use: com.amazonaws#aws-java-sdk-core;1.12.735 from central in [default] com.amazonaws#aws-java-sdk-dynamodb;1.12.735 from central in [default] com.amazonaws#aws-java-sdk-kms;1.12.735 from central in [default] com.amazonaws#aws-java-sdk-s3;1.12.735 from central in [default] com.amazonaws#dynamodb-lock-client;1.2.0 from central in [default] com.amazonaws#jmespath-java;1.12.735 from central in [default] com.beust#jcommander;1.78 from central in [default] com.fasterxml.jackson.core#jackson-annotations;2.12.7 from local-m2-cache in [default] com.fasterxml.jackson.core#jackson-core;2.12.7 from local-m2-cache in [default] com.fasterxml.jackson.core#jackson-databind;2.12.7.2 from central in [default] com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.12.6 from central in [default] com.fasterxml.jackson.datatype#jackson-datatype-jsr310;2.10.0 from local-m2-cache in [default] com.github.ben-manes.caffeine#caffeine;2.9.1 from local-m2-cache in [default] com.google.errorprone#error_prone_annotations;2.7.1 from local-m2-cache in [default] com.google.protobuf#protobuf-java;3.21.7 from local-m2-cache in [default] com.lmax#disruptor;3.4.2 from local-m2-cache in [default] com.thoughtworks.paranamer#paranamer;2.7 from local-m2-cache in [default] commons-codec#commons-codec;1.15 from local-m2-cache in [default] commons-io#commons-io;2.11.0 from local-m2-cache in [default] commons-lang#commons-lang;2.6 from local-m2-cache in [default] commons-logging#commons-logging;1.2 from local-m2-cache in [default] commons-pool#commons-pool;1.6 from local-m2-cache in [default] io.airlift#aircompressor;0.15 from local-m2-cache in [default] io.dropwizard.metrics#metrics-core;4.1.1 from local-m2-cache in [default] io.netty#netty-buffer;4.1.86.Final from central in [default] io.netty#netty-codec;4.1.86.Final from central in [default] io.netty#netty-codec-http2;4.1.86.Final from central in [default] io.netty#netty-common;4.1.86.Final from central in [default] io.netty#netty-handler;4.1.45.Final from local-m2-cache in [default] io.netty#netty-resolver;4.1.86.Final from central in [default] io.netty#netty-transport;4.1.86.Final from central in [default] io.netty#netty-transport-classes-epoll;4.1.86.Final from central in [default] io.netty#netty-transport-native-epoll;4.1.45.Final from local-m2-cache in [default] io.netty#netty-transport-native-unix-common;4.1.86.Final from central in [default] it.unimi.dsi#fastutil;7.0.13 from local-m2-cache in [default] javax.annotation#javax.annotation-api;1.2 from local-m2-cache in [default] javax.servlet.jsp#javax.servlet.jsp-api;2.3.1 from local-m2-cache in [default] javax.xml.bind#jaxb-api;2.2.11 from local-m2-cache in [default] joda-time#joda-time;2.12.7 from central in [default] org.apache.avro#avro;1.8.2 from local-m2-cache in [default] org.apache.commons#commons-compress;1.8.1 from local-m2-cache in [default] org.apache.commons#commons-crypto;1.0.0 from local-m2-cache in [default] org.apache.commons#commons-lang3;3.9 from local-m2-cache in [default] org.apache.commons#commons-math3;3.6.1 from local-m2-cache in [default] org.apache.hbase#hbase-client;2.4.9 from local-m2-cache in [default] org.apache.hbase#hbase-procedure;2.4.9 from local-m2-cache in [default] org.apache.hbase#hbase-protocol;2.4.9 from local-m2-cache in [default] org.apache.hbase#hbase-protocol-shaded;2.4.9 from local-m2-cache in [default] org.apache.hbase#hbase-replication;2.4.9 from local-m2-cache in [default] org.apache.hbase#hbase-server;2.4.9 from local-m2-cache in [default] org.apache.hbase.thirdparty#hbase-shaded-miscellaneous;3.5.1 from local-m2-cache in [default] org.apache.hbase.thirdparty#hbase-shaded-netty;3.5.1 from local-m2-cache in [default] org.apache.hbase.thirdparty#hbase-shaded-protobuf;3.5.1 from local-m2-cache in [default] org.apache.hive#hive-storage-api;2.6.0 from local-m2-cache in [default] org.apache.htrace#htrace-core4;4.2.0-incubating from local-m2-cache in [default] org.apache.httpcomponents#fluent-hc;4.4.1 from local-m2-cache in [default] org.apache.httpcomponents#httpclient;4.5.13 from local-m2-cache in [default] org.apache.httpcomponents#httpcore;4.4.13 from local-m2-cache in [default] org.apache.hudi#hudi-aws;0.14.0 from central in [default] org.apache.hudi#hudi-aws-bundle;0.14.0 from central in [default] org.apache.hudi#hudi-common;0.14.0 from central in [default] org.apache.hudi#hudi-hadoop-mr;0.14.0 from central in [default] org.apache.hudi#hudi-hive-sync;0.14.0 from central in [default] org.apache.hudi#hudi-spark3.4-bundle_2.12;0.14.0 from spark-list in [default] org.apache.hudi#hudi-sync-common;0.14.0 from central in [default] org.apache.orc#orc-core;1.6.0 from local-m2-cache in [default] org.apache.orc#orc-shims;1.6.0 from local-m2-cache in [default] org.apache.parquet#parquet-avro;1.10.1 from local-m2-cache in [default] org.apache.parquet#parquet-column;1.10.1 from local-m2-cache in [default] org.apache.parquet#parquet-common;1.10.1 from local-m2-cache in [default] org.apache.parquet#parquet-encoding;1.10.1 from local-m2-cache in [default] org.apache.parquet#parquet-format;2.4.0 from local-m2-cache in [default] org.apache.parquet#parquet-hadoop;1.10.1 from local-m2-cache in [default] org.apache.parquet#parquet-jackson;1.10.1 from local-m2-cache in [default] org.apache.yetus#audience-annotations;0.5.0 from local-m2-cache in [default] org.apache.zookeeper#zookeeper;3.5.7 from local-m2-cache in [default] org.apache.zookeeper#zookeeper-jute;3.5.7 from local-m2-cache in [default] org.checkerframework#checker-qual;3.10.0 from local-m2-cache in [default] org.codehaus.jackson#jackson-core-asl;1.9.13 from local-m2-cache in [default] org.codehaus.jackson#jackson-mapper-asl;1.9.13 from local-m2-cache in [default] org.glassfish#javax.el;3.0.1-b12 from local-m2-cache in [default] org.glassfish.web#javax.servlet.jsp;2.3.2 from local-m2-cache in [default] org.jamon#jamon-runtime;2.4.1 from local-m2-cache in [default] org.jetbrains#annotations;17.0.0 from local-m2-cache in [default] org.jruby.jcodings#jcodings;1.0.55 from local-m2-cache in [default] org.jruby.joni#joni;2.1.31 from local-m2-cache in [default] org.lz4#lz4-java;1.8.0 from local-m2-cache in [default] org.openjdk.jol#jol-core;0.16 from local-m2-cache in [default] org.reactivestreams#reactive-streams;1.0.2 from central in [default] org.roaringbitmap#RoaringBitmap;0.9.47 from local-m2-cache in [default] org.roaringbitmap#shims;0.9.47 from local-m2-cache in [default] org.rocksdb#rocksdbjni;7.5.3 from local-m2-cache in [default] org.slf4j#slf4j-api;1.7.36 from local-m2-cache in [default] org.tukaani#xz;1.5 from local-m2-cache in [default] org.xerial.snappy#snappy-java;1.1.8.3 from local-m2-cache in [default] software.amazon.awssdk#annotations;2.20.8 from central in [default] software.amazon.awssdk#apache-client;2.20.8 from central in [default] software.amazon.awssdk#auth;2.20.8 from central in [default] software.amazon.awssdk#aws-core;2.20.8 from central in [default] software.amazon.awssdk#aws-json-protocol;2.20.8 from central in [default] software.amazon.awssdk#aws-query-protocol;2.18.40 from central in [default] software.amazon.awssdk#cloudwatch;2.18.40 from central in [default] software.amazon.awssdk#dynamodb;2.20.8 from central in [default] software.amazon.awssdk#endpoints-spi;2.20.8 from central in [default] software.amazon.awssdk#glue;2.18.40 from central in [default] software.amazon.awssdk#http-client-spi;2.20.8 from central in [default] software.amazon.awssdk#json-utils;2.20.8 from central in [default] software.amazon.awssdk#metrics-spi;2.20.8 from central in [default] software.amazon.awssdk#netty-nio-client;2.20.8 from central in [default] software.amazon.awssdk#profiles;2.20.8 from central in [default] software.amazon.awssdk#protocol-core;2.20.8 from central in [default] software.amazon.awssdk#regions;2.20.8 from central in [default] software.amazon.awssdk#sdk-core;2.20.8 from central in [default] software.amazon.awssdk#sqs;2.18.40 from central in [default] software.amazon.awssdk#third-party-jackson-core;2.20.8 from central in [default] software.amazon.awssdk#utils;2.20.8 from central in [default] software.amazon.eventstream#eventstream;1.0.1 from central in [default] :: evicted modules: org.slf4j#slf4j-api;1.7.30 by [org.slf4j#slf4j-api;1.7.36] in [default] commons-logging#commons-logging;1.1.3 by [commons-logging#commons-logging;1.2] in [default] com.fasterxml.jackson.core#jackson-databind;2.12.6 by [com.fasterxml.jackson.core#jackson-databind;2.12.7.2] in [default] com.fasterxml.jackson.core#jackson-core;2.12.6 by [com.fasterxml.jackson.core#jackson-core;2.12.7] in [default] com.fasterxml.jackson.core#jackson-annotations;2.10.0 by [com.fasterxml.jackson.core#jackson-annotations;2.12.7] in [default] com.fasterxml.jackson.core#jackson-databind;2.10.0 by [com.fasterxml.jackson.core#jackson-databind;2.12.7.2] in [default] com.fasterxml.jackson.core#jackson-core;2.10.0 by [com.fasterxml.jackson.core#jackson-core;2.12.7] in [default] com.google.errorprone#error_prone_annotations;2.5.1 by [com.google.errorprone#error_prone_annotations;2.7.1] in [default] org.apache.httpcomponents#httpclient;4.4.1 by [org.apache.httpcomponents#httpclient;4.5.13] in [default] commons-codec#commons-codec;1.13 by [commons-codec#commons-codec;1.10] in [default] io.netty#netty-common;4.1.45.Final by [io.netty#netty-common;4.1.86.Final] in [default] io.netty#netty-buffer;4.1.45.Final by [io.netty#netty-buffer;4.1.86.Final] in [default] io.netty#netty-transport;4.1.45.Final by [io.netty#netty-transport;4.1.86.Final] in [default] io.netty#netty-codec;4.1.45.Final by [io.netty#netty-codec;4.1.86.Final] in [default] io.netty#netty-transport-native-unix-common;4.1.45.Final by [io.netty#netty-transport-native-unix-common;4.1.86.Final] in [default] software.amazon.awssdk#protocol-core;2.18.40 by [software.amazon.awssdk#protocol-core;2.20.8] in [default] software.amazon.awssdk#aws-core;2.18.40 by [software.amazon.awssdk#aws-core;2.20.8] in [default] software.amazon.awssdk#sdk-core;2.18.40 by [software.amazon.awssdk#sdk-core;2.20.8] in [default] software.amazon.awssdk#annotations;2.18.40 by [software.amazon.awssdk#annotations;2.20.8] in [default] software.amazon.awssdk#http-client-spi;2.18.40 by [software.amazon.awssdk#http-client-spi;2.20.8] in [default] software.amazon.awssdk#utils;2.18.40 by [software.amazon.awssdk#utils;2.20.8] in [default] software.amazon.awssdk#auth;2.18.40 by [software.amazon.awssdk#auth;2.20.8] in [default] software.amazon.awssdk#regions;2.18.40 by [software.amazon.awssdk#regions;2.20.8] in [default] software.amazon.awssdk#metrics-spi;2.18.40 by [software.amazon.awssdk#metrics-spi;2.20.8] in [default] software.amazon.awssdk#json-utils;2.18.40 by [software.amazon.awssdk#json-utils;2.20.8] in [default] software.amazon.awssdk#endpoints-spi;2.18.40 by [software.amazon.awssdk#endpoints-spi;2.20.8] in [default] software.amazon.awssdk#dynamodb;2.18.40 by [software.amazon.awssdk#dynamodb;2.20.8] in [default] software.amazon.awssdk#aws-json-protocol;2.18.40 by [software.amazon.awssdk#aws-json-protocol;2.20.8] in [default] org.apache.httpcomponents#httpcore;4.4.1 by [org.apache.httpcomponents#httpcore;4.4.13] in [default] software.amazon.awssdk#apache-client;2.18.40 by [software.amazon.awssdk#apache-client;2.20.8] in [default] software.amazon.awssdk#netty-nio-client;2.18.40 by [software.amazon.awssdk#netty-nio-client;2.20.8] in [default] commons-codec#commons-codec;1.10 by [commons-codec#commons-codec;1.15] in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 149 | 0 | 0 | 32 || 117 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent-aa8d9c29-7056-4201-b20a-c5f73fac7ea9 confs: [default] 0 artifacts copied, 117 already retrieved (0kB/10ms) 24/06/05 09:14:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). +--------+----------+----------+----------+-----------+-----+ | orderID|productSKU|customerID| orderDate|orderAmount|state| +--------+----------+----------+----------+-----------+-----+ | order1| prod001| cust001|2024-01-15| 150.0| CA| |order002| prod002| cust002|2024-01-16| 200.0| NY| |order003| prod003| cust003|2024-01-17| 300.0| TX| |order004| prod004| cust004|2024-01-18| 250.0| FL| |order005| prod005| cust005|2024-01-19| 100.0| WA| |order006| prod006| cust006|2024-01-20| 350.0| CA| |order007| prod007| cust007|2024-01-21| 400.0| NY| +--------+----------+----------+----------+-----------+-----+ {'hoodie.table.name': 'orders', 'hoodie.datasource.write.table.type': 'COPY_ON_WRITE', 'hoodie.datasource.write.table.name': 'orders', 'hoodie.datasource.write.operation': 'upsert', 'hoodie.datasource.write.recordkey.field': 'orderID', 'hoodie.datasource.write.precombine.field': 'orderDate', 'hoodie.datasource.write.partitionpath.field': 'state', 'hoodie.write.concurrency.mode': 'optimistic_concurrency_control', 'hoodie.cleaner.policy.failed.writes': 'LAZY', 'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider', 'hoodie.write.lock.dynamodb.table': 'hudi-lock-table', 'hoodie.write.lock.dynamodb.region': 'us-east-1', 'hoodie.write.lock.dynamodb.endpoint_url': 'dynamodb.us-east-1.amazonaws.com', 'hoodie.write.lock.dynamodb.billing_mode': 'PAY_PER_REQUEST'} file:///Users/soumilshah/IdeaProjects/SparkProject/tem/database=default/table_name=orders 24/06/05 09:14:12 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf 24/06/05 09:14:12 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file Traceback (most recent call last): File "/Users/soumilshah/IdeaProjects/SparkProject/deltastreamerBroadcastJoins/conflictdetection/w1.py", line 90, in <module> write_to_hudi( File "/Users/soumilshah/IdeaProjects/SparkProject/deltastreamerBroadcastJoins/conflictdetection/w1.py", line 88, in write_to_hudi .save(path) ^^^^^^^^^^ File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/sql/readwriter.py", line 1398, in save self._jwrite.save(path) File "/opt/anaconda3/lib/python3.11/site-packages/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( ^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py", line 169, in deco return f(*a, **kw) ^^^^^^^^^^^ File "/opt/anaconda3/lib/python3.11/site-packages/py4j/protocol.py", line 326, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o65.save. : org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:81) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:129) at org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:118) at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:71) at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:58) at org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1253) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1296) at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139) at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224) at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431) at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488) at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94) at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81) at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79) at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:133) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) ... 52 more Caused by: java.lang.NoSuchMethodError: 'com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions$AmazonDynamoDBLockClientOptionsBuilder com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions.builder(org.apache.hudi.software.amazon.awssdk.services.dynamodb.DynamoDbClient, java.lang.String)' at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:90) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:75) ... 57 more (base) soumilshah@Soumils-MBP conflictdetection % ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org