harishchanderramesh commented on issue #1936: URL: https://github.com/apache/hudi/issues/1936#issuecomment-671154630
Yes, I pass this `spark.sql.hive.convertMetastoreParquet=false`. Here is my spark submit command, if that may help. ``` export PYTHONIOENCODING=utf8 export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop/conf export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/conf/ export SPARK_HOME=/usr/lib/spark/ spark-submit \ --name "SPARK: KFK-HDFS-HUDI-TO-ATHENA" \ --deploy-mode cluster \ --master yarn \ --py-files "s3://bjnbi-emr-bootstrap/getDetails_From_Secrets_Manager.py,s3://bjnbi-emr-bootstrap/kafka_metadata_connection.py,s3://bjnbi-emr-bootstrap/postToslack.py,s3://bjnbi-emr-bootstrap/send_email.py,s3://bjnbi-emr-bootstrap/boto3.zip,s3://bjnbi-emr-bootstrap/connector.zip,s3://bjnbi-emr-bootstrap/pandas.zip,s3://bjnbi-emr-bootstrap/pymysql.zip,s3://bjnbi-emr-bootstrap/pymysqlreplication.zip" \ --files "/home/hadoop/customlog4j.properties,/etc/hive/conf/hive-site.xml" \ --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=customlog4j.properties" \ --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=customlog4j.properties" \ --conf "spark.yarn.submit.waitAppCompletion=true" \ --conf "spark.dynamicAllocation.enabled=true" \ --conf "spark.dynamicAllocation.minExecutors=1" \ --conf "spark.dynamicAllocation.maxExecutors=15" \ --conf "spark.databricks.optimizer.dynamicPartitionPruning=true" \ --conf "spark.databricks.delta.retentionDurationCheck.enabled=false" \ --conf "spark.shuffle.service.enabled=true" \ --conf "spark.shuffle.consolidateFiles=true" \ --conf "spark.shuffle.spill=false" \ --conf "spark.sql.shuffle.partitions=16" \ --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \ --conf "spark.streaming.unpersist=true" \ --conf "spark.dynamicAllocation.executorIdleTimeout=60s" \ --conf "spark.dynamicAllocation.cachedExecutorIdleTimeout=120s" \ --conf "spark.driver.memory=5G" \ --conf "spark.driver.cores=2" \ --conf "spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2" \ --conf "spark.executor.memoryOverhead=2048" \ --conf "spark.driver.memoryOverhead=1024" \ --conf "spark.default.parallelism=16" \ --conf "spark.eventLog.enabled=true" \ --conf "spark.sql.hive.convertMetastoreParquet=false" \ --num-executors=4 \ --executor-memory 10G \ --executor-cores 4 \ --jars /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/spark/external/lib/spark-avro.jar,/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.759.jar,/usr/share/aws/hmclient/lib/aws-glue-datacatalog-hive2-client-1.12.0.jar \ /home/hadoop/hudiProjects/archive/emr_hudi_to_athena_test.py &> /mnt/emr_hudi_to_athena.py.log & ``` @umehrot2 - Please help. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
