Hi Takeshi, I cant use coalesce in spark-sql shell right I know we can use coalesce in spark with scala application , here in my project we are not building jar or using python we are just executing hive query in spark-sql shell and submitting to yarn client .
Example:- spark-sql --verbose --queue default --name wchargeback_event.sparksql.kali --master yarn-client --driver-memory 15g --executor-memory 15g --num-executors 10 --executor-cores 2 -f /x/home/pp_dt_fin_batch/users/ srtummala/run-spark/sql/wtr_full.sql --conf "spark.yarn.executor.memoryOverhead=8000" --conf "spark.sql.shuffle.partitions=50" --conf "spark.kyroserializer.buffer.max.mb=5g" --conf "spark.driver.maxResultSize=20g" --conf "spark.storage.memoryFraction=0.8" --conf "spark.hadoopConfiguration=256000000000" --conf "spark.dynamicAllocation.enabled=false$" --conf "spark.shuffle.service.enabled=false" --conf "spark.executor.instances=10" Thanks Sri On Sat, Jul 2, 2016 at 2:53 AM, Takeshi Yamamuro <linguin....@gmail.com> wrote: > Please also see https://issues.apache.org/jira/browse/SPARK-16188. > > // maropu > > On Fri, Jul 1, 2016 at 7:39 PM, kali.tumm...@gmail.com < > kali.tumm...@gmail.com> wrote: > >> I found the jira for the issue will there be a fix in future ? or no fix ? >> >> https://issues.apache.org/jira/browse/SPARK-6221 >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-parquet-too-many-small-files-tp27264p27267.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> > > > -- > --- > Takeshi Yamamuro > -- Thanks & Regards Sri Tummala
