Cheng, yes, it works, I set the property in SparkConf before initiating SparkContext. The property name is "spark.hadoop.dfs.replication" Thanks fro the help!
-----Original Message----- From: Cheng Lian [mailto:lian.cs....@gmail.com] Sent: Monday, June 08, 2015 6:41 PM To: Haopu Wang; user Subject: Re: SparkSQL: How to specify replication factor on the persisted parquet files? Then one possible workaround is to set "dfs.replication" in "sc.hadoopConfiguration". However, this configuration is shared by all Spark jobs issued within the same application. Since different Spark jobs can be issued from different threads, you need to pay attention to synchronization. Cheng On 6/8/15 2:46 PM, Haopu Wang wrote: > Cheng, thanks for the response. > > Yes, I was using HiveContext.setConf() to set "dfs.replication". > However, I cannot change the value in Hadoop core-site.xml because that > will change every HDFS file. > I only want to change the replication factor of some specific files. > > -----Original Message----- > From: Cheng Lian [mailto:lian.cs....@gmail.com] > Sent: Sunday, June 07, 2015 10:17 PM > To: Haopu Wang; user > Subject: Re: SparkSQL: How to specify replication factor on the > persisted parquet files? > > Were you using HiveContext.setConf()? > > "dfs.replication" is a Hadoop configuration, but setConf() is only used > to set Spark SQL specific configurations. You may either set it in your > Hadoop core-site.xml. > > Cheng > > > On 6/2/15 2:28 PM, Haopu Wang wrote: >> Hi, >> >> I'm trying to save SparkSQL DataFrame to a persistent Hive table using >> the default parquet data source. >> >> I don't know how to change the replication factor of the generated >> parquet files on HDFS. >> >> I tried to set "dfs.replication" on HiveContext but that didn't work. >> Any suggestions are appreciated very much! >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
