[jira] [Commented] (SPARK-20394) Replication factor value Not changing properly
[ https://issues.apache.org/jira/browse/SPARK-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094686#comment-16094686 ] Kannan Subramanian commented on SPARK-20394: Yes. I have edited the hdfs-site.xml for replication value change and override the existing Spark environment. It works now. Thanks for your response Kannan > Replication factor value Not changing properly > -- > > Key: SPARK-20394 > URL: https://issues.apache.org/jira/browse/SPARK-20394 > Project: Spark > Issue Type: Bug > Components: Spark Core, Spark Submit >Affects Versions: 1.6.0 >Reporter: Kannan Subramanian > > To save SparkSQL dataframe to a persistent hive table using the below steps. > a) RegisterTempTable to the dataframe as a tempTable > b) create table (cols)PartitionedBy(col1, col2) stored as > parquet > c) Insert into partition(col1, col2) select * from tempTable > I have set dfs.replication is equal to "1" in hiveContext object. But It did > not work properly. That is replica is 1 for 80 % of the generated parquet > files on HDFS and default replica 3 is for remaining 20 % of parquet files in > HDFS. I am not sure why the replica is not reflecting to all the generated > parquet files. Please let me know if you have any suggestions or solutions -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20394) Replication factor value Not changing properly
[
https://issues.apache.org/jira/browse/SPARK-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081318#comment-16081318
]
Marcelo Vanzin commented on SPARK-20394:
Have you tried setting the replication to 1 in your {{hdfs-site.xml}}?
IIRC Spark 1.6 doesn't propagate the HiveContext configuration to the Hive
library in some cases.
> Replication factor value Not changing properly
> --
>
> Key: SPARK-20394
> URL: https://issues.apache.org/jira/browse/SPARK-20394
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
>Affects Versions: 1.6.0
>Reporter: Kannan Subramanian
>
> To save SparkSQL dataframe to a persistent hive table using the below steps.
> a) RegisterTempTable to the dataframe as a tempTable
> b) create table (cols)PartitionedBy(col1, col2) stored as
> parquet
> c) Insert into partition(col1, col2) select * from tempTable
> I have set dfs.replication is equal to "1" in hiveContext object. But It did
> not work properly. That is replica is 1 for 80 % of the generated parquet
> files on HDFS and default replica 3 is for remaining 20 % of parquet files in
> HDFS. I am not sure why the replica is not reflecting to all the generated
> parquet files. Please let me know if you have any suggestions or solutions
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20394) Replication factor value Not changing properly
[ https://issues.apache.org/jira/browse/SPARK-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978229#comment-15978229 ] Kannan Subramanian commented on SPARK-20394: Please let me know anyone is looking on this issue. > Replication factor value Not changing properly > -- > > Key: SPARK-20394 > URL: https://issues.apache.org/jira/browse/SPARK-20394 > Project: Spark > Issue Type: Bug > Components: Spark Core, Spark Submit >Affects Versions: 1.6.0 >Reporter: Kannan Subramanian > > To save SparkSQL dataframe to a persistent hive table using the below steps. > a) RegisterTempTable to the dataframe as a tempTable > b) create table (cols)PartitionedBy(col1, col2) stored as > parquet > c) Insert into partition(col1, col2) select * from tempTable > I have set dfs.replication is equal to "1" in hiveContext object. But It did > not work properly. That is replica is 1 for 80 % of the generated parquet > files on HDFS and default replica 3 is for remaining 20 % of parquet files in > HDFS. I am not sure why the replica is not reflecting to all the generated > parquet files. Please let me know if you have any suggestions or solutions -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
