[ https://issues.apache.org/jira/browse/SPARK-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-11475. ------------------------------- Resolution: Not A Problem > DataFrame API saveAsTable() does not work well for HDFS HA > ---------------------------------------------------------- > > Key: SPARK-11475 > URL: https://issues.apache.org/jira/browse/SPARK-11475 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.1 > Environment: Hadoop 2.4 & Spark 1.5.1 > Reporter: zhangxiongfei > Attachments: dataFrame_saveAsTable.txt, hdfs-site.xml, hive-site.xml > > > I was trying to save a DF to Hive using following code: > {quote} > sqlContext.range(1L,1000L,2L,2).coalesce(1).saveAsTable("dataframeTable") > {quote} > But got below exception: > {quote} > arning: there were 1 deprecation warning(s); re-run with -deprecation for > details > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): > Operation category READ is not supported in state standby > at > org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) > at > org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1610) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1193) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3516) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:785) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo( > {quote} > *My Hive configuration is* : > {quote} > <property> > <name>hive.metastore.warehouse.dir</name> > <value>*/apps/hive/warehouse*</value> > </property> > {quote} > It seems that the hdfs HA is not configured,then I tried below code: > {quote} > sqlContext.range(1L,1000L,2L,2).coalesce(1).saveAsParquetFile("hdfs://bitautodmp/apps/hive/warehouse/dataframeTable") > {quote} > I could verified that API *saveAsParquetFile* worked well by following > commands: > {quote} > *hadoop fs -ls /apps/hive/warehouse/dataframeTable* > Found 4 items > -rw-r--r-- 3 zhangxf hdfs 0 2015-11-03 17:57 > */apps/hive/warehouse/dataframeTable/_SUCCESS* > -rw-r--r-- 3 zhangxf hdfs 199 2015-11-03 17:57 > */apps/hive/warehouse/dataframeTable/_common_metadata* > -rw-r--r-- 3 zhangxf hdfs 325 2015-11-03 17:57 > */apps/hive/warehouse/dataframeTable/_metadata* > -rw-r--r-- 3 zhangxf hdfs 1098 2015-11-03 17:57 > */apps/hive/warehouse/dataframeTable/part-r-00000-a05a9bf3-b2a6-40e5-b180-818efb2a0f54.gz.parquet* > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org