zhangxiongfei created SPARK-11475:
-------------------------------------
Summary: DataFrame API saveAsTable() does not work well for HDFS HA
Key: SPARK-11475
URL: https://issues.apache.org/jira/browse/SPARK-11475
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.1
Environment: Hadoop 2.4 & Spark 1.5.1
Reporter: zhangxiongfei
I was trying to save a DF to Hive using following code:
{quote}
sqlContext.range(1L,1000L,2L,2).coalesce(1).saveAsTable("dataframeTable")
{quote}
But got below exception:
{quote}
arning: there were 1 deprecation warning(s); re-run with -deprecation for
details
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category READ is not supported in state standby
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1610)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1193)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3516)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:785)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(
{quote}
*My Hive configuration is* :
{quote}
<property>
<name>hive.metastore.warehouse.dir</name>
<value>*/apps/hive/warehouse*</value>
</property>
{quote}
It seems that the hdfs HA is not configured,then I tried below code:
{quote}
sqlContext.range(1L,1000L,2L,2).coalesce(1).saveAsParquetFile("hdfs://bitautodmp/apps/hive/warehouse/dataframeTable")
{quote}
I could verified that API *saveAsParquetFile* worked well by following
commands:
{quote}
*hadoop fs -ls /apps/hive/warehouse/dataframeTable*
Found 4 items
-rw-r--r-- 3 zhangxf hdfs 0 2015-11-03 17:57
*/apps/hive/warehouse/dataframeTable/_SUCCESS*
-rw-r--r-- 3 zhangxf hdfs 199 2015-11-03 17:57
*/apps/hive/warehouse/dataframeTable/_common_metadata*
-rw-r--r-- 3 zhangxf hdfs 325 2015-11-03 17:57
*/apps/hive/warehouse/dataframeTable/_metadata*
-rw-r--r-- 3 zhangxf hdfs 1098 2015-11-03 17:57
*/apps/hive/warehouse/dataframeTable/part-r-00000-a05a9bf3-b2a6-40e5-b180-818efb2a0f54.gz.parquet*
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
