回复：saving DF to HDFS in parquet format very slow in SparkSQL app

[luohui20001]

maybe this problem is not so easy to understand, so I attached my full code.
Hope this could help in solving the problem.



 

ThanksBest regards!
San.Luo

- 原始邮件 -
发件人：
收件人："user" 
主题：saving DF to HDFS in parquet format very slow in SparkSQL app
日期：2016年08月09日 15点34分

hi there:I got a problem in saving a DF to HDFS as parquet format very 
slow. And I attached a pic which shows a lot of time is spent in getting 
result.the code is 
:streamingData.write.mode(SaveMode.Overwrite).parquet("/data/streamingData")
I don't quite understand why my app is so slow in getting the result. I tried 
to access my HDFS while the app is running slow , HDFS is ok.
Any idea will be appreciated.




 

ThanksBest regards!
San.Luo


DataExtractor.scala
Description: Binary data

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

saving DF to HDFS in parquet format very slow in SparkSQL app

[luohui20001]

hi there:I got a problem in saving a DF to HDFS as parquet format very 
slow. And I attached a pic which shows a lot of time is spent in getting 
result.the code is 
:streamingData.write.mode(SaveMode.Overwrite).parquet("/data/streamingData")
I don't quite understand why my app is so slow in getting the result. I tried 
to access my HDFS while the app is running slow , HDFS is ok.
Any idea will be appreciated.




 

ThanksBest regards!
San.Luo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org