maybe this problem is not so easy to understand, so I attached my full code. Hope this could help in solving the problem.
-------------------------------- Thanks&Best regards! San.Luo ----- 原始邮件 ----- 发件人:<luohui20...@sina.com> 收件人:"user" <user@spark.apache.org> 主题:saving DF to HDFS in parquet format very slow in SparkSQL app 日期:2016年08月09日 15点34分 hi there: I got a problem in saving a DF to HDFS as parquet format very slow. And I attached a pic which shows a lot of time is spent in getting result.the code is :streamingData.write.mode(SaveMode.Overwrite).parquet("/data/streamingData") I don't quite understand why my app is so slow in getting the result. I tried to access my HDFS while the app is running slow , HDFS is ok. Any idea will be appreciated. -------------------------------- Thanks&Best regards! San.Luo
DataExtractor.scala
Description: Binary data
--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org