reading the parquet file

[Angel Angel]

Hello Sir/Madam,


I writing the spark application in spark 1.4.0.

I have one text file with the size of 8 GB.
I save that file in parquet format


val df2 =
sc.textFile("/root/Desktop/database_200/database_200.txt").map(_.split(",")).map(p
=> Table(p(0),p(1).trim.toInt, p(2).trim.toInt, p(3)))toDF


df2.write.parquet("hdfs://hadoopm0:8020/tmp/input1/database4.parquet")

After that i did the following operations


val df1 =
sqlContext.read.parquet("dfs://hadoopm0:8020/tmp/input1/database4.parquet")


var a=0

var k = df1.filter(df1("Address").equalTo(Array_Ele(0) ))


for( a <-2 until 2720 by 2){


var temp= df1.filter(df1("Address").equalTo(Array_Ele(a)))


var temp1 =
temp.select(temp("Address"),temp("Couple_time")-Array_Ele(a+1),temp("WT_ID"),temp("WT_Name"))


k =k.unionAll(temp1) }


val WT_ID_Sort  = k.groupBy("WT_ID").count().sort(desc("count"))



WT_ID_Sort.show()



after that I am getting the following warning and my task is disconnected
again and again.





[image: Inline image 1]




I need to do many iterative operations on that df1 file.


So can any one help me to solve this problem?

thanks in advance.


Thanks.