SparkR : lapplyPartition transforms the data in vertical format
Hello As per documentation, lapply works on single records and lapplyPartition works on partition However the format of output does not change When I use lapplypartition, the data is converted to vertical format Here is my code library(SparkR) sc <- sparkR.init("local") lines <- textFile(sc,"/sparkdev/datafiles/covariance.txt") totals <- lapply(lines, function(lines) { sumx <- 0 sumy <- 0 totaln <- 0 for (i in 1:length(lines)){ dataxy <- unlist(strsplit(lines[i], ",")) sumx <- sumx + as.numeric(dataxy[1]) sumy <- sumy + as.numeric(dataxy[2]) } ##list(as.numeric(sumx), as.numeric(sumy), as.numeric(sumxy), as.numeric(totaln)) ##list does same as below c(sumx,sumy) } ) output <- collect(totals) for (element in output) { cat(as.character(element[1]),as.character(element[2]), "\n") } I am expecting output as 55, 55 However it is giving 55,NA 55,NA Where am I going wrong ? Thanks Pranay -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-lapplyPartition-transforms-the-data-in-vertical-format-tp11540.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: SparkR : lapplyPartition transforms the data in vertical format
Hello Shivram Thanks for your reply. Here is a simple data set input. This data is in file called "/sparkdev/datafiles/covariance.txt" 1,1 2,2 3,3 4,4 5,5 6,6 7,7 8,8 9,9 10,10 Output I would like to see is a total of columns. It can be done with reduce, but I wanted to test lapply. Output I want to see is sum of columns in same row 55,55 But output what I get is in two rows 55, NA 55, NA Thanks Pranay -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-lapplyPartition-transforms-the-data-in-vertical-format-tp11540p11617.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: SparkR : lapplyPartition transforms the data in vertical format
Hello Zongheng Infact the problem is in lapplyPartition lapply gives output as 1,1 2,2 3,3 ... 10,10 However lapplyPartition gives output as 55, NA 55, NA Why lapply output is horizontal and lapplyPartition is vertical ? Here is my code library(SparkR) sc <- sparkR.init("local") lines <- textFile(sc,"/sparkdev/datafiles/covariance.txt") totals <- lapplyPartition(lines, function(lines) { sumx <- 0 sumy <- 0 totaln <- 0 for (i in 1:length(lines)){ dataxy <- unlist(strsplit(lines[i], ",")) sumx <- sumx + as.numeric(dataxy[1]) sumy <- sumy + as.numeric(dataxy[2]) } ##list(as.numeric(sumx), as.numeric(sumy), as.numeric(sumxy), as.numeric(totaln)) ##list does same as below c(sumx,sumy) } ) output <- collect(totals) for (element in output) { cat(as.character(element[1]),as.character(element[2]), "\n") } -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-lapplyPartition-transforms-the-data-in-vertical-format-tp11540p11726.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org