Re: Apache SystemML 1.0 Release build

2017-07-12 Thread Matthias Boehm
here is the list of open tasks (disregarding open bugs), that I think would be good to have in SystemML 1.0: SYSTEMML-1284 Code generation -> all remaining subtasks SYSTEMML-1299 Language features 1.0 -> 1301, 1304/1307, 1306, 1316, 1426, 1444 SYSTEMML-1308 Runtime reatures 1.0 -> 1313, 1637

Re: Spark Core

2017-07-12 Thread Matthias Boehm
Well, we explicitly cleanup all intermediates that are no longer used. You can use -explain to output the runtime plan, which includes rmvar (remove variable), cpvar (copy variable) and mvvar (move variable) instructions that internally cleanup intermediates. This cleanup removes data from memory,

Spark Core

2017-07-12 Thread arijit chakraborty
Hi, Suppose I've this following code: a = matrix(seq(1,10), 10,1) for(i in 1:100){ b = a + 10 write (b, "path" + ".csv", format="csv") } So what I'm doing is for 100 items, I'm adding a constant to a matrix than outputting it. And this operation occurs in spark using multiple core