Re: closure serialization behavior driving me crazy

2014-11-11 Thread Sandy Ryza
I tried turning on the extended debug info. The Scala output is a little opaque (lots of - field (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC), but it seems like, as expected, somehow the full array of OLSMultipleLinearRegression objects is

closure serialization behavior driving me crazy

2014-11-10 Thread Sandy Ryza
I'm experiencing some strange behavior with closure serialization that is totally mind-boggling to me. It appears that two arrays of equal size take up vastly different amount of space inside closures if they're generated in different ways. The basic flow of my app is to run a bunch of tiny

Re: closure serialization behavior driving me crazy

2014-11-10 Thread Matei Zaharia
Hey Sandy, Try using the -Dsun.io.serialization.extendedDebugInfo=true flag on the JVM to print the contents of the objects. In addition, something else that helps is to do the following: { val _arr = arr models.map(... _arr ...) } Basically, copy the global variable into a local one.