Re: Different versions of dependencies in assembly/target/scala-2.11/jars?

2016-09-18 Thread Sean Owen
Boy it's a long story, but I think the short answer is that it's only worth manually fixing the mismatches that are clearly going to cause a problem. Dependency version mismatch is inevitable, and Maven will always settle on one version of a particular group/artifact using a nearest-wins rule (SBT

Re: Different versions of dependencies in assembly/target/scala-2.11/jars?

2016-09-18 Thread Jacek Laskowski
Hi Sean, "Boy it's a long story"...yeah, you tell me! :) I don't seem to find anything worth reporting so...let's keep these possible discrepancies in mind and be back to them when they hit us. Thanks a lot, Sean. Your patience with dealing with people here and on JIRA has always made me wish for

Memory usage for spark types

2016-09-18 Thread assaf.mendelson
Hi, I am trying to understand how spark types are kept in memory and accessed. I tried to look at the code at the definition of MapType and ArrayType for example and I can't seem to find the relevant code for its actual implementation. I am trying to figure out how these two types are implemente

Re: Memory usage for spark types

2016-09-18 Thread Reynold Xin
Take a look at UnsafeArrayData and UnsafeMapData. On Sun, Sep 18, 2016 at 9:06 AM, assaf.mendelson wrote: > Hi, > > I am trying to understand how spark types are kept in memory and accessed. > > I tried to look at the code at the definition of MapType and ArrayType for > example and I can’t see