Re: Loading a spark dataframe column into T-Digest using java

The closure in your "foreach" loop runs in a remote executor, no the local JVM, so it's updating its own copy of the t-digest instance. The one on the driver side is never touched. On Sun, Dec 10, 2017 at 10:27 PM, Himasha de Silva wrote: > Hi, > > I want to load a spark

Loading a spark dataframe column into T-Digest using java

Hi, I want to load a spark dataframe column into T-Digest using java to calculate quantile values. I write this code to do this, but it's giving zero for size of tdigest. values are not added to tDigest. my code - https://gist.github.com/anonymous/1f2e382fdda002580154b5c43fbe9b3a Thank you.