Re: Spark ML with null labels

2019-01-10 Thread Patrick McCarthy
I actually tried that first. I moved away from it because the algorithm needs to evaluate all records for all models, for instance, a model trained on (2,4) needs to be evaluated on a record whose true label is 8. I found that if I apply the filter in the label-creation transformer, then a record

Re: Spark ML with null labels

2019-01-10 Thread Xiangrui Meng
In your custom transformer that produces labels, can you filter null labels? A transformer doesn't always need to do 1:1 mapping. On Thu, Jan 10, 2019, 7:53 AM Patrick McCarthy I'm trying to implement an algorithm on the MNIST digits that runs like so: > > >- for every pair of digits (0,1),

Spark ML with null labels

2019-01-10 Thread Patrick McCarthy
I'm trying to implement an algorithm on the MNIST digits that runs like so: - for every pair of digits (0,1), (0,2), (0,3)... assign a 0/1 label to the digits and build a LogisticRegression Classifier -- 45 in total - Fit every classifier on the test set separately - Aggregate the