Hi Niketan,
The code you provided works fine. The use of getMatrixCharacteristics
solves the basic execution problem.
However, question #3 is probably not yet unresolved. Let me explain the use
case scenario I'm trying to build.
1. Say I have a data frame (DF1) with a Unique Id (string), a bunch of
columns (say 4) which are to be used as features (double), and a column for
the dependent variable (double).
2. When I created the model I created a data frame (DF2) from DF1 using
only the feature vectors and pass that as X. And the column with dependent
value is passed as Y.
3. For calling the GLM-predict I'm using another data frame (DF3) of same
structure but with different Unique ID (essentially different
records/rows). From that data frame I'm first creating another data frame
(DF4) containing the columns representing the features. Then I'm sending
DF4 to GLM-predict which has only feature vectors.
4. The response I get from GLM-predict is the 'means'. Then I'm using the
inline predict script which returns another data frame {DF5) with ID and
Predicted values.
The question is how do I correlate the ID I'm getting from DF5 with the
Unique ID of the data frame DF3 ?
Regards,
Sourav
On Wed, Dec 9, 2015 at 9:17 AM, Niketan Pansare <npan...@us.ibm.com> wrote:
> Hi Sourav,
>
> 1. In the GLM-predict.dml I could see 'means' is the output variable. In my
> understanding it is same as the probability matrix u have mentioned in your
> mail (to be used to compute the prediction). Am I right ?
> Yes, that's correct.
>
> 2. From GLM.dml I get the 'betas' as output using
> outputs.getBinaryBlockedRDD("beta_out"). The same I pass to GLM-predict.dml
> as B.
>
> Can you try this ?
> // Get output from GLM
> val beta = outputs.getBinaryBlockedRDD("beta_out")
> val betaMC = outputs.getMatrixCharacteristics("beta_out") // This way you
> don't have to worry about dimensions.
> // -----------------------------------------
> val Xin = DataFrame/RDD of values (or even text/csv file) you want to
> predict
> // -----------------------------------------
> // Execute GLM-predict
> ml.reset()
> // Please read
> https://github.com/apache/incubator-systemml/blob/master/scripts/algorithms/GLM.dml
> // dfam Int 1 Distribution family code: 1 = Power, 2 = Binomial
> val cmdLineParamsPredict = Map("X" -> " ", "B" -> " ", "dfam" -> "...") //
> family of distribution ?
> ml.registerInput("X", Xin)
> ml.registerInput("B_full", beta, betaMC)
> ml.registerOutput("means")
> val outputsPredict =
> ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM-predict.dml",
> cmdLineParamsPredict)
> val prob = out.getBinaryBlockedRDD("means");
> val probMC = out.getMatrixCharacteristics("means");
> // -----------------------------------------
> // Get predicted label
> ml.reset()
> ml.registerInput("Prob",prob, probMC)
> ml.registerOutput("Prediction")
> val outputsLabels = = mlNew.executeScript("Prob = read(\"temp1\"); "
> + "Prediction = rowIndexMax(Prob); "
> + "write(Prediction, \"tempOut\", \"csv\")")
> val pred = outputsLabels.getDF(sqlContext,
> "Prediction").withColumnRenamed("C1", "prediction")
> // -----------------------------------------
>
>
> 3. Say I get back prediction matrix as an output (from predictions =
> rowIndexMax(means);). Now can I read add that as a column to my original
> data frame (the one from which I created the feature vector for the
> original model) ? My concern is whether adding back will ensure the right
> order so that teh key for the feature vector and the predicted value remain
> same ? If not how to achieve the same ?
> In above example 'pred' is a DataFrame with column 'ID' which provides the
> row ID.
>
> Thanks,
>
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>
> [image: Inactive hide details for Sourav Mazumder ---12/08/2015 10:53:40
> PM---Hi Niketan, Thanks again for the detailed inputs.]Sourav Mazumder
> ---12/08/2015 10:53:40 PM---Hi Niketan, Thanks again for the detailed
> inputs.
>
> From: Sourav Mazumder <sourav.mazumde...@gmail.com>
> To: dev@systemml.incubator.apache.org, Niketan Pansare/Almaden/IBM@IBMUS
> Date: 12/08/2015 10:53 PM
> Subject: Re: Using GLM-predict
> ------------------------------
>
>
>
> Hi Niketan,
>
> Thanks again for the detailed inputs.
>
> Some more follow up Qs -
>
> 1. In the GLM-predict.dml I could see 'means' is the output variable. In my
> understanding it is same as the probability matrix u have mentioned in your
> mail (to be used to compute the prediction). Am I right ?
>
> 2. From GLM.dml I get the 'betas' as output using
> outputs.getBinaryBlockedRDD("beta_out"). The same I pass to GLM-predict.dml
> as B. For registering B following statements are used
> val beta = outputs.getBinaryBlockedRDD("beta_out")
> ml.registerInput("B", beta, 1, 4) // I have four feature vectors so I get 4
> coefficients
>
> However, when I execute GLM-predict.dml I get following error.
>
> val outputs =
> ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM-predict.dml",
> cmdLineParams)
>
> 15/12/09 05:32:47 WARN Expression: Metadata file: .mtd not provided
> 15/12/09 05:32:47 ERROR Expression: ERROR:
> /home/system-ml-0.9.0-SNAPSHOT/algori
> thms/GLM-predict.dml -- line 117, column 8 -- Missing or incomplete
> dimensio
> n information in read statement: .mtd
> com.ibm.bi.dml.parser.LanguageException: Invalid Parameters : ERROR:
> /home/syste
> m-ml-0.9.0-SNAPSHOT/algorithms/GLM-predict.dml -- line 117, column 8 --
> Miss
> ing or incomplete dimension information in read statement: .mtd
>
> In line 117 we have following statement : X = read (fileX);
>
> 3. Say I get back prediction matrix as an output (from predictions =
> rowIndexMax(means);). Now can I read add that as a column to my original
> data frame (the one from which I created the feature vector for the
> original model) ? My concern is whether adding back will ensure the right
> order so that teh key for the feature vector and the predicted value remain
> same ? If not how to achieve the same ?
>
> Regards,
> Sourav
>
>
>
>
>
> On Tue, Dec 8, 2015 at 2:08 PM, Niketan Pansare <npan...@us.ibm.com>
> wrote:
>
> > Hi Sourav,
> >
> > For some reason, I didn't get your email on "*Tue, 08 Dec 2015 12:56:38
> > -0800*
> > <
> https://www.mail-archive.com/search?l=dev@systemml.incubator.apache.org&q=date:20151208>
> "
> > (which I noticed in the archive).
> >
> > >> Not sure how exactly I can modify the GLM-predict.dml to get some
> > prediction to start with.
> > There are two options here:
> > 1. Modify GLM-predict.dml as suggested by Shirish (better approach with
> > respect to the SystemML optimizer) or
> >
> > 2. Run a new script on the output of GLM-predict. Please see:
> >
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/api/ml/LogisticRegressionModel.java#L163
> > If you chose to go with option 2, you might also want to read the
> > documentation of following two built-in functions:
> > a. rowIndexMax (See
> >
> http://apache.github.io/incubator-systemml/dml-language-reference.html#matrix-andor-scalar-comparison-built-in-functions
> > <
> http://apache.github.io/incubator-systemml/dml-language-reference.html#matrix-andor-scalar-comparison-built-in-functions
> >
> > )
> > b. ppred
> >
> > >> Can you give me some idea how from here I can calculate the predicted
> > value of the label using some value of probability threshold ?
> > Very simple way to predict the label given probability matrix:
> > Prediction = rowIndexMax(Prob) # predicts the label with highest
> > probability. This assumes one-based labels.
> >
> > Thanks,
> >
> > Niketan Pansare
> > IBM Almaden Research Center
> > E-mail: npansar At us.ibm.com
> > http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
> >
> > [image: Inactive hide details for Shirish Tatikonda ---12/08/2015
> 12:49:47
> > PM---Hi Sourav, Yes, GLM-predict.dml gives out only the prob]Shirish
> > Tatikonda ---12/08/2015 12:49:47 PM---Hi Sourav, Yes, GLM-predict.dml
> gives
> > out only the probabilities. You can put a
> >
> > From: Shirish Tatikonda <shirish.tatiko...@gmail.com>
> > To: dev@systemml.incubator.apache.org
> > Date: 12/08/2015 12:49 PM
> > Subject: Re: Using GLM-predict
> > ------------------------------
> >
> >
> >
> > Hi Sourav,
> >
> > Yes, GLM-predict.dml gives out only the probabilities. You can put a
> > threshold on the resulting probabilities to get the actual class labels
> --
> > for example, prob > 0.5 is positive and <=0.5 as negative.
> >
> > The exact value of threshold typically depends on the data and the
> > application. Different thresholds yield different classifiers with
> > different performance (precision, recall, etc.). You can find the best
> > threshold for the given data set by finding a value that gives the
> desired
> > classifier performance (for example, a threshold that gives roughly equal
> > precision and recall). Such an optimization is obviously done during the
> > training phase using a held out test set.
> >
> > If you wish, you can also modify the DML script to perform this entire
> > process.
> >
> > Shirish
> >
> >
> > On Tue, Dec 8, 2015 at 12:23 PM, Sourav Mazumder <
> > sourav.mazumde...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I have used GLM.dml to create a model using some sample data. It
> returns
> > to
> > > me the matrix of Beta, B.
> > >
> > > Now I want to use this matrix of Beta on a new set of data points and
> > > generate predicted value of the dependent variable/observation.
> > >
> > > When I checked GLM-predict, I could see that one can pass feature
> vector
> > > for the new data set and also the matrix of beta.
> > >
> > > But I could not see any way to get the predicted value of the dependent
> > > variable/observation. The output parameter only supports matrix of
> > > predicted means/probabilities.
> > >
> > > Is there a way one can get the predicted value of the dependent
> > > variable/observation from GLM-predict ?
> > >
> > > Regards,
> > > Sourav
> > >
> >
> >
> >
>
>
>
