Thanks Donald, By agggregative, do you mean using aggregateProperties()?
We are looking to use the aggregate function to get the comprehensive most recent value for a given entity. If I were able to use LEvent store I'm assuming I can stay with P2Algorithm instead of switching to PAlgorithm. Is that correct? @Pat Ferrel, did your team do something similar, use ids to go get aggregate properties in the predict method? I prototyped with the universal Recommendation template a couple months back and remember the predict endpoint only required ids. Thanks for the support. On Thu, Oct 5, 2017 at 5:15 PM Donald Szeto <[email protected]> wrote: > Hi Shane, > > If you are not looking to do aggregative on Spark when you retrieve > additional information from event store, you probably should look at using > LEventStore that does not go through Spark. Depending on your use case, the > roundtrip time of involving Spark in your predict method might not be > feasible. (If you use `pio batchpredict` that could be an exception.) > > Regards, > Donald > > On Thu, Oct 5, 2017 at 3:59 PM, Shane Johnson < > [email protected]> wrote: > >> Thanks Daniel! I'll go look for that in the docs. >> >> On Thu, Oct 5, 2017 at 3:23 PM Shane Johnson < >> [email protected]> wrote: >> >>> Thanks Daniel. I may be missing what you are saying. I actually think I >>> need the Spark Context for what I am trying to do. I am wanting to extend >>> the predict method to use id's from the Query object and then go back into >>> the EventStore to get additional attributes that were not passed in the >>> Query. >>> >>> To do this I need to have a sc or I cannot get the properties from the >>> EventStore. Does that make sense. If I remove the Spark Context it does >>> indeed work but not for what I am trying to do. >>> >>> Thanks >>> >>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >>> <https://www.facebook.com/shane.johnson.71653> >>> >>> On Thu, Oct 5, 2017 at 3:19 PM, Daniel O' Shaughnessy < >>> [email protected]> wrote: >>> >>>> It actually doesn't look like you use spark context within the predict >>>> method itself... >>>> >>>> Try Removing the spark context ref from the method Params and also the >>>> (sc) at the end of the predict method. >>>> >>>> On Thu, Oct 5, 2017 at 10:13 PM Shane Johnson < >>>> [email protected]> wrote: >>>> >>>>> Thanks Daniel. I had the P2Algorithm working before I had to query the >>>>> EventStore within the predict method. Do you think this is still the issue >>>>> with the context that I had it working before attempting to add the >>>>> SparkContext? >>>>> >>>>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >>>>> <https://www.facebook.com/shane.johnson.71653> >>>>> >>>>> On Thu, Oct 5, 2017 at 3:07 PM, Daniel O' Shaughnessy < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Shane, >>>>>> >>>>>> Your RFAlgorithm class needs to use PAlgorithm instead of >>>>>> P2Algorithm. You then need to write some code to save and load your model >>>>>> and spark context etc. >>>>>> >>>>>> There should be examples of this on the predictionio site somewhere >>>>>> >>>>>> On Thu, Oct 5, 2017 at 10:00 PM Shane Johnson < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi team, >>>>>>> >>>>>>> Can someone guide how I can add SparkContext into the predict >>>>>>> method. I am using unique ids that I gather from Query and pulling back >>>>>>> additional attributes from the PEventStore and am getting an error that >>>>>>> "sc" cannot be found. When I add SparkContext to the method I get the >>>>>>> following error. >>>>>>> >>>>>>> Can anyone provide direction here? >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> Adding SparkContext eliminates the first error but produces another. >>>>>>> >>>>>>> def predict(sc: SparkContext, model: RFModel, query: Query): >>>>>>> PredictedResult = { >>>>>>> >>>>>>> val featureIndex = model.featureIndex >>>>>>> val featureCategoricalIntMap = model.featureCategoricalIntMap >>>>>>> >>>>>>> >>>>>>> val responses: List[PredictionResponses] = >>>>>>> query.predictionRequests >>>>>>> .map { >>>>>>> Predictions => >>>>>>> >>>>>>> val oppPost = PEventStore.aggregateProperties( >>>>>>> appName = sys.env("PIO_EVENTSERVER_APP_NAME"), >>>>>>> entityType = "Opportunity" >>>>>>> )(sc) >>>>>>> >>>>>>> >>>>>>> First error when sc: SparkContext is not added to the method >>>>>>> parameters: >>>>>>> >>>>>>> not found: value sc >>>>>>> [INFO] [Engine$] [error] )(sc) >>>>>>> [INFO] [Engine$] [error] ^ >>>>>>> >>>>>>> >>>>>>> Error after adding SparkContext: >>>>>>> >>>>>>> class RFAlgorithm needs to be abstract, since method predict in >>>>>>> class P2LAlgorithm of type (model: org.template.liftscori >>>>>>> ng.RFModel, query: org.template.liftscoring.Query)org.template >>>>>>> .liftscoring.PredictedResult is not defined >>>>>>> [INFO] [Engine$] [error] class RFAlgorithm(val ap: RFAlgorithmParams >>>>>>> ) >>>>>>> [INFO] [Engine$] [error] ^ >>>>>>> [INFO] [Engine$] [error] one error found >>>>>>> [INFO] [Engine$] [error] (compile:compileIncremental) Compilation >>>>>>> failed >>>>>>> [INFO] [Engine$] [error] Total time: 6 s, completed Oct 5, 2017 2:44 >>>>>>> :51 PM >>>>>>> >>>>>>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >>>>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >>>>>>> <https://www.facebook.com/shane.johnson.71653> >>>>>>> >>>>>> >>>>> >>> -- >> >> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >> <https://www.facebook.com/shane.johnson.71653> >> > > -- *Shane Johnson | 801.360.3350* LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook <https://www.facebook.com/shane.johnson.71653>
