Re: Adding Spark Context (sc) to predict method?

Hi Shane, It looks like your use case will involve a lot of computation within Spark in the predict method, so in that case using PAlgorithm is recommended. Depending on your SLA though, the roundtrip time could be somewhat in the hundreds ms and up under load. If this is acceptable, PAlgorithm

Re: Adding Spark Context (sc) to predict method?

Thanks for the guidance Daniel and Donald. A few follow up items to check my understanding... Daniel, here is the link to the documentation I think you were referencing on how to save a model with RDD. I read through it and see a way to train with an RDD but did not see a way to get the

Re: Adding Spark Context (sc) to predict method?

Thanks Donald, By agggregative, do you mean using aggregateProperties()? We are looking to use the aggregate function to get the comprehensive most recent value for a given entity. If I were able to use LEvent store I'm assuming I can stay with P2Algorithm instead of switching to PAlgorithm. Is

Re: Adding Spark Context (sc) to predict method?

Hi Shane, If you are not looking to do aggregative on Spark when you retrieve additional information from event store, you probably should look at using LEventStore that does not go through Spark. Depending on your use case, the roundtrip time of involving Spark in your predict method might not

Re: Adding Spark Context (sc) to predict method?

Thanks Daniel! I'll go look for that in the docs. On Thu, Oct 5, 2017 at 3:23 PM Shane Johnson wrote: > Thanks Daniel. I may be missing what you are saying. I actually think I > need the Spark Context for what I am trying to do. I am wanting to extend > the predict

Re: Adding Spark Context (sc) to predict method?

Thanks Daniel. I may be missing what you are saying. I actually think I need the Spark Context for what I am trying to do. I am wanting to extend the predict method to use id's from the Query object and then go back into the EventStore to get additional attributes that were not passed in the

Re: Adding Spark Context (sc) to predict method?

Sorry ignore that last one, read your post wrong. Yes try the PAlgorithm and you should be able to use spark context like you want On Thu, Oct 5, 2017 at 10:19 PM Daniel O' Shaughnessy < danieljamesda...@gmail.com> wrote: > It actually doesn't look like you use spark context within the predict

Re: Adding Spark Context (sc) to predict method?

It actually doesn't look like you use spark context within the predict method itself... Try Removing the spark context ref from the method Params and also the (sc) at the end of the predict method. On Thu, Oct 5, 2017 at 10:13 PM Shane Johnson wrote: > Thanks

Re: Adding Spark Context (sc) to predict method?

Thanks Daniel. I had the P2Algorithm working before I had to query the EventStore within the predict method. Do you think this is still the issue with the context that I had it working before attempting to add the SparkContext? *Shane Johnson | 801.360.3350* LinkedIn

Re: Adding Spark Context (sc) to predict method?

Hi Shane, Your RFAlgorithm class needs to use PAlgorithm instead of P2Algorithm. You then need to write some code to save and load your model and spark context etc. There should be examples of this on the predictionio site somewhere On Thu, Oct 5, 2017 at 10:00 PM Shane Johnson

Adding Spark Context (sc) to predict method?

Hi team, Can someone guide how I can add SparkContext into the predict method. I am using unique ids that I gather from Query and pulling back additional attributes from the PEventStore and am getting an error that "sc" cannot be found. When I add SparkContext to the method I get the following