Re: Error in getting Total Events in a predictionIo App

2017-11-14 Thread Pat Ferrel 
You should use pio 0.12.0 if you need Elasticsearch 5.x


On Nov 14, 2017, at 6:39 AM, Abhimanyu Nagrath  
wrote:

Hi , I am new to predictionIo using version V0.11-incubating (spark - 2.6.1 , 
hbase - 1.2.6 , elasticsearch - 5.2.1) . Started the prediction server with 
./pio-start-all and checked Pio status these are working fine. Then I created 
an app 'testApp' and imported some events into that predictionIO app, Now 
inorder to verify the count of imported events .I ran the following commands 

 1. pio-shell --with-spark
 2. import org.apache.predictionio.data.store.PEventStore
 3. val eventsRDD = PEventStore.find(appName="testApp")(sc)

I got the error:

ERROR Storage$: Error initializing storage client for source ELASTICSEARCH
java.lang.ClassNotFoundException: elasticsearch.StorageClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at 
org.apache.predictionio.data.storage.Storage$.getClient(Storage.scala:228)
at 
org.apache.predictionio.data.storage.Storage$.org$apache$predictionio$data$storage$Storage$$updateS2CM(Storage.scala:254)
at 
org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:215)
at 
org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:215)
at 
scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
at 
org.apache.predictionio.data.storage.Storage$.sourcesToClientMeta(Storage.scala:215)
at 
org.apache.predictionio.data.storage.Storage$.getDataObject(Storage.scala:284)
at 
org.apache.predictionio.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:269)
at 
org.apache.predictionio.data.storage.Storage$.getMetaDataApps(Storage.scala:387)
at 
org.apache.predictionio.data.store.Common$.appsDb$lzycompute(Common.scala:27)
at org.apache.predictionio.data.store.Common$.appsDb(Common.scala:27)
at 
org.apache.predictionio.data.store.Common$.appNameToId(Common.scala:32)
at 
org.apache.predictionio.data.store.PEventStore$.find(PEventStore.scala:71)
at 
$line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:28)
at $line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:33)
at $line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:35)
at $line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC.(:37)
at $line19.$read$$iwC$$iwC$$iwC$$iwC.(:39)
at $line19.$read$$iwC$$iwC$$iwC.(:41)
at $line19.$read$$iwC$$iwC.(:43)
at $line19.$read$$iwC.(:45)
at $line19.$read.(:47)
at $line19.$read$.(:51)
at $line19.$read$.()
at $line19.$eval$.(:7)
at $line19.$eval$.()
at $line19.$eval.$print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at 
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at 
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at 
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at 
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org 
$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at 
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at 
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at 
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at 
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org

Re: Which template for predicting ratings?

2017-11-14 Thread Noelia Osés Fernández 
Thanks Pat.

I am now using the Recommendation template (
http://predictionio.incubator.apache.org/templates/recommendation/quickstart/)
(https://github.com/apache/incubator-predictionio-template-recommender). I
believe this template uses MLlib ALS.

I am using the movielens ratings data. In the sample that I'm using, the
minimum rating is 0.5 and the max is 5.

However, the predictions returned by the recommendation engine are above 5.
For example:

Recommendations for user: 1

{"itemScores":[{"item":"2492","score":8.760136688429496},{"item":"103228","score":8.074123814810278},{"item":"2907","score":7.659090305689766},{"item":"6755","score":7.65084600130184}]}

Shouldn't these predictions be in the range from 0.5 to 5 ?



On 13 November 2017 at 18:53, Pat Ferrel  wrote:

> What I was saying is the UR can use ratings, but not predict them. Use
> MLlib ALS recommenders if you want to predict them for all items.
>
>
> On Nov 13, 2017, at 9:32 AM, Pat Ferrel  wrote:
>
> What we did in the article I attached is assume 1-2 is dislike, and 4-5 is
> like.
>
> These are treated as indicators and will produce a score from the
> recommender but these do not relate to 1-5 scores.
>
> If you need to predict what the user would score an item MLlib ALS
> templates will do it.
>
>
>
> On Nov 13, 2017, at 2:42 AM, Noelia Osés Fernández 
> wrote:
>
> Hi Pat,
>
> I truly appreciate your advice.
>
> However, what to do with a client that is adamant that they want to
> display the predicted ratings in the form of 1 to 5-stars? That's my case
> right now.
>
> I will pose a more concrete question. *Is there any template for which
> the scores predicted by the algorithm are in the same range as the ratings
> in the training set?*
>
> Thank you very much for your help!
> Noelia
>
> On 10 November 2017 at 17:57, Pat Ferrel  wrote:
>
>> Any of the Spark MLlib ALS recommenders in the PIO template gallery
>> support ratings.
>>
>> However I must warn that ratings are not very good for recommendations
>> and none of the big players use ratings anymore, Netflix doesn’t even
>> display them. The reason is that your 2 may be my 3 or 4 and that people
>> rate different categories differently. For instance Netflix found Comedies
>> were rated lower than Independent films. There have been many solutions
>> proposed and tried but none have proven very helpful.
>>
>> There is another more fundamental problem, why would you want to
>> recommend the highest rated item? What do you buy on Amazon or watch on
>> Netflix? Are they only your highest rated items. Research has shown that
>> they are not. There was a whole misguided movement around ratings that
>> affected academic papers and cross-validation metrics that has fairly well
>> been discredited. It all came from the Netflix prize that used both.
>> Netflix has since led the way in dropping ratings as they saw the things I
>> have mentioned.
>>
>> What do you do? Categorical indicators work best (like, dislike)or
>> implicit indicators (buy) that are unambiguous. If a person buys something,
>> they like it, if the rate it 3 do they like it? I buy many 3 rated items on
>> Amazon if I need them.
>>
>> My advice is drop ratings and use thumbs up or down. These are
>> unambiguous and the thumbs down can be used in some cases to predict thumbs
>> up: https://developer.ibm.com/dwblog/2017/mahout-spark-corre
>> lated-cross-occurences/ This uses data from a public web site to show
>> significant lift by using “like” and “dislike” in recommendations. This
>> used the Universal Recommender.
>>
>>
>> On Nov 10, 2017, at 5:02 AM, Noelia Osés Fernández 
>> wrote:
>>
>>
>> Hi all,
>>
>> I'm new to PredictionIO so I apologise if this question is silly.
>>
>> I have an application in which users are rating different items in a
>> scale of 1 to 5 stars. I want to recommend items to a new user and give her
>> the predicted rating in number of stars. Which template should I use to do
>> this? Note that I need the predicted rating to be in the same range of 1 to
>> 5 stars.
>>
>> Is it possible to do this with the ecommerce recommendation engine?
>>
>> Thank you very much for your help!
>> Noelia
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> 
>
> Noelia Osés Fernández, PhD
> Senior Researcher |
> Investigadora Senior
>
> no...@vicomtech.org
> +[34] 943 30 92 30
> Data Intelligence for Energy and
> Industrial Processes | Inteligencia
> de Datos para Energía y Procesos
> Industriales
>
> 
> 
> 
>
> member of:   
>
> Legal Notice - Privacy policy
> 
>
>
>


-- 


Noelia Osés Fernández, PhD
Senior Researcher |
Investigadora Senior

Error in getting Total Events in a predictionIo App

2017-11-14 Thread Abhimanyu Nagrath 
Hi , I am new to predictionIo using version V0.11-incubating (spark - 2.6.1
, hbase - 1.2.6 , elasticsearch - 5.2.1) . Started the prediction server
with ./pio-start-all and checked Pio status these are working fine. Then I
created an app 'testApp' and imported some events into that predictionIO
app, Now inorder to verify the count of imported events .I ran the
following commands

 1. pio-shell --with-spark
 2. import org.apache.predictionio.data.store.PEventStore
 3. val eventsRDD = PEventStore.find(appName="testApp")(sc)

I got the error:

ERROR Storage$: Error initializing storage client for source
ELASTICSEARCH
java.lang.ClassNotFoundException: elasticsearch.StorageClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.predictionio.data.storage.Storage$.getClient(Storage.scala:228)
at
org.apache.predictionio.data.storage.Storage$.org$apache$predictionio$data$storage$Storage$$updateS2CM(Storage.scala:254)
at
org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:215)
at
org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:215)
at
scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
at
org.apache.predictionio.data.storage.Storage$.sourcesToClientMeta(Storage.scala:215)
at
org.apache.predictionio.data.storage.Storage$.getDataObject(Storage.scala:284)
at
org.apache.predictionio.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:269)
at
org.apache.predictionio.data.storage.Storage$.getMetaDataApps(Storage.scala:387)
at
org.apache.predictionio.data.store.Common$.appsDb$lzycompute(Common.scala:27)
at org.apache.predictionio.data.store.Common$.appsDb(Common.scala:27)
at
org.apache.predictionio.data.store.Common$.appNameToId(Common.scala:32)
at
org.apache.predictionio.data.store.PEventStore$.find(PEventStore.scala:71)
at
$line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:28)
at $line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:33)
at $line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:35)
at $line19.$read$$iwC$$iwC$$iwC$$iwC$$iwC.(:37)
at $line19.$read$$iwC$$iwC$$iwC$$iwC.(:39)
at $line19.$read$$iwC$$iwC$$iwC.(:41)
at $line19.$read$$iwC$$iwC.(:43)
at $line19.$read$$iwC.(:45)
at $line19.$read.(:47)
at $line19.$read$.(:51)
at $line19.$read$.()
at $line19.$eval$.(:7)
at $line19.$eval$.()
at $line19.$eval.$print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at