Hi guys, I have another error that I cannot get past:
vagrant:MyRecommendation/ (develop*) $ pio train [21:47:39] [ERROR] [Console$] unable to read engine factory from /home/vagrant/MyRecommendation/engine.json. Aborting. Does anyone have any advice? Thanks, Jim -- Jim Miller On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1...@gmail.com) wrote: Shinsuke, you rock! That was it. Thank you so much, Jim -- Jim Miller On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (shinsuke.sug...@gmail.com) wrote: Hi, > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 Elasticsearch 5 uses 9200 port. Could you try 9200 port? Thanks, shinsuke 2017-09-24 3:02 GMT+09:00 Jim Miller <jemiller1...@gmail.com>: > Hi Donald, > > Tried just now and received the following error: > > vagrant:~/ $ pio status > [13:34:52] > [INFO] [Management$] Inspecting PredictionIO... > [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at > /home/vagrant/pio/PredictionIO-0.12.0-incubating > [INFO] [Management$] Inspecting Apache Spark... > [INFO] [Management$] Apache Spark is installed at > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7 > [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement > of 1.3.0) > [INFO] [Management$] Inspecting storage backend connections... > [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)... > [ERROR] [Management$] Unable to connect to all storage backends > successfully. > The following shows the error message from the storage backend. > > error while performing request (java.lang.RuntimeException) > > Dumping configuration of initialized storage backend sources. > Please make sure they are correct. > > Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2, > HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE -> > elastic search > > > HERE IS MY PIO-ENV.SH > > # PredictionIO Main Configuration > # > # This section controls core behavior of PredictionIO. It is very likely > that > # you need to change these to fit your site. > > # SPARK_HOME: Apache Spark is a hard dependency and must be configured. > # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7 > SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7 > > POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar > MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar > > # ES_CONF_DIR: You must configure this if you have advanced configuration > for > # your Elasticsearch setup. > ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2 > # HADOOP_CONF_DIR=/opt/hadoop > > # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO > # with HBase on a remote cluster. > HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf > > # Filesystem paths where PredictionIO uses as block storage. > PIO_FS_BASEDIR=$HOME/.pio_store > PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines > PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp > > # PredictionIO Storage Configuration > # > # This section controls programs that make use of PredictionIO's built-in > # storage facilities. Default values are shown below. > # > # For more information on storage configuration please refer to > # http://predictionio.incubator.apache.org/system/anotherdatastore/ > > # Storage Repositories > > # Default is to use PostgreSQL > PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta > PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH > > PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event > PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE > > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model > PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS > > # Storage Data Sources > > # PostgreSQL Default Settings > # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL > # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly > # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc > # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio > # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio > > # MySQL Example > # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc > # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio > # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio > # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio > > # Elasticsearch Example > # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch > # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200 > # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http > # > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2 > # Optional basic HTTP auth > # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret > # Elasticsearch 1.x Example > PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch > PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2 > > # Local File System Example > PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs > PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models > > # HBase Example > PIO_STORAGE_SOURCES_HBASE_TYPE=hbase > PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1 > > # AWS S3 Example > # PIO_STORAGE_SOURCES_S3_TYPE=s3 > # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket > # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model > > > ELASTICSEARCH.YML > > #======================== Elasticsearch Configuration > ========================= > # > # NOTE: Elasticsearch comes with reasonable defaults for most settings. > # Before you set out to tweak and tune the configuration, make sure > you > # understand what are you trying to accomplish and the consequences. > # > # The primary way of configuring a node is via this file. This template > lists > # the most important settings you may want to configure for a production > cluster. > # > # Please consult the documentation for further information on configuration > options: > # https://www.elastic.co/guide/en/elasticsearch/reference/index.html > # > # ---------------------------------- Cluster > ----------------------------------- > # > # Use a descriptive name for your cluster: > # > cluster.name: firstCluster > # > # ------------------------------------ Node > ------------------------------------ > # > # Use a descriptive name for the node: > # > #node.name: node-1 > # > # Add custom attributes to the node: > # > #node.attr.rack: r1 > # > # ----------------------------------- Paths > ------------------------------------ > # > # Path to directory where to store the data (separate multiple locations by > comma): > # > #path.data: /path/to/data > # > # Path to log files: > # > #path.logs: /path/to/logs > # > # ----------------------------------- Memory > ----------------------------------- > # > # Lock the memory on startup: > # > #bootstrap.memory_lock: true > # > # Make sure that the heap size is set to about half the memory available > # on the system and that the owner of the process is allowed to use this > # limit. > # > # Elasticsearch performs poorly when the system is swapping the memory. > # > # ---------------------------------- Network > ----------------------------------- > # > # Set the bind address to a specific IP (IPv4 or IPv6): > # > #network.host: 192.168.0.1 > # > # Set a custom port for HTTP: > # > #http.port: 9200 > # > # For more information, consult the network module documentation. > # > # --------------------------------- Discovery > ---------------------------------- > # > # Pass an initial list of hosts to perform discovery when new node is > started: > # The default list of hosts is ["127.0.0.1", "[::1]"] > # > #discovery.zen.ping.unicast.hosts: ["host1", "host2"] > # > # Prevent the "split brain" by configuring the majority of nodes (total > number of master-eligible nodes / 2 + 1): > # > #discovery.zen.minimum_master_nodes: 3 > # > # For more information, consult the zen discovery module documentation. > # > # ---------------------------------- Gateway > ----------------------------------- > # > # Block initial recovery after a full cluster restart until N nodes are > started: > # > #gateway.recover_after_nodes: 3 > # > # For more information, consult the gateway module documentation. > # > # ---------------------------------- Various > ----------------------------------- > # > # Require explicit names when deleting indices: > # > #action.destructive_requires_name: true > > > Thanks for all of your help! > > Jim > -- > Jim Miller > > On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1...@gmail.com) > wrote: > > Hi Donald, > > I did not. I will read the release notes and update accordingly. > > Thanks! > > Jim > > -- > Jim Miller > > On September 20, 2017 at 1:01:53 PM, Donald Szeto (don...@apache.org) wrote: > > Hey Jim, > > Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in > 0.12 so the default build will use ES 5.x support. > > See the upcoming release notes: > https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes > > Regards, > Donald > > On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <jemiller1...@gmail.com> wrote: >> >> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0: >> >> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8 >> >> -- >> Jim Miller >> >> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (p...@occamsmachete.com) >> wrote: >> >> meaning is “firstcluster” the cluster name in your Elasticsearch >> configuration? >> >> >> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <vaghawan...@gmail.com> wrote: >> >> I think the problem is with Elasticsearch, are you sure the cluster exists >> in elasticsearch configuration? >> >> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1...@gmail.com> >> wrote: >>> >>> Hi, >>> >>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase: >>> >>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4 >>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0 >>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6 >>> >>> >>> All starts with no errors but with pio status I get: >>> >>> [INFO] [Management$] Inspecting PredictionIO... >>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating >>> [INFO] [Management$] Inspecting Apache Spark... >>> [INFO] [Management$] Apache Spark is installed at >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6 >>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum >>> requirement of 1.3.0) >>> [INFO] [Management$] Inspecting storage backend connections... >>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)... >>> [ERROR] [Management$] Unable to connect to all storage backends >>> successfully. >>> The following shows the error message from the storage backend. >>> >>> Connection closed >>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException) >>> >>> Dumping configuration of initialized storage backend sources. >>> Please make sure they are correct. >>> >>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4, >>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE -> >>> elasticsearch >>> >>> >>> Can anyone give me an idea of what I need to fix this issue? Here is >>> >>> >>> # PredictionIO Main Configuration >>> # >>> # This section controls core behavior of PredictionIO. It is very likely >>> that >>> # you need to change these to fit your site. >>> >>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured. >>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7 >>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6 >>> >>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar >>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar >>> >>> # ES_CONF_DIR: You must configure this if you have advanced configuration >>> for >>> # your Elasticsearch setup. >>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf >>> >>> # HADOOP_CONF_DIR: You must configure this if you intend to run >>> PredictionIO >>> # with Hadoop 2. >>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf >>> >>> # HBASE_CONF_DIR: You must configure this if you intend to run >>> PredictionIO >>> # with HBase on a remote cluster. >>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf >>> >>> # Filesystem paths where PredictionIO uses as block storage. >>> PIO_FS_BASEDIR=$HOME/.pio_store >>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines >>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp >>> >>> # PredictionIO Storage Configuration >>> # >>> # This section controls programs that make use of PredictionIO's built-in >>> # storage facilities. Default values are shown below. >>> # >>> # For more information on storage configuration please refer to >>> # http://predictionio.incubator.apache.org/system/anotherdatastore/ >>> >>> # Storage Repositories >>> >>> # Default is to use PostgreSQL >>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta >>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH >>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE >>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model >>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS >>> >>> # Storage Data Sources >>> >>> # PostgreSQL Default Settings >>> # Please change "pio" to your database name in >>> PIO_STORAGE_SOURCES_PGSQL_URL >>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly >>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc >>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio >>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio >>> >>> # MySQL Example >>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc >>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio >>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio >>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio >>> >>> # Elasticsearch Example >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200 >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http >>> # >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2 >>> # Optional basic HTTP auth >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret >>> # Elasticsearch 1.x Example >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 >>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4 >>> >>> # Local File System Example >>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models >>> >>> # HBase Example >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase >>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0 >>> >>> # AWS S3 Example >>> # PIO_STORAGE_SOURCES_S3_TYPE=s3 >>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket >>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model >>> >>> >>> Thanks, >>> >>> Jim >>> >>> >>> -- >>> Jim Miller >> >> >> >
