Hi Bansari, All you need to make sure is that wherever you run build, train, and deploy, all nodes have the same data storage configuration and can access those storage.
"scratch-uri" helps you properly run any "pio" commands in YARN cluster mode, and is otherwise unrelated with the above. Regards, Donald On Tue, Oct 4, 2016 at 9:37 PM, Bansari Shah <[email protected]> wrote: > Thank you for your guidance. > > We have to build and train on development machine which can be standalone > or 3 node cluster and deploy on production environment which is completely > different cluster. In this case does scratch-uri will work or we have to > follow another process. > > Please suggest me. > > Thank you > Regards, > Bansari > > On Wed, Oct 5, 2016 at 1:30 AM, Donald Szeto <[email protected]> wrote: > >> Hi Bansari, >> >> The --scratch-uri switch is only useful with "pio train/deploy" using >> YARN cluster mode, which is your case. It tells PredictionIO where to copy >> PredictionIO JARs and engine.json for YARN cluster mode to work properly. >> >> 1. Make sure HADOOP_CONF_DIR is set properly in conf/pio-env.sh. >> 2. Provide an HDFS URL to --scratch-uri. You need to have write access to >> this location. >> >> Regards, >> Donald >> >> On Tue, Oct 4, 2016 at 11:21 AM, Pat Ferrel <[email protected]> >> wrote: >> >>> No idea about 'scratch-uri’ but once you build a model if you have >>> specified (in pio-env.sh) that pio use hdfs for the model storage it will >>> already be available to any machine that has access to hdfs. It somewhat >>> depends on the template, the Universal Recommender uses Elasticsearch for >>> model storage so any machine with access to ES will have the model. >>> >>> >>> On Oct 4, 2016, at 10:09 AM, Bansari Shah <[email protected]> >>> wrote: >>> >>> Hi Donald, >>> >>> I am running my spark cluster of 3 node with YARN and spark driver is >>> within cluster. >>> >>> Thanks >>> Regards, >>> Bansari >>> >>> On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <[email protected]> wrote: >>> >>>> Hi Bansari, >>>> >>>> How are you running your Spark cluster? Standalone, YARN, or Mesos? Are >>>> you running the Spark driver on the client or within the cluster? >>>> >>>> Regards, >>>> Donald >>>> >>>> On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> Can you please guide me how to use 'scratch-uri' argument in case of >>>>> transferring all necessary files to remote location. >>>>> >>>>> And can you please suggest me any way for deploying model on remote >>>>> location which is trained and build on other machine. >>>>> >>>>> Please consider it. >>>>> >>>>> Thank you, >>>>> >>>>> Regards, >>>>> Bansari >>>>> >>>> >>>> >>> >>> >> >
