Can you show the relevant parts from pio.log, please? If you don't care about existing log messages, the easiest way would be to delete pio.log from where you run the pio command and start fresh.
On Fri, Mar 31, 2017 at 8:46 AM, Malay Tripathi <[email protected]> wrote: > I think it's Yarn based, setup through Ambari. > > > On Mar 31, 2017, at 6:29 AM, Donald Szeto <[email protected]> wrote: > > Hi Malay, > > Is your Spark cluster a standalone deployment or based on YARN? > > Regards, > Donald > > On Thu, Mar 30, 2017 at 11:48 PM Malay Tripathi <[email protected]> > wrote: > >> Hello, >> >> I am running pio train *on an edge node* of distributed 8 node spark >> cluster & 3 node Hbase. >> When I run "pio train" the job runs but it runs on local spark & not >> submitted to cluster. >> If I do "pio train *--master spark://localhost:7077" *or "pio train *--master >> yarn-cluster" *I get below error - >> >> * File does not exist: >> hdfs://mdc2vra179.federated.fds:8020/user/da_mcom_milan/.sparkStaging/application_1489598450058_0024/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar* >> >> *java.io.FileNotFoundException: File does not exist: >> hdfs://mdc2vra179.federated.fds:8020/user/da_mcom_milan/.sparkStaging/application_1489598450058_0024/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar* >> >> *at >> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1427)* >> >> *at >> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)* >> >> >> mdc2vra179 is my Hbase cluster node, also running Namenode. Not sure why >> my spark expecting a jar file on hbase/Namenode. >> *$PIO_HOME/conf/pio-env.sh-* >> >> SPARK_HOME=$PIO_HOME/vendors/spark >> >> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase/conf >> >> PIO_FS_BASEDIR=$HOME/.pio_store >> >> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines >> >> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp >> >> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta >> >> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH >> >> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event >> >> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE >> >> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model >> >> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS >> >> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-prod >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176 >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 >> >> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models >> >> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase >> >> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase >> >> >> Thanks, >> >> Malay >> >
