Thanks Mars, trying to get back to a stable state and then I will respond with what I find re: Classpath. Here is some context in the meantime. I was able to deploy to Heroku again. Not sure how I introduced that error, perhaps what Donald said.
> Does this error occur locally when you `pio build` this engine? The error does not occur locally. Are there any diffs in your local code? There was not any diffs in my local code. I am trying to find a previous commit to see if I can deploy it successfully. I'm in git land right now that is why I haven't gotten back to you until now:) I was able to get a older version of the code to deploy to heroku so I think I will be able to solve deploying and then I will resume were we left off on confirming my URL is pointing to the correct distribution. How are you running locally? Are you using the buildpack's "local dev" > setup? Yes, I am using the buildpacks "local dev" setup. https://github.com/heroku/predictionio-buildpack/blob/master/DEV.md *Shane Johnson | LIFT IQ* *Founder | CEO* *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com <sh...@liftiq.com>* mobile: (801) 360-3350 LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter <https://twitter.com/SWaldenJ> | Facebook <https://www.facebook.com/shane.johnson.71653> On Fri, Mar 9, 2018 at 12:59 PM, Donald Szeto <don...@apache.org> wrote: > This error looks like pio build was not run at the engine template > directory, which would have the proper configuration to enable that command. > > On Fri, Mar 9, 2018 at 11:28 AM Mars Hall <mars.h...@salesforce.com> > wrote: > >> At this point, I'm just searching the internet for "Not a valid command: >> assemblyPackageDependency" errors, which I image you are too. >> >> Does this error occur locally when you `pio build` this engine? >> >> Are there any diffs in your local code? >> >> How are you running locally? Are you using the buildpack's "local dev" >> setup? >> >> On Fri, Mar 9, 2018 at 6:48 PM, Shane Johnson <sh...@liftiq.com> wrote: >> >>> Mars, to test what may be happening I just reverted to the original >>> buildpack (https://github.com/heroku/predictionio-buildpack.git) and >>> removed the variable PREDICTIONIO_DIST_URL and I am still getting the >>> same error. I don't know where I would have introduced this. Have you seen >>> it before, perhaps this is what is causing the custom dist to not be used? >>> I'm guessing the build failed before and I was overlooking that. Now I need >>> to find out why the build is failing:) >>> >>> Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.) >>> >>> [INFO] [Engine$] Using command >>> '/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' >>> at >>> /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e >>> to build. >>> >>> [INFO] [Engine$] If the path above is incorrect, this process will >>> fail. >>> >>> [INFO] [Engine$] Uber JAR disabled. Making sure >>> lib/pio-assembly-0.12.0-incubating.jar is absent. >>> >>> [INFO] [Engine$] Going to run: >>> /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt >>> package assemblyPackageDependency in >>> /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e >>> >>> [ERROR] [Engine$] [error] Not a valid command: >>> assemblyPackageDependency >>> >>> [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency >>> (similar: sbtDependency) >>> >>> [ERROR] [Engine$] [error] assemblyPackageDependency >>> >>> [ERROR] [Engine$] [error] ^ >>> >>> [ERROR] [Engine$] Return code of build command: >>> /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt >>> package assemblyPackageDependency is 1. Aborting. >>> >>> ! Push rejected, failed to compile PredictionIO app. >>> >>> ! Push failed >>> >>> >>> >>> >>> *Shane Johnson | LIFT IQ* >>> *Founder | CEO* >>> >>> *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com >>> <sh...@liftiq.com>* >>> mobile: (801) 360-3350 >>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter >>> <https://twitter.com/SWaldenJ> | Facebook >>> <https://www.facebook.com/shane.johnson.71653> >>> >>> >>> >>> On Fri, Mar 9, 2018 at 11:35 AM, Shane Johnson <sh...@liftiq.com> wrote: >>> >>>> It looks like the URL is the correct URL of the custom PredictionIO >>>> dist. It looks like there is another error that might be occurring here >>>> when I went to deploy. >>>> >>>> -----> JVM Common app detected >>>> >>>> -----> Installing JDK 1.8... done >>>> >>>> -----> PredictionIO app detected >>>> >>>> -----> Install core components >>>> >>>> *+ PredictionIO >>>> (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz >>>> >>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)* >>>> >>>> + Spark (spark-2.1.1-bin-hadoop2.7) >>>> >>>> -----> Install supplemental components >>>> >>>> + PostgreSQL (JDBC) >>>> >>>> + S3 HDFS (AWS SDK) >>>> >>>> + S3 HDFS (Hadoop-AWS) >>>> >>>> Writing default 'core-site.xml.erb' >>>> >>>> + local Maven repo from buildpack (contents) >>>> >>>> -----> Configure PredictionIO >>>> >>>> Writing default 'pio-env.sh' >>>> >>>> Writing default 'spark-defaults.conf.erb' >>>> >>>> + Maven repo from buildpack (build.sbt entry) >>>> >>>> Set-up environment via '.profile.d/' scripts >>>> >>>> -----> Install JVM (heroku/jvm-common) >>>> >>>> -----> PredictionIO engine >>>> >>>> Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.) >>>> >>>> [INFO] [Engine$] Using command >>>> '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' >>>> at >>>> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e >>>> to build. >>>> >>>> [INFO] [Engine$] If the path above is incorrect, this process will >>>> fail. >>>> >>>> [INFO] [Engine$] Uber JAR disabled. Making sure >>>> lib/pio-assembly-0.12.0-incubating.jar is absent. >>>> >>>> [INFO] [Engine$] Going to run: >>>> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt >>>> package assemblyPackageDependency in >>>> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e >>>> >>>> [ERROR] [Engine$] [error] Not a valid command: >>>> assemblyPackageDependency >>>> >>>> [ERROR] [Engine$] [error] Not a valid key: >>>> assemblyPackageDependency (similar: sbtDependency) >>>> >>>> [ERROR] [Engine$] [error] assemblyPackageDependency >>>> >>>> [ERROR] [Engine$] [error] ^ >>>> >>>> [ERROR] [Engine$] Return code of build command: >>>> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt >>>> package assemblyPackageDependency is 1. Aborting. >>>> >>>> ! Push rejected, failed to compile PredictionIO app. >>>> >>>> ! Push failed >>>> >>>> >>>> >>>> >>>> *Shane Johnson | LIFT IQ* >>>> *Founder | CEO* >>>> >>>> *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com >>>> <sh...@liftiq.com>* >>>> mobile: (801) 360-3350 >>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter >>>> <https://twitter.com/SWaldenJ> | Facebook >>>> <https://www.facebook.com/shane.johnson.71653> >>>> >>>> >>>> >>>> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <mars.h...@salesforce.com> >>>> wrote: >>>> >>>>> I'm lost as to how such direct manipulation of CLASSPATH is not >>>>> appearing in the logged spark-submit command. >>>>> >>>>> What could cause this!? >>>>> >>>>> I just pushed a version of the buildpack which should help debug. >>>>> Assuming only a single buildpack is assigned to the app, here's how to set >>>>> it: >>>>> >>>>> heroku buildpacks:set https://github.com/heroku/ >>>>> predictionio-buildpack#debug-custom-dist >>>>> >>>>> Then redeploy the engine an check the build log for the line: >>>>> >>>>> + PredictionIO ($URL) >>>>> >>>>> Please confirm that it is the URL of your custom PredictionIO dist. >>>>> >>>>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> >>>>> wrote: >>>>> >>>>>> Thanks Donald and Mars, >>>>>> >>>>>> I created a new distribution ( >>>>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz> >>>>>> https://s3-us-west-1.amazonaws.com/predictionio/0. >>>>>> 12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz) >>>>>> with the added CLASSPATH code and pointed to the distribution with >>>>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku. >>>>>> >>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar >>>>>> :$CLASSPATH" >>>>>> echo "$CLASSPATH" >>>>>> >>>>>> It didn't seem to force the aws-java-sdk to load first as I reviewed >>>>>> the release logs. Should the aws-java-sdk.jar show up as the first file >>>>>> within the --jars section when this is added CLASSPATH="/app/ >>>>>> PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH". >>>>>> >>>>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads >>>>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you >>>>>> have other suggestions to try? I was also testing locally to change the >>>>>> order of the --jars but changes to the compute-classpath.sh didn't seem >>>>>> to >>>>>> change the order of the jars in the logs. >>>>>> >>>>>> Running train on release… >>>>>> >>>>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8 >>>>>> >>>>>> [INFO] [Runner$] Submission command: >>>>>> /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit >>>>>> --driver-memory 13g --class >>>>>> org.apache.predictionio.workflow.CreateWorkflow --jars >>>>>> file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* >>>>>> --files >>>>>> file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml >>>>>> --driver-class-path >>>>>> /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf >>>>>> --driver-java-options -Dpio.log.dir=/app >>>>>> file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar >>>>>> --engine-id org.template.liftscoring.LiftScoringEngine --engine-version >>>>>> 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant >>>>>> file:/app/engine.json --verbosity 0 --json-extractor Both --env >>>>>> >>>>>> >>>>>> Error: >>>>>> >>>>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>>>> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V >>>>>> >>>>>> at >>>>>> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) >>>>>> >>>>>> at >>>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) >>>>>> >>>>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Shane Johnson | LIFT IQ* >>>>>> *Founder | CEO* >>>>>> >>>>>> *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com >>>>>> <sh...@liftiq.com>* >>>>>> mobile: (801) 360-3350 >>>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter >>>>>> <https://twitter.com/SWaldenJ> | Facebook >>>>>> <https://www.facebook.com/shane.johnson.71653> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <mars.h...@salesforce.com> >>>>>> wrote: >>>>>> >>>>>>> Shane, >>>>>>> >>>>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you >>>>>>>> are referring to...(line at the bottom in red)? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> I believe the code would need to look like this to effect the output >>>>>>> classpath as intended: >>>>>>> >>>>>>> >>>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar >>>>>>>> :$CLASSPATH" >>>>>>>> echo "$CLASSPATH" >>>>>>>> >>>>>>> >>>>>>> >>>>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script >>>>>>> will need to be skip or remove it first. >>>>>>> >>>>>>> -- >>>>>>> *Mars Hall >>>>>>> 415-818-7039 <(415)%20818-7039> >>>>>>> Customer Facing Architect >>>>>>> Salesforce Platform / Heroku >>>>>>> San Francisco, California >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Mars Hall >>>>> 415-818-7039 <(415)%20818-7039> >>>>> Customer Facing Architect >>>>> Salesforce Platform / Heroku >>>>> San Francisco, California >>>>> >>>> >>>> >>> >> >> >> -- >> *Mars Hall >> 415-818-7039 <(415)%20818-7039> >> Customer Facing Architect >> Salesforce Platform / Heroku >> San Francisco, California >> >
