It looks like the URL is the correct URL of the custom PredictionIO dist. It looks like there is another error that might be occurring here when I went to deploy.
-----> JVM Common app detected -----> Installing JDK 1.8... done -----> PredictionIO app detected -----> Install core components *+ PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)* + Spark (spark-2.1.1-bin-hadoop2.7) -----> Install supplemental components + PostgreSQL (JDBC) + S3 HDFS (AWS SDK) + S3 HDFS (Hadoop-AWS) Writing default 'core-site.xml.erb' + local Maven repo from buildpack (contents) -----> Configure PredictionIO Writing default 'pio-env.sh' Writing default 'spark-defaults.conf.erb' + Maven repo from buildpack (build.sbt entry) Set-up environment via '.profile.d/' scripts -----> Install JVM (heroku/jvm-common) -----> PredictionIO engine Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.) [INFO] [Engine$] Using command '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build. [INFO] [Engine$] If the path above is incorrect, this process will fail. [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent. [INFO] [Engine$] Going to run: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt package assemblyPackageDependency in /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency) [ERROR] [Engine$] [error] assemblyPackageDependency [ERROR] [Engine$] [error] ^ [ERROR] [Engine$] Return code of build command: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt package assemblyPackageDependency is 1. Aborting. ! Push rejected, failed to compile PredictionIO app. ! Push failed *Shane Johnson | LIFT IQ* *Founder | CEO* *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com <sh...@liftiq.com>* mobile: (801) 360-3350 LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter <https://twitter.com/SWaldenJ> | Facebook <https://www.facebook.com/shane.johnson.71653> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <mars.h...@salesforce.com> wrote: > I'm lost as to how such direct manipulation of CLASSPATH is not appearing > in the logged spark-submit command. > > What could cause this!? > > I just pushed a version of the buildpack which should help debug. Assuming > only a single buildpack is assigned to the app, here's how to set it: > > heroku buildpacks:set https://github.com/heroku/ > predictionio-buildpack#debug-custom-dist > > Then redeploy the engine an check the build log for the line: > > + PredictionIO ($URL) > > Please confirm that it is the URL of your custom PredictionIO dist. > > On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote: > >> Thanks Donald and Mars, >> >> I created a new distribution ( >> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz> >> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0- >> incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the >> added CLASSPATH code and pointed to the distribution with >> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku. >> >> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH" >> echo "$CLASSPATH" >> >> It didn't seem to force the aws-java-sdk to load first as I reviewed the >> release logs. Should the aws-java-sdk.jar show up as the first file within >> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/ >> lib/spark/aws-java-sdk.jar:$CLASSPATH". >> >> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads >> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have >> other suggestions to try? I was also testing locally to change the order of >> the --jars but changes to the compute-classpath.sh didn't seem to change >> the order of the jars in the logs. >> >> Running train on releaseā¦ >> >> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8 >> >> [INFO] [Runner$] Submission command: >> /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory >> 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars >> file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* >> --files >> file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml >> --driver-class-path >> /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf >> --driver-java-options -Dpio.log.dir=/app >> file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar >> --engine-id org.template.liftscoring.LiftScoringEngine --engine-version >> 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant >> file:/app/engine.json --verbosity 0 --json-extractor Both --env >> >> >> Error: >> >> Exception in thread "main" java.lang.NoSuchMethodError: >> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V >> >> at >> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) >> >> at >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) >> >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) >> >> >> >> >> >> *Shane Johnson | LIFT IQ* >> *Founder | CEO* >> >> *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com >> <sh...@liftiq.com>* >> mobile: (801) 360-3350 >> LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter >> <https://twitter.com/SWaldenJ> | Facebook >> <https://www.facebook.com/shane.johnson.71653> >> >> >> >> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <mars.h...@salesforce.com> >> wrote: >> >>> Shane, >>> >>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote: >>> >>>> >>>> Re: adding a line to ensure a jar is loaded first. Is this what you are >>>> referring to...(line at the bottom in red)? >>>> >>> >>> >>> I believe the code would need to look like this to effect the output >>> classpath as intended: >>> >>> >>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar >>>> :$CLASSPATH" >>>> echo "$CLASSPATH" >>>> >>> >>> >>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will >>> need to be skip or remove it first. >>> >>> -- >>> *Mars Hall >>> 415-818-7039 <(415)%20818-7039> >>> Customer Facing Architect >>> Salesforce Platform / Heroku >>> San Francisco, California >>> >> >> > > > -- > *Mars Hall > 415-818-7039 <(415)%20818-7039> > Customer Facing Architect > Salesforce Platform / Heroku > San Francisco, California >