Mars, to test what may be happening I just reverted to the original
buildpack (https://github.com/heroku/predictionio-buildpack.git) and
removed the variable PREDICTIONIO_DIST_URL and I am still getting the same
error. I don't know where I would have introduced this. Have you seen it
before, perhaps this is what is causing the custom dist to not be used? I'm
guessing the build failed before and I was overlooking that. Now I need to
find out why the build is failing:)
Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
[INFO] [Engine$] Using command
'/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt'
at
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
to build.
[INFO] [Engine$] If the path above is incorrect, this process will fail.
[INFO] [Engine$] Uber JAR disabled. Making sure
lib/pio-assembly-0.12.0-incubating.jar is absent.
[INFO] [Engine$] Going to run:
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
package assemblyPackageDependency in
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
[ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
[ERROR] [Engine$] [error] Not a valid key:
assemblyPackageDependency (similar: sbtDependency)
[ERROR] [Engine$] [error] assemblyPackageDependency
[ERROR] [Engine$] [error] ^
[ERROR] [Engine$] Return code of build command:
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
package assemblyPackageDependency is 1. Aborting.
! Push rejected, failed to compile PredictionIO app.
! Push failed
*Shane Johnson | LIFT IQ*
*Founder | CEO*
*www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter
<https://twitter.com/SWaldenJ> | Facebook
<https://www.facebook.com/shane.johnson.71653>
On Fri, Mar 9, 2018 at 11:35 AM, Shane Johnson <sh...@liftiq.com> wrote:
> It looks like the URL is the correct URL of the custom PredictionIO dist.
> It looks like there is another error that might be occurring here when I
> went to deploy.
>
> -----> JVM Common app detected
>
> -----> Installing JDK 1.8... done
>
> -----> PredictionIO app detected
>
> -----> Install core components
>
> *+ PredictionIO
> (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz
>
> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)*
>
> + Spark (spark-2.1.1-bin-hadoop2.7)
>
> -----> Install supplemental components
>
> + PostgreSQL (JDBC)
>
> + S3 HDFS (AWS SDK)
>
> + S3 HDFS (Hadoop-AWS)
>
> Writing default 'core-site.xml.erb'
>
> + local Maven repo from buildpack (contents)
>
> -----> Configure PredictionIO
>
> Writing default 'pio-env.sh'
>
> Writing default 'spark-defaults.conf.erb'
>
> + Maven repo from buildpack (build.sbt entry)
>
> Set-up environment via '.profile.d/' scripts
>
> -----> Install JVM (heroku/jvm-common)
>
> -----> PredictionIO engine
>
> Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>
> [INFO] [Engine$] Using command
> '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt'
> at
> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
> to build.
>
> [INFO] [Engine$] If the path above is incorrect, this process will
> fail.
>
> [INFO] [Engine$] Uber JAR disabled. Making sure
> lib/pio-assembly-0.12.0-incubating.jar is absent.
>
> [INFO] [Engine$] Going to run:
> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
> package assemblyPackageDependency in
> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>
> [ERROR] [Engine$] [error] Not a valid command:
> assemblyPackageDependency
>
> [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency
> (similar: sbtDependency)
>
> [ERROR] [Engine$] [error] assemblyPackageDependency
>
> [ERROR] [Engine$] [error] ^
>
> [ERROR] [Engine$] Return code of build command:
> /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
> package assemblyPackageDependency is 1. Aborting.
>
> ! Push rejected, failed to compile PredictionIO app.
>
> ! Push failed
>
>
>
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter
> <https://twitter.com/SWaldenJ> | Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <mars.h...@salesforce.com>
> wrote:
>
>> I'm lost as to how such direct manipulation of CLASSPATH is not appearing
>> in the logged spark-submit command.
>>
>> What could cause this!?
>>
>> I just pushed a version of the buildpack which should help debug.
>> Assuming only a single buildpack is assigned to the app, here's how to set
>> it:
>>
>> heroku buildpacks:set https://github.com/heroku/pred
>> ictionio-buildpack#debug-custom-dist
>>
>> Then redeploy the engine an check the build log for the line:
>>
>> + PredictionIO ($URL)
>>
>> Please confirm that it is the URL of your custom PredictionIO dist.
>>
>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>> Thanks Donald and Mars,
>>>
>>> I created a new distribution (
>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incu
>>> bating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the added
>>> CLASSPATH code and pointed to the distribution with
>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>
>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>>> echo "$CLASSPATH"
>>>
>>> It didn't seem to force the aws-java-sdk to load first as I reviewed the
>>> release logs. Should the aws-java-sdk.jar show up as the first file within
>>> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
>>> lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>
>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>> other suggestions to try? I was also testing locally to change the order of
>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>> the order of the jars in the logs.
>>>
>>> Running train on release…
>>>
>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>
>>> [INFO] [Runner$] Submission command:
>>> /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit
>>> --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow
>>> --jars
>>> file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar*
>>> --files
>>> file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml
>>> --driver-class-path
>>> /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf
>>> --driver-java-options -Dpio.log.dir=/app
>>> file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar
>>> --engine-id org.template.liftscoring.LiftScoringEngine --engine-version
>>> 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant
>>> file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>
>>>
>>> Error:
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>
>>> at
>>> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>
>>>
>>>
>>>
>>>
>>> *Shane Johnson | LIFT IQ*
>>> *Founder | CEO*
>>>
>>> *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com
>>> <sh...@liftiq.com>*
>>> mobile: (801) 360-3350
>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter
>>> <https://twitter.com/SWaldenJ> | Facebook
>>> <https://www.facebook.com/shane.johnson.71653>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <mars.h...@salesforce.com>
>>> wrote:
>>>
>>>> Shane,
>>>>
>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>>>
>>>>>
>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>> are referring to...(line at the bottom in red)?
>>>>>
>>>>
>>>>
>>>> I believe the code would need to look like this to effect the output
>>>> classpath as intended:
>>>>
>>>>
>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>> :$CLASSPATH"
>>>>> echo "$CLASSPATH"
>>>>>
>>>>
>>>>
>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>> will need to be skip or remove it first.
>>>>
>>>> --
>>>> *Mars Hall
>>>> 415-818-7039 <(415)%20818-7039>
>>>> Customer Facing Architect
>>>> Salesforce Platform / Heroku
>>>> San Francisco, California
>>>>
>>>
>>>
>>
>>
>> --
>> *Mars Hall
>> 415-818-7039 <(415)%20818-7039>
>> Customer Facing Architect
>> Salesforce Platform / Heroku
>> San Francisco, California
>>
>
>
