Thanks for the response Donald. Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you are using?
I do not know what version is being used. Is this something that I can specify or control? I am using the PredictionIO buildpack https://github.com/heroku/predictionio-buildpack. I am *not* specifying these in my build.sbt currently. You are also right that you can modify the class path in bin/compute-classpath.sh as a short term fix. The current order is following the output of your target system's `ls`, so the order is not guaranteed like you speak. Right before the last line (echo $CLASSPATH), you can add a line to make sure that the JAR you want to be loaded first is at the very beginning. I believe this would need to be something that is edited on master ( https://github.com/apache/predictionio) as the buildpack leverages it vs. a cloned version of the code that I can edit, am I thinking through that correctly? I may need to circle back with Mars to see if there are any other options to get this to work with Heroku. Is this something that can be committed to master? The aws-java-sdk.jar is the one that we need to load first. Best, Shane *Shane Johnson | LIFT IQ* *Founder | CEO* *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com <sh...@liftiq.com>* mobile: (801) 360-3350 LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter <https://twitter.com/SWaldenJ> | Facebook <https://www.facebook.com/shane.johnson.71653> On Tue, Mar 6, 2018 at 4:08 PM, Donald Szeto <don...@apache.org> wrote: > Hi Shane, > > Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you > are using? > > You are also right that you can modify the class path in > bin/compute-classpath.sh as a short term fix. The current order is > following the output of your target system's `ls`, so the order is not > guaranteed like you speak. Right before the last line (echo $CLASSPATH), > you can add a line to make sure that the JAR you want to be loaded first is > at the very beginning. > > The long term fix would be matching the hadoop-aws.jar, aws-java-sdk.jar, > and PIO's S3 storage JAR version to have a binary compatible transitive > dependency tree. This could be quite time consuming though. > > Regards, > Donald > > On Mon, Mar 5, 2018 at 6:57 AM, Shane Johnson < > shanewaldenjohn...@gmail.com> wrote: > >> Hi team, >> >> I have been working with Mars to get a little more insight to an error I >> am seeing. After narrowing down the issue, I am trying to find a way to >> force the order of loading the jars. I am a novice in Scala and am looking >> for a way to continue leveraging PredictionIO on Heroku but have run into a >> roadblock as I believe ordering the jars would be a change to PredictionIO >> itself. I would greatly appreciate your expertise and guidance. >> >> Error: >> >> Exception >> >> in thread "main" java.lang.NoSuchMethodError: >> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V >> >> 2018-03-02T02:49:42.245083+00:00 app[release.7253]: at >> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) >> >> 2018-03-02T02:49:42.245086+00:00 app[release.7253]: at >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) >> >> 2018-03-02T02:49:42.245087+00:00 app[release.7253]: at >> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) >> >> 2018-03-02T02:49:42.245088+00:00 app[release.7253]: at >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) >> >> 2018-03-02T02:49:42.245090+00:00 app[release.7253]: at >> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) >> >> >> Context: >> >> - I am using the predictionIO buildpack on heroku and using version >> 12. >> - I am using S3 to Persist the model objects as I am using the >> PAlgorithm vs L2LAlgorithm. >> - I see an intermittent error when the Algorithm.scala file tries to >> connect to S3. Sometimes it works and sometime is does not. >> - When it fails I get the message above - I think I have narrowed >> this down to the ordering of loading the jars. >> >> *I have done a number of tests and can see that the problem always occurs >> with a certain order of jars. This issue appears to be intermittent and >> could be solved by forcing the order of the jars. I am assuming this could >> be a straightforward fix, * >> *I believe it is in the compute-classpath.sh. Perhaps it's something I >> can even do on my end as an end user but I don't believe that to be the >> case.* >> >> *Please let me know if this is something that can be changed and >> committed or if I can contribute in someway. Again, I am fairly new to >> Scala and big data infrastructure so I am intimidated by the thought of >> making the change myself but we are at a roadblock:)* >> >> *When aws-java-sdk.jar is loaded before >> (hadoop-aws.jar and pio-data-s3-assembly-0.12.0-incubating.jar) it is >> always successful. If it is loaded after it fails with the message above.* >> >> *Success Examples:* >> >> --jars >> file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, >> file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, >> file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0 >> .12.0-incubating.jar, >> ... >> >> --jars >> file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, >> file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0 >> .12.0-incubating.jar, >> file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, >> ... >> >> *Failure Examples:* >> >> --jars >> file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, >> file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0 >> .12.0-incubating.jar, >> file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, >> ... >> >> --jars >> file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0 >> .12.0-incubating.jar, >> file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, >> file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, >> ... >> >> >> Thank you again for your help! >> >> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >> <https://www.facebook.com/shane.johnson.71653> >> > >
