Hi team, I have been working with Mars to get a little more insight to an error I am seeing. After narrowing down the issue, I am trying to find a way to force the order of loading the jars. I am a novice in Scala and am looking for a way to continue leveraging PredictionIO on Heroku but have run into a roadblock as I believe ordering the jars would be a change to PredictionIO itself. I would greatly appreciate your expertise and guidance.
Error: Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V 2018-03-02T02:49:42.245083+00:00 app[release.7253]: at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) 2018-03-02T02:49:42.245086+00:00 app[release.7253]: at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) 2018-03-02T02:49:42.245087+00:00 app[release.7253]: at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) 2018-03-02T02:49:42.245088+00:00 app[release.7253]: at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) 2018-03-02T02:49:42.245090+00:00 app[release.7253]: at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) Context: - I am using the predictionIO buildpack on heroku and using version 12. - I am using S3 to Persist the model objects as I am using the PAlgorithm vs L2LAlgorithm. - I see an intermittent error when the Algorithm.scala file tries to connect to S3. Sometimes it works and sometime is does not. - When it fails I get the message above - I think I have narrowed this down to the ordering of loading the jars. *I have done a number of tests and can see that the problem always occurs with a certain order of jars. This issue appears to be intermittent and could be solved by forcing the order of the jars. I am assuming this could be a straightforward fix, * *I believe it is in the compute-classpath.sh. Perhaps it's something I can even do on my end as an end user but I don't believe that to be the case.* *Please let me know if this is something that can be changed and committed or if I can contribute in someway. Again, I am fairly new to Scala and big data infrastructure so I am intimidated by the thought of making the change myself but we are at a roadblock:)* *When aws-java-sdk.jar is loaded before (hadoop-aws.jar and pio-data-s3-assembly-0.12.0-incubating.jar) it is always successful. If it is loaded after it fails with the message above.* *Success Examples:* --jars file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar, ... --jars file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar, file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, ... *Failure Examples:* --jars file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar, file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, ... --jars file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar, file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, ... Thank you again for your help! *Shane Johnson | LIFT IQ* *Founder | CEO* *www.liftiq.com <http://www.liftiq.com/>* or *sh...@liftiq.com <sh...@liftiq.com>* mobile: (801) 360-3350 LinkedIn <https://www.linkedin.com/in/shanewjohnson/> | Twitter <https://twitter.com/SWaldenJ> | Facebook <https://www.facebook.com/shane.johnson.71653>