Re: Spark Streaming S3 Error

I got my answer. The way to access S3 has changed. val hadoopConf = sc.hadoopConfiguration hadoopConf.set("fs.s3a.access.key", accessKey) hadoopConf.set("fs.s3a.secret.key", secretKey) val lines = ssc.textFileStream("s3a://amg-events-out/") This worked. Cheers, Ben > On May 21, 2016, at

Re: Spark Streaming S3 Error

Ted, I only see 1 jets3t-0.9.0 jar in the classpath after running this to list the jars. val cl = ClassLoader.getSystemClassLoader cl.asInstanceOf[java.net.URLClassLoader].getURLs.foreach(println) /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/jets3t-0.9.0.jar I don’t know what else

Re: Spark Streaming S3 Error

Maybe more than one version of jets3t-xx.jar was on the classpath. FYI On Fri, May 20, 2016 at 8:31 PM, Benjamin Kim wrote: > I am trying to stream files from an S3 bucket using CDH 5.7.0’s version of > Spark 1.6.0. It seems not to work. I keep getting this error. > >

Spark Streaming S3 Error

I am trying to stream files from an S3 bucket using CDH 5.7.0’s version of Spark 1.6.0. It seems not to work. I keep getting this error. Exception in thread "JobGenerator" java.lang.VerifyError: Bad type on operand stack Exception Details: Location: