JetS3T settings spark
I am not sure , the way I can pass jets3t.properties file for spark-submit. --file option seems not working. can some one please help me. My production spark jobs get hung up when reading s3 file sporadically. Thanks, -D -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JetS3T-settings-spark-tp20916.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: JetS3T settings spark
This file needs to be on your CLASSPATH actually, not just in a directory. The best way to pass it in is probably to package it into your application JAR. You can put it in src/main/resources in a Maven or SBT project, and check that it makes it into the JAR using jar tf yourfile.jar. Matei On Dec 30, 2014, at 4:21 PM, durga durgak...@gmail.com wrote: I am not sure , the way I can pass jets3t.properties file for spark-submit. --file option seems not working. can some one please help me. My production spark jobs get hung up when reading s3 file sporadically. Thanks, -D -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JetS3T-settings-spark-tp20916.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: JetS3T settings spark
Thanks Matei. -D On Tue, Dec 30, 2014 at 4:49 PM, Matei Zaharia matei.zaha...@gmail.com wrote: This file needs to be on your CLASSPATH actually, not just in a directory. The best way to pass it in is probably to package it into your application JAR. You can put it in src/main/resources in a Maven or SBT project, and check that it makes it into the JAR using jar tf yourfile.jar. Matei On Dec 30, 2014, at 4:21 PM, durga durgak...@gmail.com wrote: I am not sure , the way I can pass jets3t.properties file for spark-submit. --file option seems not working. can some one please help me. My production spark jobs get hung up when reading s3 file sporadically. Thanks, -D -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JetS3T-settings-spark-tp20916.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
