Mich, did you run this locally or on EC2 (I use EC2)? Is this problem universal or specific to, say EC2? Many thanks
From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: Wednesday, December 02, 2015 5:01 PM To: Lin, Hao; user@spark.apache.org Subject: RE: starting spark-shell throws /tmp/hive on HDFS should be writable error Hi, Actually I went back to 1.3, 1.3.1 and 1.4 and built spark from source code with no luck. So I am not sure if any good result is going to come from back tracking? Cheers, Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf[login.sybase.com]<https://urldefense.proofpoint.com/v2/url?u=http-3A__login.sybase.com_files_Product-5FOverviews_ASE-2DWinning-2DStrategy-2D091908.pdf&d=CwMFAg&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=QinEV1-DOnM0qRbga2o_-IP-6cPniwts0F8Rk3CmpCQ&s=S876LfQmjdqIc4Wh6_spNAYyN6n6pX1ltzZp3VuSb-s&e=> Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly http://talebzadehmich.wordpress.com[talebzadehmich.wordpress.com]<https://urldefense.proofpoint.com/v2/url?u=http-3A__talebzadehmich.wordpress.com_&d=CwMFAg&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=QinEV1-DOnM0qRbga2o_-IP-6cPniwts0F8Rk3CmpCQ&s=IupCrV9sVc16ecw3H1ecZZ3jzcEP04V5cTuHS_m46Nw&e=> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Lin, Hao [mailto:hao....@finra.org] Sent: 02 December 2015 21:49 To: Mich Talebzadeh <m...@peridale.co.uk<mailto:m...@peridale.co.uk>>; user@spark.apache.org<mailto:user@spark.apache.org> Subject: RE: starting spark-shell throws /tmp/hive on HDFS should be writable error I also have the same problem on my side using version 1.5.0. Just wonder if anyone has any update on this. Should I go back to the earlier version, like 1.4.0? thanks From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: Friday, November 20, 2015 5:43 PM To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: FW: starting spark-shell throws /tmp/hive on HDFS should be writable error From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: 20 November 2015 21:14 To: u...@hive.apache.org<mailto:u...@hive.apache.org> Subject: starting spark-shell throws /tmp/hive on HDFS should be writable error Hi, Has this been resolved. I don't think this has anything to do with /tmp/hive directory permission spark-shell log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig[logging.apache.org]<https://urldefense.proofpoint.com/v2/url?u=http-3A__logging.apache.org_log4j_1.2_faq.html-23noconfig&d=CwMFAg&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=oHOKhu2DKTwCpcPeTgb9VszWiqFOXGY6GF4SGjoqEy4&s=W2rVckqSVOz4yoVE5AE9_-zp32w4jlYKl4E5YGWs5IM&e=> for more info. Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties To adjust logging level use sc.setLogLevel("INFO") Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_25) Type in expressions to have them evaluated. Type :help for more information. java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------ <console>:10: error: not found: value sqlContext import sqlContext.implicits._ ^ <console>:10: error: not found: value sqlContext import sqlContext.sql ^ scala> Thanks, Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf[login.sybase.com]<https://urldefense.proofpoint.com/v2/url?u=http-3A__login.sybase.com_files_Product-5FOverviews_ASE-2DWinning-2DStrategy-2D091908.pdf&d=CwMFAg&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=oHOKhu2DKTwCpcPeTgb9VszWiqFOXGY6GF4SGjoqEy4&s=V95DBv5vhnm12KvrAt4n-jop1fqIzGekX1ZwtD7NZu8&e=> Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly http://talebzadehmich.wordpress.com[talebzadehmich.wordpress.com]<https://urldefense.proofpoint.com/v2/url?u=http-3A__talebzadehmich.wordpress.com_&d=CwMFAg&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=oHOKhu2DKTwCpcPeTgb9VszWiqFOXGY6GF4SGjoqEy4&s=jwR5bSMaHKsDqspGZTsF2WG1Rc3AUfQpir-KQEr3MvM&e=> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you. Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
