Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile
Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > Does not work > > 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal > ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark > 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 > Invalid path > ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': > No files matching path > file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt > org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path > ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': > No files matching path > file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt > at > org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) > at > org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) > > > > Does the input file needs to be passed to executor via -- jars ? > > On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Try to give the complete path to the file kv1.txt. >> On 26 Mar 2015 11:48, "ÐΞ€ρ@Ҝ (๏̯͡๏)" <deepuj...@gmail.com> wrote: >> >>> I am now seeing this error. >>> >>> >>> >>> >>> >>> 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw >>> exception: FAILED: SemanticException Line 1:23 Invalid path >>> ''examples/src/main/resources/kv1.txt'': No files matching path >>> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >>> >>> org.apache.spark.sql.execution.QueryExecutionException: FAILED: >>> SemanticException Line 1:23 Invalid path >>> ''examples/src/main/resources/kv1.txt'': No files matching path >>> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >>> >>> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) >>> >>> at >>> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) >>> >>> >>> >>> >>> -sh-4.1$ pwd >>> >>> /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 >>> >>> -sh-4.1$ ls examples/src/main/resources/kv1.txt >>> >>> examples/src/main/resources/kv1.txt >>> >>> -sh-4.1$ >>> >>> >>> >>> On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang <zzh...@hortonworks.com> >>> wrote: >>> >>>> You can do it in $SPARK_HOME/conf/spark-defaults.con >>>> >>>> spark.driver.extraJavaOptions -XX:MaxPermSize=512m >>>> >>>> Thanks. >>>> >>>> Zhan Zhang >>>> >>>> >>>> On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>> wrote: >>>> >>>> Where and how do i pass this or other JVM argument ? >>>> -XX:MaxPermSize=512m >>>> >>>> On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang <zzh...@hortonworks.com> >>>> wrote: >>>> >>>>> I solve this by increase the PermGen memory size in driver. >>>>> >>>>> -XX:MaxPermSize=512m >>>>> >>>>> Thanks. >>>>> >>>>> Zhan Zhang >>>>> >>>>> On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>>> wrote: >>>>> >>>>> I am facing same issue, posted a new thread. Please respond. >>>>> >>>>> On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzh...@hortonworks.com> >>>>> wrote: >>>>> >>>>>> Hi Folks, >>>>>> >>>>>> I am trying to run hive context in yarn-cluster mode, but met some >>>>>> error. Does anybody know what cause the issue. >>>>>> >>>>>> I use following cmd to build the distribution: >>>>>> >>>>>> ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn >>>>>> -Phadoop-2.4 >>>>>> >>>>>> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: >>>>>> YarnClusterScheduler.postStartHook done >>>>>> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering >>>>>> block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB >>>>>> RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) >>>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE >>>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed >>>>>> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store >>>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >>>>>> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize >>>>>> called >>>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>>> datanucleus.cache.level2 unknown - will be ignored >>>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but >>>>>> not present in CLASSPATH (or one of dependencies) >>>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but >>>>>> not present in CLASSPATH (or one of dependencies) >>>>>> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore >>>>>> object pin classes with >>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>>>> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check >>>>>> failed, assuming we are not on mysql: Lexical error at line 1, column 5. >>>>>> Encountered: "@" (64), after : "". >>>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>> "embedded-only" so does not have its own datastore table. >>>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>> "embedded-only" so does not have its own datastore table. >>>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>> "embedded-only" so does not have its own datastore table. >>>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>> "embedded-only" so does not have its own datastore table. >>>>>> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore >>>>>> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not >>>>>> found in metastore. hive.metastore.schema.verification is not enabled so >>>>>> recording the schema version 0.13.1aa >>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in >>>>>> metastore >>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in >>>>>> metastore >>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in >>>>>> admin role, since config is empty >>>>>> 15/01/13 18:00:01 INFO session.SessionState: No Tez session required >>>>>> at this point. hive.execution.engine=mr. >>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not >>>>>> creating a lock manager >>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE >>>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse >>>>>> start=1421190003030 end=1421190003031 duration=1 >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>> method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic >>>>>> Analysis >>>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src >>>>>> position=27 >>>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : >>>>>> db=default tbl=src >>>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src >>>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: >>>>>> default >>>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>>> ip=unknown-ip-addr cmd=get_database: default >>>>>> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG >>>>>> method=semanticAnalyze start=1421190003031 end=1421190003406 duration=375 >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: >>>>>> Schema(fieldSchemas:null, properties:null) >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile >>>>>> start=1421190002998 end=1421190003416 duration=418 >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=Driver.execute >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF >>>>>> NOT EXISTS src (key INT, value STRING) >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit >>>>>> start=1421190002995 end=1421190003421 duration=426 >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>> method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver> >>>>>> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for >>>>>> table src >>>>>> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG >>>>>> method=Driver.execute start=1421190003416 end=1421190005498 duration=2082 >>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>> Exception in thread "Driver" >>>>>> Exception: java.lang.OutOfMemoryError thrown from the >>>>>> UncaughtExceptionHandler in thread "Driver" >>>>>> -- >>>>>> CONFIDENTIALITY NOTICE >>>>>> NOTICE: This message is intended for the use of the individual or >>>>>> entity to >>>>>> which it is addressed and may contain information that is >>>>>> confidential, >>>>>> privileged and exempt from disclosure under applicable law. If the >>>>>> reader >>>>>> of this message is not the intended recipient, you are hereby >>>>>> notified that >>>>>> any printing, copying, dissemination, distribution, disclosure or >>>>>> forwarding of this communication is strictly prohibited. If you have >>>>>> received this communication in error, please contact the sender >>>>>> immediately >>>>>> and delete it from your system. Thank You. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Deepak >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Deepak >>>> >>>> >>>> >>> >>> >>> -- >>> Deepak >>> >>> > > > -- > Deepak > >
