Not sure, but you can create that path in all workers and put that file in it.
Thanks Best Regards On Thu, Mar 26, 2015 at 1:56 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > The Hive command > > LOAD DATA LOCAL INPATH > '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' > INTO TABLE src_spark > > 1. LOCAL INPATH. if i push to HDFS then how will it work ? > > 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. > > On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Now its clear that the workers are not having the file kv1.txt in their >> local filesystem. You can try putting that in hdfs and use the URI to that >> file or try adding the file with sc.addFile >> >> Thanks >> Best Regards >> >> On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >> wrote: >> >>> Does not work >>> >>> 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal >>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark >>> 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 >>> Invalid path >>> ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': >>> No files matching path >>> file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt >>> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid >>> path >>> ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': >>> No files matching path >>> file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt >>> at >>> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) >>> at >>> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) >>> at >>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) >>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) >>> >>> >>> >>> Does the input file needs to be passed to executor via -- jars ? >>> >>> On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das <ak...@sigmoidanalytics.com> >>> wrote: >>> >>>> Try to give the complete path to the file kv1.txt. >>>> On 26 Mar 2015 11:48, "ÐΞ€ρ@Ҝ (๏̯͡๏)" <deepuj...@gmail.com> wrote: >>>> >>>>> I am now seeing this error. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw >>>>> exception: FAILED: SemanticException Line 1:23 Invalid path >>>>> ''examples/src/main/resources/kv1.txt'': No files matching path >>>>> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >>>>> >>>>> org.apache.spark.sql.execution.QueryExecutionException: FAILED: >>>>> SemanticException Line 1:23 Invalid path >>>>> ''examples/src/main/resources/kv1.txt'': No files matching path >>>>> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >>>>> >>>>> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) >>>>> >>>>> at >>>>> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) >>>>> >>>>> >>>>> >>>>> >>>>> -sh-4.1$ pwd >>>>> >>>>> /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 >>>>> >>>>> -sh-4.1$ ls examples/src/main/resources/kv1.txt >>>>> >>>>> examples/src/main/resources/kv1.txt >>>>> >>>>> -sh-4.1$ >>>>> >>>>> >>>>> >>>>> On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang <zzh...@hortonworks.com> >>>>> wrote: >>>>> >>>>>> You can do it in $SPARK_HOME/conf/spark-defaults.con >>>>>> >>>>>> spark.driver.extraJavaOptions -XX:MaxPermSize=512m >>>>>> >>>>>> Thanks. >>>>>> >>>>>> Zhan Zhang >>>>>> >>>>>> >>>>>> On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Where and how do i pass this or other JVM argument ? >>>>>> -XX:MaxPermSize=512m >>>>>> >>>>>> On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang <zzh...@hortonworks.com> >>>>>> wrote: >>>>>> >>>>>>> I solve this by increase the PermGen memory size in driver. >>>>>>> >>>>>>> -XX:MaxPermSize=512m >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> Zhan Zhang >>>>>>> >>>>>>> On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> I am facing same issue, posted a new thread. Please respond. >>>>>>> >>>>>>> On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzh...@hortonworks.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Folks, >>>>>>>> >>>>>>>> I am trying to run hive context in yarn-cluster mode, but met some >>>>>>>> error. Does anybody know what cause the issue. >>>>>>>> >>>>>>>> I use following cmd to build the distribution: >>>>>>>> >>>>>>>> ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn >>>>>>>> -Phadoop-2.4 >>>>>>>> >>>>>>>> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: >>>>>>>> YarnClusterScheduler.postStartHook done >>>>>>>> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering >>>>>>>> block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB >>>>>>>> RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) >>>>>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE >>>>>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed >>>>>>>> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw >>>>>>>> store with implemenation >>>>>>>> class:org.apache.hadoop.hive.metastore.ObjectStore >>>>>>>> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, >>>>>>>> initialize called >>>>>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>>>>> datanucleus.cache.level2 unknown - will be ignored >>>>>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>>>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but >>>>>>>> not present in CLASSPATH (or one of dependencies) >>>>>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but >>>>>>>> not present in CLASSPATH (or one of dependencies) >>>>>>>> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore >>>>>>>> object pin classes with >>>>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>>>>>> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check >>>>>>>> failed, assuming we are not on mysql: Lexical error at line 1, column >>>>>>>> 5. >>>>>>>> Encountered: "@" (64), after : "". >>>>>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized >>>>>>>> ObjectStore >>>>>>>> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information >>>>>>>> not found in metastore. hive.metastore.schema.verification is not >>>>>>>> enabled >>>>>>>> so recording the schema version 0.13.1aa >>>>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in >>>>>>>> metastore >>>>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role >>>>>>>> in metastore >>>>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in >>>>>>>> admin role, since config is empty >>>>>>>> 15/01/13 18:00:01 INFO session.SessionState: No Tez session >>>>>>>> required at this point. hive.execution.engine=mr. >>>>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not >>>>>>>> creating a lock manager >>>>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE >>>>>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse >>>>>>>> start=1421190003030 end=1421190003031 duration=1 >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>>>> method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic >>>>>>>> Analysis >>>>>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src >>>>>>>> position=27 >>>>>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : >>>>>>>> db=default tbl=src >>>>>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>>>>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src >>>>>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: >>>>>>>> default >>>>>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>>>>> ip=unknown-ip-addr cmd=get_database: default >>>>>>>> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG >>>>>>>> method=semanticAnalyze start=1421190003031 end=1421190003406 >>>>>>>> duration=375 >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: >>>>>>>> Schema(fieldSchemas:null, properties:null) >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile >>>>>>>> start=1421190002998 end=1421190003416 duration=418 >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>>>> method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF >>>>>>>> NOT EXISTS src (key INT, value STRING) >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG >>>>>>>> method=TimeToSubmit start=1421190002995 end=1421190003421 duration=426 >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>>>> method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for >>>>>>>> table src >>>>>>>> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG >>>>>>>> method=Driver.execute start=1421190003416 end=1421190005498 >>>>>>>> duration=2082 >>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>> Exception in thread "Driver" >>>>>>>> Exception: java.lang.OutOfMemoryError thrown from the >>>>>>>> UncaughtExceptionHandler in thread "Driver" >>>>>>>> -- >>>>>>>> CONFIDENTIALITY NOTICE >>>>>>>> NOTICE: This message is intended for the use of the individual or >>>>>>>> entity to >>>>>>>> which it is addressed and may contain information that is >>>>>>>> confidential, >>>>>>>> privileged and exempt from disclosure under applicable law. If the >>>>>>>> reader >>>>>>>> of this message is not the intended recipient, you are hereby >>>>>>>> notified that >>>>>>>> any printing, copying, dissemination, distribution, disclosure or >>>>>>>> forwarding of this communication is strictly prohibited. If you have >>>>>>>> received this communication in error, please contact the sender >>>>>>>> immediately >>>>>>>> and delete it from your system. Thank You. >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Deepak >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Deepak >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Deepak >>>>> >>>>> >>> >>> >>> -- >>> Deepak >>> >>> >> > > > -- > Deepak > >
