When you run it in local mode ^^ Thanks Best Regards
On Thu, Mar 26, 2015 at 2:06 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > I don;t think thats correct. load data local should pick input from local > directory. > > On Thu, Mar 26, 2015 at 1:59 PM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Not sure, but you can create that path in all workers and put that file >> in it. >> >> Thanks >> Best Regards >> >> On Thu, Mar 26, 2015 at 1:56 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >> wrote: >> >>> The Hive command >>> >>> LOAD DATA LOCAL INPATH >>> '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' >>> INTO TABLE src_spark >>> >>> 1. LOCAL INPATH. if i push to HDFS then how will it work ? >>> >>> 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. >>> >>> On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das <ak...@sigmoidanalytics.com> >>> wrote: >>> >>>> Now its clear that the workers are not having the file kv1.txt in their >>>> local filesystem. You can try putting that in hdfs and use the URI to that >>>> file or try adding the file with sc.addFile >>>> >>>> Thanks >>>> Best Regards >>>> >>>> On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>> wrote: >>>> >>>>> Does not work >>>>> >>>>> 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal >>>>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark >>>>> 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 >>>>> Invalid path >>>>> ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': >>>>> No files matching path >>>>> file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt >>>>> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid >>>>> path >>>>> ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': >>>>> No files matching path >>>>> file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt >>>>> at >>>>> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) >>>>> at >>>>> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) >>>>> at >>>>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) >>>>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) >>>>> >>>>> >>>>> >>>>> Does the input file needs to be passed to executor via -- jars ? >>>>> >>>>> On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das < >>>>> ak...@sigmoidanalytics.com> wrote: >>>>> >>>>>> Try to give the complete path to the file kv1.txt. >>>>>> On 26 Mar 2015 11:48, "ÐΞ€ρ@Ҝ (๏̯͡๏)" <deepuj...@gmail.com> wrote: >>>>>> >>>>>>> I am now seeing this error. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw >>>>>>> exception: FAILED: SemanticException Line 1:23 Invalid path >>>>>>> ''examples/src/main/resources/kv1.txt'': No files matching path >>>>>>> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >>>>>>> >>>>>>> org.apache.spark.sql.execution.QueryExecutionException: FAILED: >>>>>>> SemanticException Line 1:23 Invalid path >>>>>>> ''examples/src/main/resources/kv1.txt'': No files matching path >>>>>>> file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt >>>>>>> >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) >>>>>>> >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -sh-4.1$ pwd >>>>>>> >>>>>>> /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 >>>>>>> >>>>>>> -sh-4.1$ ls examples/src/main/resources/kv1.txt >>>>>>> >>>>>>> examples/src/main/resources/kv1.txt >>>>>>> >>>>>>> -sh-4.1$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang <zzh...@hortonworks.com> >>>>>>> wrote: >>>>>>> >>>>>>>> You can do it in $SPARK_HOME/conf/spark-defaults.con >>>>>>>> >>>>>>>> spark.driver.extraJavaOptions -XX:MaxPermSize=512m >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Zhan Zhang >>>>>>>> >>>>>>>> >>>>>>>> On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Where and how do i pass this or other JVM argument ? >>>>>>>> -XX:MaxPermSize=512m >>>>>>>> >>>>>>>> On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang < >>>>>>>> zzh...@hortonworks.com> wrote: >>>>>>>> >>>>>>>>> I solve this by increase the PermGen memory size in driver. >>>>>>>>> >>>>>>>>> -XX:MaxPermSize=512m >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Zhan Zhang >>>>>>>>> >>>>>>>>> On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I am facing same issue, posted a new thread. Please respond. >>>>>>>>> >>>>>>>>> On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang < >>>>>>>>> zzh...@hortonworks.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Folks, >>>>>>>>>> >>>>>>>>>> I am trying to run hive context in yarn-cluster mode, but met >>>>>>>>>> some error. Does anybody know what cause the issue. >>>>>>>>>> >>>>>>>>>> I use following cmd to build the distribution: >>>>>>>>>> >>>>>>>>>> ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn >>>>>>>>>> -Phadoop-2.4 >>>>>>>>>> >>>>>>>>>> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: >>>>>>>>>> YarnClusterScheduler.postStartHook done >>>>>>>>>> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: >>>>>>>>>> Registering block manager cn122-10.l42scl.hortonworks.com:56157 >>>>>>>>>> with 1589.8 MB RAM, BlockManagerId(2, >>>>>>>>>> cn122-10.l42scl.hortonworks.com, 56157) >>>>>>>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE >>>>>>>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>>>>>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed >>>>>>>>>> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw >>>>>>>>>> store with implemenation >>>>>>>>>> class:org.apache.hadoop.hive.metastore.ObjectStore >>>>>>>>>> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, >>>>>>>>>> initialize called >>>>>>>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>>>>>>> datanucleus.cache.level2 unknown - will be ignored >>>>>>>>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>>>>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>>>>>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified >>>>>>>>>> but not present in CLASSPATH (or one of dependencies) >>>>>>>>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified >>>>>>>>>> but not present in CLASSPATH (or one of dependencies) >>>>>>>>>> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore >>>>>>>>>> object pin classes with >>>>>>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>>>>>>>> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check >>>>>>>>>> failed, assuming we are not on mysql: Lexical error at line 1, >>>>>>>>>> column 5. >>>>>>>>>> Encountered: "@" (64), after : "". >>>>>>>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>>>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>>>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>>>>> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized >>>>>>>>>> ObjectStore >>>>>>>>>> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information >>>>>>>>>> not found in metastore. hive.metastore.schema.verification is not >>>>>>>>>> enabled >>>>>>>>>> so recording the schema version 0.13.1aa >>>>>>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role >>>>>>>>>> in metastore >>>>>>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role >>>>>>>>>> in metastore >>>>>>>>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added >>>>>>>>>> in admin role, since config is empty >>>>>>>>>> 15/01/13 18:00:01 INFO session.SessionState: No Tez session >>>>>>>>>> required at this point. hive.execution.engine=mr. >>>>>>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG >>>>>>>>>> method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, >>>>>>>>>> not creating a lock manager >>>>>>>>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE >>>>>>>>>> TABLE IF NOT EXISTS src (key INT, value STRING) >>>>>>>>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse >>>>>>>>>> start=1421190003030 end=1421190003031 duration=1 >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>>>>>> method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic >>>>>>>>>> Analysis >>>>>>>>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src >>>>>>>>>> position=27 >>>>>>>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : >>>>>>>>>> db=default tbl=src >>>>>>>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>>>>>>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src >>>>>>>>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: >>>>>>>>>> default >>>>>>>>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>>>>>>>> ip=unknown-ip-addr cmd=get_database: default >>>>>>>>>> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG >>>>>>>>>> method=semanticAnalyze start=1421190003031 end=1421190003406 >>>>>>>>>> duration=375 >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: >>>>>>>>>> Schema(fieldSchemas:null, properties:null) >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile >>>>>>>>>> start=1421190002998 end=1421190003416 duration=418 >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>>>>>> method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE >>>>>>>>>> IF NOT EXISTS src (key INT, value STRING) >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG >>>>>>>>>> method=TimeToSubmit start=1421190002995 end=1421190003421 >>>>>>>>>> duration=426 >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG >>>>>>>>>> method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe >>>>>>>>>> for table src >>>>>>>>>> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG >>>>>>>>>> method=Driver.execute start=1421190003416 end=1421190005498 >>>>>>>>>> duration=2082 >>>>>>>>>> from=org.apache.hadoop.hive.ql.Driver> >>>>>>>>>> Exception in thread "Driver" >>>>>>>>>> Exception: java.lang.OutOfMemoryError thrown from the >>>>>>>>>> UncaughtExceptionHandler in thread "Driver" >>>>>>>>>> -- >>>>>>>>>> CONFIDENTIALITY NOTICE >>>>>>>>>> NOTICE: This message is intended for the use of the individual or >>>>>>>>>> entity to >>>>>>>>>> which it is addressed and may contain information that is >>>>>>>>>> confidential, >>>>>>>>>> privileged and exempt from disclosure under applicable law. If >>>>>>>>>> the reader >>>>>>>>>> of this message is not the intended recipient, you are hereby >>>>>>>>>> notified that >>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or >>>>>>>>>> forwarding of this communication is strictly prohibited. If you >>>>>>>>>> have >>>>>>>>>> received this communication in error, please contact the sender >>>>>>>>>> immediately >>>>>>>>>> and delete it from your system. Thank You. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Deepak >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Deepak >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Deepak >>>>>>> >>>>>>> >>>>> >>>>> >>>>> -- >>>>> Deepak >>>>> >>>>> >>>> >>> >>> >>> -- >>> Deepak >>> >>> >> > > > -- > Deepak > >
