Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, "ÐΞ€ρ@Ҝ (๏̯͡๏)" <deepuj...@gmail.com> wrote:
> I am now seeing this error. > > > > > > 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw > exception: FAILED: SemanticException Line 1:23 Invalid path > ''examples/src/main/resources/kv1.txt'': No files matching path > file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt > > org.apache.spark.sql.execution.QueryExecutionException: FAILED: > SemanticException Line 1:23 Invalid path > ''examples/src/main/resources/kv1.txt'': No files matching path > file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_000002/examples/src/main/resources/kv1.txt > > at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) > > at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) > > > > > -sh-4.1$ pwd > > /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 > > -sh-4.1$ ls examples/src/main/resources/kv1.txt > > examples/src/main/resources/kv1.txt > > -sh-4.1$ > > > > On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang <zzh...@hortonworks.com> > wrote: > >> You can do it in $SPARK_HOME/conf/spark-defaults.con >> >> spark.driver.extraJavaOptions -XX:MaxPermSize=512m >> >> Thanks. >> >> Zhan Zhang >> >> >> On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: >> >> Where and how do i pass this or other JVM argument ? >> -XX:MaxPermSize=512m >> >> On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang <zzh...@hortonworks.com> >> wrote: >> >>> I solve this by increase the PermGen memory size in driver. >>> >>> -XX:MaxPermSize=512m >>> >>> Thanks. >>> >>> Zhan Zhang >>> >>> On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >>> wrote: >>> >>> I am facing same issue, posted a new thread. Please respond. >>> >>> On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzh...@hortonworks.com> >>> wrote: >>> >>>> Hi Folks, >>>> >>>> I am trying to run hive context in yarn-cluster mode, but met some >>>> error. Does anybody know what cause the issue. >>>> >>>> I use following cmd to build the distribution: >>>> >>>> ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 >>>> >>>> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: >>>> YarnClusterScheduler.postStartHook done >>>> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering >>>> block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB >>>> RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) >>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE >>>> IF NOT EXISTS src (key INT, value STRING) >>>> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed >>>> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store >>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >>>> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize >>>> called >>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>> datanucleus.cache.level2 unknown - will be ignored >>>> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property >>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not >>>> present in CLASSPATH (or one of dependencies) >>>> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not >>>> present in CLASSPATH (or one of dependencies) >>>> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object >>>> pin classes with >>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check >>>> failed, assuming we are not on mysql: Lexical error at line 1, column 5. >>>> Encountered: "@" (64), after : "". >>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>> "embedded-only" so does not have its own datastore table. >>>> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class >>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>> "embedded-only" so does not have its own datastore table. >>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>> "embedded-only" so does not have its own datastore table. >>>> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class >>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>> "embedded-only" so does not have its own datastore table. >>>> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore >>>> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not >>>> found in metastore. hive.metastore.schema.verification is not enabled so >>>> recording the schema version 0.13.1aa >>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in >>>> metastore >>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in >>>> metastore >>>> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in >>>> admin role, since config is empty >>>> 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at >>>> this point. hive.execution.engine=mr. >>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not >>>> creating a lock manager >>>> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE >>>> IF NOT EXISTS src (key INT, value STRING) >>>> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed >>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse >>>> start=1421190003030 end=1421190003031 duration=1 >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic >>>> Analysis >>>> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src >>>> position=27 >>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : >>>> db=default tbl=src >>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>> ip=unknown-ip-addr cmd=get_table : db=default tbl=src >>>> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default >>>> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang >>>> ip=unknown-ip-addr cmd=get_database: default >>>> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed >>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=semanticAnalyze >>>> start=1421190003031 end=1421190003406 duration=375 >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: >>>> Schema(fieldSchemas:null, properties:null) >>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile >>>> start=1421190002998 end=1421190003416 duration=418 >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=Driver.execute >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF NOT >>>> EXISTS src (key INT, value STRING) >>>> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit >>>> start=1421190002995 end=1421190003421 duration=426 >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=task.DDL.Stage-0 >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for >>>> table src >>>> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG method=Driver.execute >>>> start=1421190003416 end=1421190005498 duration=2082 >>>> from=org.apache.hadoop.hive.ql.Driver> >>>> Exception in thread "Driver" >>>> Exception: java.lang.OutOfMemoryError thrown from the >>>> UncaughtExceptionHandler in thread "Driver" >>>> -- >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to >>>> which it is addressed and may contain information that is confidential, >>>> privileged and exempt from disclosure under applicable law. If the >>>> reader >>>> of this message is not the intended recipient, you are hereby notified >>>> that >>>> any printing, copying, dissemination, distribution, disclosure or >>>> forwarding of this communication is strictly prohibited. If you have >>>> received this communication in error, please contact the sender >>>> immediately >>>> and delete it from your system. Thank You. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >>> >>> -- >>> Deepak >>> >>> >>> >> >> >> -- >> Deepak >> >> >> > > > -- > Deepak > >
