Re: OOM for HiveFromSpark example
Not sure, but you can create that path in all workers and put that file in it. Thanks Best Regards On Thu, Mar 26, 2015 at 1:56 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: The Hive command LOAD DATA LOCAL INPATH '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' INTO TABLE src_spark 1. LOCAL INPATH. if i push to HDFS then how will it work ? 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13
Re: OOM for HiveFromSpark example
Could you try putting that file in hdfs and try like: LOAD DATA INPATH 'hdfs://sigmoid/test/kv1.txt' INTO TABLE src_spark Thanks Best Regards On Thu, Mar 26, 2015 at 2:07 PM, Akhil Das ak...@sigmoidanalytics.com wrote: When you run it in local mode ^^ Thanks Best Regards On Thu, Mar 26, 2015 at 2:06 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I don;t think thats correct. load data local should pick input from local directory. On Thu, Mar 26, 2015 at 1:59 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Not sure, but you can create that path in all workers and put that file in it. Thanks Best Regards On Thu, Mar 26, 2015 at 1:56 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: The Hive command LOAD DATA LOCAL INPATH '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' INTO TABLE src_spark 1. LOCAL INPATH. if i push to HDFS then how will it work ? 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering
Re: OOM for HiveFromSpark example
I don;t think thats correct. load data local should pick input from local directory. On Thu, Mar 26, 2015 at 1:59 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Not sure, but you can create that path in all workers and put that file in it. Thanks Best Regards On Thu, Mar 26, 2015 at 1:56 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: The Hive command LOAD DATA LOCAL INPATH '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' INTO TABLE src_spark 1. LOCAL INPATH. if i push to HDFS then how will it work ? 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening
Re: OOM for HiveFromSpark example
The Hive command LOAD DATA LOCAL INPATH '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' INTO TABLE src_spark 1. LOCAL INPATH. if i push to HDFS then how will it work ? 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property
Re: OOM for HiveFromSpark example
When you run it in local mode ^^ Thanks Best Regards On Thu, Mar 26, 2015 at 2:06 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I don;t think thats correct. load data local should pick input from local directory. On Thu, Mar 26, 2015 at 1:59 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Not sure, but you can create that path in all workers and put that file in it. Thanks Best Regards On Thu, Mar 26, 2015 at 1:56 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: The Hive command LOAD DATA LOCAL INPATH '/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt' INTO TABLE src_spark 1. LOCAL INPATH. if i push to HDFS then how will it work ? 2. I cant use sc.addFile, cause i want to run Hive (Spark SQL) queries. On Thu, Mar 26, 2015 at 1:41 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT,
Re: OOM for HiveFromSpark example
I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG
Re: OOM for HiveFromSpark example
Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO
Re: OOM for HiveFromSpark example
Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO
Re: OOM for HiveFromSpark example
Now its clear that the workers are not having the file kv1.txt in their local filesystem. You can try putting that in hdfs and use the URI to that file or try adding the file with sc.addFile Thanks Best Regards On Thu, Mar 26, 2015 at 1:38 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Does not work 15/03/26 01:07:05 INFO HiveMetaStore.audit: ugi=dvasthimal ip=unknown-ip-addr cmd=get_table : db=default tbl=src_spark 15/03/26 01:07:06 ERROR ql.Driver: FAILED: SemanticException Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt'': No files matching path file:/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/examples/src/main/resources/kv1.txt at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:142) at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:233) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) Does the input file needs to be passed to executor via -- jars ? On Thu, Mar 26, 2015 at 12:15 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Try to give the complete path to the file kv1.txt. On 26 Mar 2015 11:48, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am now seeing this error. 15/03/25 19:44:03 ERROR yarn.ApplicationMaster: User class threw exception: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''examples/src/main/resources/kv1.txt'': No files matching path file:/hadoop/10/scratch/local/usercache/dvasthimal/appcache/application_1426715280024_89893/container_1426715280024_89893_01_02/examples/src/main/resources/kv1.txt at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:312) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:280) -sh-4.1$ pwd /home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4 -sh-4.1$ ls examples/src/main/resources/kv1.txt examples/src/main/resources/kv1.txt -sh-4.1$ On Thu, Mar 26, 2015 at 8:08 AM, Zhan Zhang zzh...@hortonworks.com wrote: You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with
Re: OOM for HiveFromSpark example
I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=parse start=1421190003030 end=1421190003031 duration=1 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src position=27 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr cmd=get_table : db=default tbl=src 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr cmd=get_database: default 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=semanticAnalyze start=1421190003031 end=1421190003406 duration=375 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO
Re: OOM for HiveFromSpark example
I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.commailto:deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.commailto:zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157http://cn122-10.l42scl.hortonworks.com:56157/ with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.comhttp://cn122-10.l42scl.hortonworks.com/, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=parse start=1421190003030 end=1421190003031 duration=1 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src position=27 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr cmd=get_table : db=default tbl=src 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr
Re: OOM for HiveFromSpark example
Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=parse start=1421190003030 end=1421190003031 duration=1 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src position=27 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr cmd=get_table : db=default tbl=src 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default 15/01/13 18:00:03
Re: OOM for HiveFromSpark example
You can do it in $SPARK_HOME/conf/spark-defaults.con spark.driver.extraJavaOptions -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 7:25 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.commailto:deepuj...@gmail.com wrote: Where and how do i pass this or other JVM argument ? -XX:MaxPermSize=512m On Wed, Mar 25, 2015 at 11:36 PM, Zhan Zhang zzh...@hortonworks.commailto:zzh...@hortonworks.com wrote: I solve this by increase the PermGen memory size in driver. -XX:MaxPermSize=512m Thanks. Zhan Zhang On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.commailto:deepuj...@gmail.com wrote: I am facing same issue, posted a new thread. Please respond. On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang zzh...@hortonworks.commailto:zzh...@hortonworks.com wrote: Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157http://cn122-10.l42scl.hortonworks.com:56157/ with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.comhttp://cn122-10.l42scl.hortonworks.com/, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=parse start=1421190003030 end=1421190003031 duration=1 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis 15/01/13 18:00:03 INFO
OOM for HiveFromSpark example
Hi Folks, I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know what cause the issue. I use following cmd to build the distribution: ./make-distribution.sh -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157) 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: @ (64), after : . 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as embedded-only so does not have its own datastore table. 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so does not have its own datastore table. 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 15/01/13 18:00:02 INFO log.PerfLogger: PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING) 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=parse start=1421190003030 end=1421190003031 duration=1 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO log.PerfLogger: PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src position=27 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr cmd=get_table : db=default tbl=src 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang ip=unknown-ip-addr cmd=get_database: default 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=semanticAnalyze start=1421190003031 end=1421190003406 duration=375 from=org.apache.hadoop.hive.ql.Driver 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) 15/01/13 18:00:03 INFO log.PerfLogger: /PERFLOG method=compile start=1421190002998 end=1421190003416
