Re: convert JavaRDD<List> to JavaRDD

[Vinay Parekar]

Sorry, I guess my mac autocorrected it . Yeah its flatmap().

From: Jiang Jacky <jiang0...@gmail.com<mailto:jiang0...@gmail.com>>
Date: Wednesday, April 5, 2017 at 12:10 PM
To: ostkadm <vpare...@overstock.com<mailto:vpare...@overstock.com>>
Cc: Hamza HACHANI <hamza.hach...@supcom.tn<mailto:hamza.hach...@supcom.tn>>, 
"user@spark.apache.org<mailto:user@spark.apache.org>" 
<user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: convert JavaRDD<List> to JavaRDD

There is no flattop just flatMap

On Apr 5, 2017, at 12:24 PM, Vinay Parekar 
<vpare...@overstock.com<mailto:vpare...@overstock.com>> wrote:

I think flattop() will be helpful in this case . Correct me if I am wrong.

From: Hamza HACHANI <hamza.hach...@supcom.tn<mailto:hamza.hach...@supcom.tn>>
Date: Wednesday, April 5, 2017 at 3:43 AM
To: "user@spark.apache.org<mailto:user@spark.apache.org>" 
<user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: convert JavaRDD<List> to JavaRDD


I want to convert a JavaRDD<List> to JavaRDD. For example if 
there is 3 elment in List 3 Object would be created in my new 
JavaRDD.

Does any one have an idea ?



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.

Re: convert JavaRDD<List> to JavaRDD

[Vinay Parekar]

I think flattop() will be helpful in this case . Correct me if I am wrong.

From: Hamza HACHANI >
Date: Wednesday, April 5, 2017 at 3:43 AM
To: "user@spark.apache.org" 
>
Subject: convert JavaRDD to JavaRDD


I want to convert a JavaRDD to JavaRDD. For example if 
there is 3 elment in List 3 Object would be created in my new 
JavaRDD.

Does any one have an idea ?



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.

not table to connect to table using hiveContext

[vinay parekar]

$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1998)
   at com.sun.proxy.$Proxy103.getTable(Unknown Source)
   at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1150)
   ... 85 more


I have added the hive-site.xml in /conf.
and here is code snippet i used to connect:

SparkConf conf = new SparkConf();
   conf.set("hive.metastore.warehouse.dir","/user/beeswax/warehouse")
   .set("hive.metastore.uris","thrift://cms01.hadoop.test.ostk.com:9083");

   JavaSparkContext jsc = new JavaSparkContext(conf);

   String loadToRnowImports = "INSERT INTO TABLE prod_working.rnow_imports SELECT * 
FROM prod_working.rnow_imports_text";
   final HiveContext hiveSqlContext = new 
org.apache.spark.sql.hive.HiveContext(jsc.sc());
   hiveSqlContext.setConf("hive.metastore.warehouse.dir", 
"/user/beeswax/warehouse");
   
hiveSqlContext.setConf("hive.metastore.uris","thrift://cms01.hadoop.test.ostk.com:9083");
   //hiveSqlContext.sql("use prod_working");
   hiveSqlContext.sql(loadToRnowImports);

I suppose I am missing some config value. But not sure what config value I am 
exactly missing. tried to debug it but it led me no where .
Still a beginner in spark, hive and big data technologies, will appreciate any 
suggestions.




--
Thanks & Regards,
Vinay Parekar
Software Developer I



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.