Re: Tableau beta connector
Hi, I would like you to read my stack overflow answer to this question. If you need more clarification feel free to drop a msg. http://stackoverflow.com/questions/28403664/connect-to-existing-hive-in-intellij-using-sbt-as-build Regards, Ashutosh From: ganterm [via Apache Spark User List] ml-node+s1001560n21709...@n3.nabble.com Sent: Thursday, February 19, 2015 12:49 AM To: Ashutosh Trivedi (MT2013030) Subject: Re: Tableau beta connector Ashutosh, Were you able to figure this out? I am having the exact some question. I think the answer is to use Spark SQL to create/load a table in Hive (e.g. execute the HiveQL CREATE TABLE statement) but I am not sure. Hoping for something more simple than that. Anybody? Thanks! If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512p21709.html To unsubscribe from Tableau beta connector, click herehttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=21512code=YXNodXRvc2gudHJpdmVkaUBpaWl0Yi5vcmd8MjE1MTJ8LTM5MzMxOTc2MQ==. NAMLhttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
Re: Tableau beta connector
I am able to connect by doing the following using the Tableau Initial SQL and a custom query: 1. First ingest csv file or json and save out to file system: import org.apache.spark.sql.SQLContext import com.databricks.spark.csv._ val sqlContext = new SQLContext(sc) val demo = sqlContext.csvFile(/user/data/csv/demo.csv) demo.toJSON.saveAsTextFile(/user/data/json/test”) 2. Start $SPARK_HOME/sbin/start-thirftserver: ./sbin/start-thriftserver.sh --master spark://radtech.io:7077 --total-executor-cores 2 --driver-class-path --hiveconf hive.server2.thrift.port=10001 --hiveconf hive.server2.thrift.bind.host radtech.io 3. Start tableau session. Create a connection to thrift server via SparkSQL (Beta) connector. 4. In Tableau add the following to the “Initial SQL” create temporary table test using org.apache.spark.sql.json options (path '/user/data/json/test/*’); cache table test; 1. Refresh connection. Then select “New Custom SQL” and issue something like: select * from test; You will see your table appear. HTH. -Todd On Thu, Feb 19, 2015 at 5:41 AM, ashu ashutosh.triv...@iiitb.org wrote: Hi, I would like you to read my stack overflow answer to this question. If you need more clarification feel free to drop a msg. http://stackoverflow.com/questions/28403664/connect-to-existing-hive-in-intellij-using-sbt-as-build Regards, Ashutosh -- *From:* ganterm [via Apache Spark User List] ml-node+[hidden email] http:///user/SendEmail.jtp?type=nodenode=21719i=0 *Sent:* Thursday, February 19, 2015 12:49 AM *To:* Ashutosh Trivedi (MT2013030) *Subject:* Re: Tableau beta connector Ashutosh, Were you able to figure this out? I am having the exact some question. I think the answer is to use Spark SQL to create/load a table in Hive (e.g. execute the HiveQL CREATE TABLE statement) but I am not sure. Hoping for something more simple than that. Anybody? Thanks! -- If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512p21709.html To unsubscribe from Tableau beta connector, click here. NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: Re: Tableau beta connector http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512p21719.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Tableau beta connector
Thanks Todd. great stuff :) Regards, Ashu From: Todd Nist tsind...@gmail.com Sent: Thursday, February 19, 2015 7:46 PM To: Ashutosh Trivedi (MT2013030) Cc: user@spark.apache.org Subject: Re: Tableau beta connector I am able to connect by doing the following using the Tableau Initial SQL and a custom query: 1. First ingest csv file or json and save out to file system: import org.apache.spark.sql.SQLContext import com.databricks.spark.csv._ val sqlContext = new SQLContext(sc) val demo = sqlContext.csvFile(/user/data/csv/demo.csv) demo.toJSON.saveAsTextFile(/user/data/json/test) 2. Start $SPARK_HOME/sbin/start-thirftserver: ./sbin/start-thriftserver.sh --master spark://radtech.io:7077http://radtech.io:7077 --total-executor-cores 2 --driver-class-path --hiveconf hive.server2.thrift.port=10001 --hiveconf hive.server2.thrift.bind.host radtech.iohttp://radtech.io 3. Start tableau session. Create a connection to thrift server via SparkSQL (Beta) connector. 4. In Tableau add the following to the Initial SQL create temporary table test using org.apache.spark.sql.json options (path '/user/data/json/test/*'); cache table test; * Refresh connection. Then select New Custom SQL and issue something like: select * from test; You will see your table appear. HTH. -Todd On Thu, Feb 19, 2015 at 5:41 AM, ashu ashutosh.triv...@iiitb.orgmailto:ashutosh.triv...@iiitb.org wrote: Hi, I would like you to read my stack overflow answer to this question. If you need more clarification feel free to drop a msg. http://stackoverflow.com/questions/28403664/connect-to-existing-hive-in-intellij-using-sbt-as-build Regards, Ashutosh From: ganterm [via Apache Spark User List] ml-node+[hidden email]http:///user/SendEmail.jtp?type=nodenode=21719i=0 Sent: Thursday, February 19, 2015 12:49 AM To: Ashutosh Trivedi (MT2013030) Subject: Re: Tableau beta connector Ashutosh, Were you able to figure this out? I am having the exact some question. I think the answer is to use Spark SQL to create/load a table in Hive (e.g. execute the HiveQL CREATE TABLE statement) but I am not sure. Hoping for something more simple than that. Anybody? Thanks! If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512p21709.html To unsubscribe from Tableau beta connector, click here. NAMLhttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml View this message in context: Re: Tableau beta connectorhttp://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512p21719.html Sent from the Apache Spark User List mailing list archivehttp://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Tableau beta connector
Ashutosh, Were you able to figure this out? I am having the exact some question. I think the answer is to use Spark SQL to create/load a table in Hive (e.g. execute the HiveQL CREATE TABLE statement) but I am not sure. Hoping for something more simple than that. Anybody? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512p21709.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Tableau beta connector
okay. So the queries tableau will run on the persisted data will be through SPARK SQL to improve performance and to take advantage of SPARK SQL. Thanks again Denny From: Denny Lee denny.g@gmail.com Sent: Thursday, February 5, 2015 1:27 PM To: Ashutosh Trivedi (MT2013030); İsmail Keskin Cc: user@spark.apache.org Subject: Re: Tableau beta connector The context is that you would create your RDDs and then persist them in Hive. Once in Hive, the data is accessible from the Tableau extract through Spark thrift server. On Wed, Feb 4, 2015 at 23:36 Ashutosh Trivedi (MT2013030) ashutosh.triv...@iiitb.orgmailto:ashutosh.triv...@iiitb.org wrote: Thanks Denny and Ismail. Denny ,I went through your blog, It was great help. I guess tableau beta connector also following the same procedure,you described in blog. I am building the Spark now. Basically what I don't get is, where to put my data so that tableau can extract. So Ismail,its just Spark SQL. No RDDs I think I am getting it now . We use spark for our big data processing and we want processed data (Rdd) into tableau. So we should put our data in hive metastore and tableau will extract it from there using this connector? Correct me if I am wrong. I guess I have to look at how thrift server works. From: Denny Lee denny.g@gmail.commailto:denny.g@gmail.com Sent: Thursday, February 5, 2015 12:20 PM To: İsmail Keskin; Ashutosh Trivedi (MT2013030) Cc: user@spark.apache.orgmailto:user@spark.apache.org Subject: Re: Tableau beta connector Some quick context behind how Tableau interacts with Spark / Hive can also be found at https://www.concur.com/blog/en-us/connect-tableau-to-sparksql - its for how to connect from Tableau to the thrift server before the official Tableau beta connector but should provide some of the additional context called out. HTH! On Wed Feb 04 2015 at 10:47:23 PM İsmail Keskin ismail.kes...@dilisim.commailto:ismail.kes...@dilisim.com wrote: Tableau connects to Spark Thrift Server via an ODBC driver. So, none of the RDD stuff applies, you just issue SQL queries from Tableau. The table metadata can come from Hive Metastore if you place your hive-site.xml to configuration directory of Spark. On Thu, Feb 5, 2015 at 8:11 AM, ashu ashutosh.triv...@iiitb.orgmailto:ashutosh.triv...@iiitb.org wrote: Hi, I am trying out the tableau beta connector to Spark SQL. I have few basics question: Will this connector be able to fetch the schemaRDDs into tableau. Will all the schemaRDDs be exposed to tableau? Basically I am not getting what tableau will fetch at data-source? Is it existing files in HDFS? RDDs or something else. Question may be naive but I did not get answer anywhere else. Would really appreciate if someone has already tried it, can help me with this. Thanks, Ashutosh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.orgmailto:user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.orgmailto:user-h...@spark.apache.org
Re: Tableau beta connector
Could you clarify what you mean by build another Spark and work through Spark Submit? If you are referring to utilizing Spark spark and thrift, you could start the Spark service and then have your spark-shell, spark-submit, and/or thrift service aim at the master you have started. On Thu Feb 05 2015 at 2:02:04 AM Ashutosh Trivedi (MT2013030) ashutosh.triv...@iiitb.org wrote: Hi Denny , Ismail one last question.. Is it necessary to build another Spark and work through Spark-submit ? I work on IntelliJ using SBT as build script, I have Hive set up with postgres as metastore, I can run the hive server using command *hive --service metastore* *hive --service hiveserver2* After that if I can use hive-context in my code val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) Do some processing on RDD and persist it on hive using registerTempTable and tableau can extract that RDD persisted on hive. Regards, Ashutosh -- *From:* Denny Lee denny.g@gmail.com *Sent:* Thursday, February 5, 2015 1:27 PM *To:* Ashutosh Trivedi (MT2013030); İsmail Keskin *Cc:* user@spark.apache.org *Subject:* Re: Tableau beta connector The context is that you would create your RDDs and then persist them in Hive. Once in Hive, the data is accessible from the Tableau extract through Spark thrift server. On Wed, Feb 4, 2015 at 23:36 Ashutosh Trivedi (MT2013030) ashutosh.triv...@iiitb.org wrote: Thanks Denny and Ismail. Denny ,I went through your blog, It was great help. I guess tableau beta connector also following the same procedure,you described in blog. I am building the Spark now. Basically what I don't get is, where to put my data so that tableau can extract. So Ismail,its just Spark SQL. No RDDs I think I am getting it now . We use spark for our big data processing and we want *processed data (Rdd)* into tableau. So we should put our data in hive metastore and tableau will extract it from there using this connector? Correct me if I am wrong. I guess I have to look at how thrift server works. -- *From:* Denny Lee denny.g@gmail.com *Sent:* Thursday, February 5, 2015 12:20 PM *To:* İsmail Keskin; Ashutosh Trivedi (MT2013030) *Cc:* user@spark.apache.org *Subject:* Re: Tableau beta connector Some quick context behind how Tableau interacts with Spark / Hive can also be found at https://www.concur.com/blog/en-us/connect-tableau-to-sparksql - its for how to connect from Tableau to the thrift server before the official Tableau beta connector but should provide some of the additional context called out. HTH! On Wed Feb 04 2015 at 10:47:23 PM İsmail Keskin ismail.kes...@dilisim.com wrote: Tableau connects to Spark Thrift Server via an ODBC driver. So, none of the RDD stuff applies, you just issue SQL queries from Tableau. The table metadata can come from Hive Metastore if you place your hive-site.xml to configuration directory of Spark. On Thu, Feb 5, 2015 at 8:11 AM, ashu ashutosh.triv...@iiitb.org wrote: Hi, I am trying out the tableau beta connector to Spark SQL. I have few basics question: Will this connector be able to fetch the schemaRDDs into tableau. Will all the schemaRDDs be exposed to tableau? Basically I am not getting what tableau will fetch at data-source? Is it existing files in HDFS? RDDs or something else. Question may be naive but I did not get answer anywhere else. Would really appreciate if someone has already tried it, can help me with this. Thanks, Ashutosh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Tableau beta connector
Hi, I am trying out the tableau beta connector to Spark SQL. I have few basics question: Will this connector be able to fetch the schemaRDDs into tableau. Will all the schemaRDDs be exposed to tableau? Basically I am not getting what tableau will fetch at data-source? Is it existing files in HDFS? RDDs or something else. Question may be naive but I did not get answer anywhere else. Would really appreciate if someone has already tried it, can help me with this. Thanks, Ashutosh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Tableau beta connector
Thanks Denny and Ismail. Denny ,I went through your blog, It was great help. I guess tableau beta connector also following the same procedure,you described in blog. I am building the Spark now. Basically what I don't get is, where to put my data so that tableau can extract. So Ismail,its just Spark SQL. No RDDs I think I am getting it now . We use spark for our big data processing and we want processed data (Rdd) into tableau. So we should put our data in hive metastore and tableau will extract it from there using this connector? Correct me if I am wrong. I guess I have to look at how thrift server works. From: Denny Lee denny.g@gmail.com Sent: Thursday, February 5, 2015 12:20 PM To: İsmail Keskin; Ashutosh Trivedi (MT2013030) Cc: user@spark.apache.org Subject: Re: Tableau beta connector Some quick context behind how Tableau interacts with Spark / Hive can also be found at https://www.concur.com/blog/en-us/connect-tableau-to-sparksql - its for how to connect from Tableau to the thrift server before the official Tableau beta connector but should provide some of the additional context called out. HTH! On Wed Feb 04 2015 at 10:47:23 PM İsmail Keskin ismail.kes...@dilisim.commailto:ismail.kes...@dilisim.com wrote: Tableau connects to Spark Thrift Server via an ODBC driver. So, none of the RDD stuff applies, you just issue SQL queries from Tableau. The table metadata can come from Hive Metastore if you place your hive-site.xml to configuration directory of Spark. On Thu, Feb 5, 2015 at 8:11 AM, ashu ashutosh.triv...@iiitb.orgmailto:ashutosh.triv...@iiitb.org wrote: Hi, I am trying out the tableau beta connector to Spark SQL. I have few basics question: Will this connector be able to fetch the schemaRDDs into tableau. Will all the schemaRDDs be exposed to tableau? Basically I am not getting what tableau will fetch at data-source? Is it existing files in HDFS? RDDs or something else. Question may be naive but I did not get answer anywhere else. Would really appreciate if someone has already tried it, can help me with this. Thanks, Ashutosh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.orgmailto:user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.orgmailto:user-h...@spark.apache.org
Re: Tableau beta connector
Tableau connects to Spark Thrift Server via an ODBC driver. So, none of the RDD stuff applies, you just issue SQL queries from Tableau. The table metadata can come from Hive Metastore if you place your hive-site.xml to configuration directory of Spark. On Thu, Feb 5, 2015 at 8:11 AM, ashu ashutosh.triv...@iiitb.org wrote: Hi, I am trying out the tableau beta connector to Spark SQL. I have few basics question: Will this connector be able to fetch the schemaRDDs into tableau. Will all the schemaRDDs be exposed to tableau? Basically I am not getting what tableau will fetch at data-source? Is it existing files in HDFS? RDDs or something else. Question may be naive but I did not get answer anywhere else. Would really appreciate if someone has already tried it, can help me with this. Thanks, Ashutosh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Tableau beta connector
The context is that you would create your RDDs and then persist them in Hive. Once in Hive, the data is accessible from the Tableau extract through Spark thrift server. On Wed, Feb 4, 2015 at 23:36 Ashutosh Trivedi (MT2013030) ashutosh.triv...@iiitb.org wrote: Thanks Denny and Ismail. Denny ,I went through your blog, It was great help. I guess tableau beta connector also following the same procedure,you described in blog. I am building the Spark now. Basically what I don't get is, where to put my data so that tableau can extract. So Ismail,its just Spark SQL. No RDDs I think I am getting it now . We use spark for our big data processing and we want *processed data (Rdd)* into tableau. So we should put our data in hive metastore and tableau will extract it from there using this connector? Correct me if I am wrong. I guess I have to look at how thrift server works. -- *From:* Denny Lee denny.g@gmail.com *Sent:* Thursday, February 5, 2015 12:20 PM *To:* İsmail Keskin; Ashutosh Trivedi (MT2013030) *Cc:* user@spark.apache.org *Subject:* Re: Tableau beta connector Some quick context behind how Tableau interacts with Spark / Hive can also be found at https://www.concur.com/blog/en-us/connect-tableau-to-sparksql - its for how to connect from Tableau to the thrift server before the official Tableau beta connector but should provide some of the additional context called out. HTH! On Wed Feb 04 2015 at 10:47:23 PM İsmail Keskin ismail.kes...@dilisim.com wrote: Tableau connects to Spark Thrift Server via an ODBC driver. So, none of the RDD stuff applies, you just issue SQL queries from Tableau. The table metadata can come from Hive Metastore if you place your hive-site.xml to configuration directory of Spark. On Thu, Feb 5, 2015 at 8:11 AM, ashu ashutosh.triv...@iiitb.org wrote: Hi, I am trying out the tableau beta connector to Spark SQL. I have few basics question: Will this connector be able to fetch the schemaRDDs into tableau. Will all the schemaRDDs be exposed to tableau? Basically I am not getting what tableau will fetch at data-source? Is it existing files in HDFS? RDDs or something else. Question may be naive but I did not get answer anywhere else. Would really appreciate if someone has already tried it, can help me with this. Thanks, Ashutosh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Tableau-beta-connector-tp21512.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
