For this simple example, we are importing 4 lines of 3 columns of a CSV file:
Administrator,FiveHundredAddresses1,92121
Ann,FiveHundredAddresses2,92109
Bobby,FiveHundredAddresses3,92101
Charles,FiveHundredAddresses4,92111
We are running spark-1.5.1-bin-hadoop2.6 with master and one slave, and the
JDBC thrift server and beeline client. They seem to all interconnect and are
able to communicate. From what I can understand, Hive is included in this
release in the datanucleus jars. I have configured directories to hold the
Hive files, but have no conf/hive-config.xml.
The users table has been pre-created in the beeline client using
CREATE TABLE users(first_name STRING, last_name STRING, zip_code STRING);
show tables; // it's there
For the scala REPL session on the master:
val connectionUrl = "jdbc:hive2://x.y.z.t:10000/users?user=blah&password="
val userCsvFile = sc.textFile("/home/jpowell/Downloads/Users4.csv")
case class User(first_name:String, last_name:String, work_zip:String)
val users = userCsvFile.map(_.split(",")).map(l => User(l(0), l(1), l(2)))
val usersDf = sqlContext.createDataFrame(users)
usersDf.count() // 4
usersDf.schema // res92: org.apache.spark.sql.types.StructType =
StructType(StructField(first_name,StringType,true),
StructField(last_name,StringType,true),
StructField(work_zip,StringType,true))
usersDf.insertIntoJDBC(connectionUrl,"users",true)
OR
usersDf.createJDBCTable(connectionUrl, "users", true) // w/o beeline
creation
*throws*
warning: there were 1 deprecation warning(s); re-run with -deprecation for
details
*java.sql.SQLException: org.apache.spark.sql.AnalysisException: cannot
recognize input near 'TEXT' ',' 'last_name' in column type; line 1 pos 31
* at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at
org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:406)
at
org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:119)
at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:275)
at org.apache.spark.sql.DataFrame.insertIntoJDBC(DataFrame.scala:1629)
Any ideas where I'm going wrong? Can this version actually write JDBC files
from a DataFrame?
Thanks for any help!
Jon
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/DataFrame-insertIntoJDBC-throws-AnalysisException-cannot-save-tp25422.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
