BTW what assumption is there that the thread owner is writing to the
cluster? The thrift server is running locally on localhost:1. I concur
that JDBC to remote Hive is needed. However, this is not the impression I
get here.
df.write
.format("jdbc")
.option("url",
>From the Cloudera Documentation:
https://docs.cloudera.com/documentation/other/connectors/hive-jdbc/latest/Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf
UseNativeQuery
1: The driver does not transform the queries emitted by applications, so
the native query is used.
0: The driver
Insert mode is "overwrite", it shouldn't doesn't matter if the table
already exists or not. The JDBC driver should be based on the Cloudera Hive
version, we can't know the CDH version he's using.
On Tue, Jul 20, 2021 at 1:21 PM Mich Talebzadeh
wrote:
> The driver is fine and latest and it
The driver is fine and latest and it should work.
I have asked the thread owner to send the DDL of the table and how the
table is created. In this case JDBC from Spark expects the table to be
there.
The error below
java.sql.SQLException: [Cloudera][HiveJDBCDriver](500051) ERROR processing
Badrinath is trying to write to a Hive in a cluster where he doesn't have
permission to submit spark jobs, he doesn't have Hive/Spark metadata
access.
The only way to communicate with this third-party Hive cluster is through
JDBC protocol.
[ Cloudera Data Hub - Hive Server] <-> [Spark Standalone]
As Mich mentioned, no need to use jdbc API, using the DataFrameWriter's
saveAsTable method is the way to go. JDBC Driver is for a JDBC client
(a Java client for instance) to access the Hive tables in Spark via the
Thrift server interface.
-- ND
On 7/19/21 2:42 AM, Badrinath Patchikolla
I have trying to create table in hive from spark itself,
And using local mode it will work what I am trying here is from spark
standalone I want to create the manage table in hive (another spark cluster
basically CDH) using jdbc mode.
When I try that below are the error I am facing.
On Thu, 15
Your Driver seems to be OK.
hive_driver: com.cloudera.hive.jdbc41.HS2Driver
However this is theSQL error you are getting
Caused by: com.cloudera.hiveserver2.support.exceptions.GeneralException:
[Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error
Code: 4, SQL state:
Have you created that table in Hive or are you trying to create it from
Spark itself.
You Hive is local. In this case you don't need a JDBC connection. Have you
tried:
df2.write.mode("overwrite").saveAsTable(mydb.mytable)
HTH
view my Linkedin profile
Hi,
Trying to write data in spark to the hive as JDBC mode below is the sample
code:
spark standalone 2.4.7 version
21/07/15 08:04:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Setting default log level to
10 matches
Mail list logo
