Kirito-Aus opened a new issue, #6273:
URL: https://github.com/apache/iceberg/issues/6273
### Query engine
### Apache Iceberg version
1.0.0 (latest release)
### Query engine
Hive + Iceberg + MySQL + HDFS
## Please describe the bug š
### Application:
I want to use Hive+Spark to implement queries.
hostName HDFS
YARN
hadoopMaster NameNodećDataNode
NodeManagerćResourceManager
hadoopSlave0 DataNodećSecondaryNameNode NodeManager
hadoopSlave1 DataNode
NodeManager
### Question:
There is no problem when Hive is used to add, delete, modify, and query
data.
However, when using the table created by Iceberg, there is no problem when
creating a new table and querying, but an error is reported when inserting new
data.
It's ok to creating a new table and querying
hive> create table test_iceberg_tbl1(
id int ,
name string,
age int)
partitioned by (dt string)
stored by 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler';
hive> add jar /usr/local/hive/lib/iceberg-hive-runtime-1.0.0.jar;
hive> add jar /usr/local/hive/lib/libfb303-0.9.3.jar;
hive> add jar /usr/local/hive/lib/datanucleus-core-4.1.17.jar;
hive> add jar /usr/local/hive/lib/datanucleus-rdbms-4.1.19.jar;
hive> add jar /usr/local/hive/lib/datanucleus-api-jdo-4.2.4.jar;
hive> add jar /usr/local/hive/lib/mysql-connector-java-8.0.11.jar;
hive> add jar /usr/local/hive/lib/mysql-metadata-storage-0.12.0.jar;
hive> select * from test_iceberg_tbl1;
question arise
hive> insert into test_iceberg_tbl1 values (1,"zs",18,"20211212");
2022-11-25 17:02:12,348 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1669349629626_0010_m_000000_0:
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1]
org.apache.hadoop.hive.metastore.conf.MetastoreConf: Unable to find config file
hivemetastore-site.xml
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1]
org.apache.hadoop.hive.metastore.conf.MetastoreConf: Found configuration file
null
2022-11-25 17:02:12,348 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1669349629626_0010_m_000000_0 TaskAttempt Transitioned from
SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1]
org.apache.hadoop.hive.metastore.conf.MetastoreConf: Unable to find config file
metastore-site.xml
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1]
org.apache.hadoop.hive.metastore.conf.MetastoreConf: Found configuration file
null
2022-11-25 17:02:12,348 INFO [ContainerLauncher #1]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
the event EventType: CONTAINER_COMPLETED for container
container_1669349629626_0010_01_000002 taskAttempt
attempt_1669349629626_0010_m_000000_0
2022-11-25 17:02:12,605 INFO [CommitterEvent Processor #1]
DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be
ignored
2022-11-25 17:02:12,792 INFO [CommitterEvent Processor #1]
com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Starting...
2022-11-25 17:02:12,802 WARN [CommitterEvent Processor #1]
com.zaxxer.hikari.util.DriverDataSource: Registered driver with
driverClassName=com.mysql.jdbc.Driver was not found, trying direct
instantiation.
2022-11-25 17:02:14,022 ERROR [CommitterEvent Processor #1]
com.zaxxer.hikari.pool.HikariPool: HikariPool-1 - Exception during pool
initialization.
java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using
password: YES)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:127)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:862)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:444)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230)
at
com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:112)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:118)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193)
at
com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:428)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:499)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at
org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
at
org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
......
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
.......
2022-11-25 17:02:14,024 ERROR [CommitterEvent Processor #1]
DataNucleus.Datastore: Exception thrown creating StoreManager. See the nested
exception
Error creating transactional connection factory
org.datanucleus.exceptions.NucleusException: Error creating transactional
connection factory
at
org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:214)
at
org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at
org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
at
org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
at
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
at
### Debug checks:
I have tried many relevant methods on the Internet:
1. Turn off the firewall
2. Set the accessible hosts of MySQL connected users to%, that is, all can
be accessed.
3. Set the ConnectionURL in Hive configuration file to ājdbc:
mysql://hadoopMaster:3306/hive?createDatabaseIfNotExist=true ā
etc.
None can be solved
But when Hive is set to the local mode, you can insert and query normally.
It seems that there is a problem with the connection with Yarn.
### Environment
We have tried with the following versions:
I want to use Hive+Spark to implement queries.
Build Hive on hadoopMaster and use MySQL to manage metadata.
Hive 3.1.2
Hadoop 3.1.3
MySQL Distrib 5.7.40
Iceberg 1.0.0
### Question
Question
So anybody know what is going on?
I think Yarn's connection was not successfully built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
