zhouhanker opened a new issue, #8480:
URL: https://github.com/apache/seatunnel/issues/8480
Hello
I want to MySQL Source Sink to Hive
dev Environment
datawarehouse cdh6.3.2
hive version 2.1.1
seatunnel 2.3.8
seatunnel conf
env {
parallelism = 2
job.mode = "BATCH"
}
source{
Jdbc {
url =
"jdbc:mysql://10.39.48.36:3306/insurance?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "Zh1028,./"
query = "select * from area"
}
}
transform {
}
sink{
Hive{
table_name="dev_insurance_ws.dim_area",
metastore_uri="thrift://cdh02:9083",
hive.hadoop.conf-path="/etc/hive/conf/hive-site.xml",
}
}
ERROR Info
2025-01-08 18:17:57,579 INFO [s.c.s.s.c.ClientExecuteCommand] [main] -
Closed HazelcastInstance ......
2025-01-08 18:17:57,579 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================
2025-01-08 18:17:57,579 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
Fatal Error,
2025-01-08 18:17:57,579 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
Please submit bug report in https://github.com/apache/seatunnel/issues
2025-01-08 18:17:57,580 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
Reason:SeaTunnel job executed failed
2025-01-08 18:17:57,582 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
Exception
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException:
SeaTunnel job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException:
ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to
create a sink for identifier 'Hive'.
at
org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSink(FactoryUtil.java:142)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.createSinkAction(MultipleTableJobConfigParser.java:665)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:596)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:221)
at
org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:114)
at
org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:182)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:160)
... 2 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/conf/Configuration;)V
at
org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:110)
at
org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:139)
at
org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveTableUtils.getTableInfo(HiveTableUtils.java:41)
at
org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.getTableInformation(HiveSink.java:235)
at
org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.<init>(HiveSink.java:85)
at
org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSinkFactory.lambda$createSink$0(HiveSinkFactory.java:61)
at
org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSink(FactoryUtil.java:139)
... 8 more
2025-01-08 18:17:57,582 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================
Exception in thread "main"
org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel
job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException:
ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to
create a sink for identifier 'Hive'.
at
org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSink(FactoryUtil.java:142)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.createSinkAction(MultipleTableJobConfigParser.java:665)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:596)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:221)
at
org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:114)
at
org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:182)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:160)
... 2 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/conf/Configuration;)V
at
org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:110)
at
org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:139)
at
org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveTableUtils.getTableInfo(HiveTableUtils.java:41)
at
org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.getTableInformation(HiveSink.java:235)
at
org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.<init>(HiveSink.java:85)
at
org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSinkFactory.lambda$createSink$0(HiveSinkFactory.java:61)
at
org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSink(FactoryUtil.java:139)
... 8 more
commod shell
./seatunnel.sh --config ../local_sh_conf/mysql_to_hive_dim_area.conf -m local
dep lib
<img width="1019" alt="image"
src="https://github.com/user-attachments/assets/14f86872-2b72-43a2-a4a3-0a34d18ea79f";
/>
Thanks
B&R !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
