This is an automated email from the ASF dual-hosted git repository.
zhouyao2023 pushed a commit to branch 2.3.4-release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit f562f1def2c8059c8a688b62a4ad6db89b45ea56
Author: Carl-Zhou-CN <67902676+carl-zhou...@users.noreply.github.com>
AuthorDate: Thu Jan 25 15:55:52 2024 +0800
[Release][2.3.4][release-note] Add release-note
---
release-note.md | 585 ++++++++++++++++++++++++++++++++++++++++----------------
1 file changed, 417 insertions(+), 168 deletions(-)
diff --git a/release-note.md b/release-note.md
index 831018d273..4de313a9fb 100644
--- a/release-note.md
+++ b/release-note.md
@@ -1,222 +1,471 @@
-# next-release
+# 2.3.4-release
-## Bug fix
+## Bug Fix
### Core
- [Core] [API] Fixed generic class loss for lists (#4421)
-- [Core] [API] Fix parse nested row data type key changed upper (#4459)
-- [Starter][Flink]Support transform-v2 for flink #3396
-- [Flink] Support flink 1.14.x #3963
-- [Core][Translation][Spark] Fix SeaTunnelRowConvertor fail to convert when
schema contains row type (#5170)
-
-### Transformer
-- [Spark] Support transform-v2 for spark (#3409)
-- [ALL]Add FieldMapper Transform #3781
-- [All]Add JsonPath Transform #5633
-### Connectors
-- [Elasticsearch] Support https protocol & compatible with opensearch
-- [Hbase] Add hbase sink connector #4049
-- [Clickhouse] Fix clickhouse old version compatibility #5326
+- [Starter] Fix the problem of "," being divided in [] (#5401)
+- [Core] [API] Fix ReadonlyConfig lose key error (#5565)
+- [Core] [API] Fix get bytes from LinkHashMap (#5622)
+- [Core] [API] Fix log error when multi-table sink close (#5683)
+- [Core] [API] Fix MultiTableSink return committer but sink do not support
(#5710)
+- [Core] [API] Fix the error msg when parse schema with unsupported type
(#5790)
+- [Core] [API] Fix flaky test `OptionUtilTest.test` (#5894)
+- [Core] [API] Fix SaveModeHandler not be closed (#5843)
+- [Core] [API] Fix MultiTableSinkWriter thread index always 1 (#5832)
+- [Core] [API] Fix `SeaTunnelRow::getBytesSize` not support map interface
(#5990)
+- [Core] [Common] Fix `FileUtils::createNewFile` not create new file (#5943)
+- [Core] [API] Fix Debezium format cannot parse date/time/timestamp (#5887)
+- [Starter] When inside double quotes,',' are treated as normal characters
instead of delimiters (#6042
+- [Core] [Common] Replace CommonErrorCodeDeprecated.JSON_OPERATION_FAILED
(#5978)
+- [Core] [API] Fix `Object.class` option value can not return normal value
(#6247)
+
+### Transformer-V2
+
+- [All] Fix PrimaryKey in transform (#5704)
+- [All] Fix cast to timestamp, date, time bug (#5812)
+
### Formats
-- [Canal]Support read canal format message #3950
-- [Debezium]Support debezium canal format message #3981
+
+- [Text] Allow the entry in the map to be null and allow the key in the entry
to be null (#5277)
### Connector-V2
-- [Json-format] [Canal-Json] Fix json deserialize NPE (#4195)
-- [Connector-V2] [Jdbc] Field aliases are not supported in the query of jdbc
source. (#4210)
-- [Connector-V2] [Jdbc] Fix connection failure caused by connection timeout.
(#4322)
-- [Connector-V2] [Jdbc] Set default value to false of JdbcOption:
generate_sink_sql (#4471)
-- [Connector-V2] [JDBC] Fix TiDBCatalog without open (#4718)
-- [Connector-V2] [Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer)
(#4866)
-- [Connector-V2] [Pulsar] Fix the bug that can't consume messages all the
time. (#4125)
-- [Connector-V2] [Eleasticsearch] Document description error (#4390)
-- [Connector-V2] [Eleasticsearch] Source deserializer error and inappropriate
(#4233)
-- [Connector-V2] [Kafka] Fix KafkaProducer resources have never been released.
(#4302)
-- [Connector-V2] [Kafka] Fix the permission problem caused by client.id.
(#4246)
-- [Connector-V2] [Kafka] Fix KafkaConsumerThread exit caused by commit offset
error. (#4379)
-- [Connector-V2] [Mongodb] Mongodb support cdc sink. (#4833)
-- [Connector-V2] [kafka] Fix the problem that the partition information can
not be obtained when kafka is restored (#4764)
-- [Connector-V2] [SFTP] Fix incorrect exception handling logic (#4720)
-- [Connector-V2] [File] Fix read temp file (#4876)
-- [Connector-V2] [CDC Base] Solving the ConcurrentModificationException caused
by snapshotState being modified concurrently. (#4877)
-- [Connector-V2] [Doris] update last checkpoint id when doing snapshot (#4881)
-- [Connector-v2] [kafka] Fix the short interval of pull data settings and
revise the format (#4875)
-- [Connector-v2] [RabbitMQ] Fix reduplicate ack msg bug and code style (#4842)
-- [Connector-V2] [Jdbc] Fix the error of extracting primary key column in sink
(#4815)
-- [Connector-V2] [Jdbc] Fix reconnect throw close statement exception (#4801)
-- [Connector-V2] [Jdbc] Fix sqlserver system table case sensitivity (#4806)
-- [Connector-v2] [File] Fix configuration file format and error comments
(#4762)
-- [Connector-v2] [Jdbc] Fix oracle sql table identifier (#4754)
-- [Connector-v2] [Clickhouse] fix get clickhouse local table name with closing
bracket from distributed table engineFull (#4710)
-- [Connector-v2] [CDC] Fix jdbc connection leak for mysql (#5037)
-- [Connector-v2] [File] Fix WriteStrategy parallel writing thread unsafe issue
#5546
+- [Connector-V2] [Clickhouse] Fix clickhouse old version compatibility (#5326)
+- [Connector-V2] [Clickhouse] Fix http header cover (#5446)
+- [Connector-V2] [StarRocks] Fix starrocks template sql parser (#5332)
+- [Connector-V2] [Hive] Fix the bug that hive-site.xml can not be injected in
HiveConf (#5261)
+- [Connector-V2] [Clickhouse] Fix clickhouse sink flush bug (#5448)
+- [Connector-V2] [Hive] Fix An error occurred reading an empty directory
(#5427)
+- [Connector-V2] [Hive] Fix An error occurred reading an empty directory
(#5427)" (#5487)
+- [Connector-V2] [Oss jindo] Fix the problem of jindo driver download failure.
(#5511)
+- [Connector-V2] [Oss jindo] Remove useless code (#5540)
+- [Connector-V2] [File] Fix WriteStrategy parallel writing thread unsafe issue
(#5546)
+- [Connector-V2] [CDC] Fix the cdc bug about NPE when the original table
deletes a field (#5579)
+- [Connector-V2] [Jdbc] oracle catalog create table repeat and oracle pg null
point (#5517)
+- [Connector-V2] [CDC] Fix thread-unsafe collection container in cdc
enumerator (#5614)
+- [Connector-V2] [Mongodb] Fixed unsupported exception caused by bsonNull
(#5659)
+- [Connector-V2] [File] Fix file sink `isPartitionFieldWriteInFile` occurred
exception when no columns are given (#5508)
+- [Connector-V2] [Doris] Fix RestService report NullPointerException (#5319)
+- [Connector-V2] [MaxCompute] Fix MaxCompute use not exist SCHEMA option
(#5708)
+- [Connector-V2] [Doris] Using try-with-resources to simplify the code. (#4995)
+- [Connector-V2] [Clickhouse] Fixed an out-of-order BUG with output data
fields of clickhouse-sink (#5346)
+- [Connector-V2] [Jdbc] support postgresql xml type (#5724)
+- [Connector-V2] [Jdbc] Nullable Column source have null data could be
unexpected results. (#5560)
+- [Connector-V2] [Iceberg] Iceberg source lost data with parallelism option
(#5732)
+- [Connector-V2] [Jdbc] Fix PG will not create index when using auto create
table #5721
+- [Connector-V2] [Jdbc] Fix database identifier (#5756)
+- [Connector-V2] [CDC] Fix MultiTableSink restore failed when add new table
(#5746)
+- [Connector-V2] [Postgres CDC] Fix Postgres create table test case failed
(#5778)
+- [Connector-V2] [CDC] Clean unused code (#5785)
+- [Connector-V2] [CDC] Fix state recovery error when switching a single table
to multiple tables (#5784)
+- [Connector-V2] [ElasticSearch] Fixed conversion exception of elasticsearch
array format (#5825)
+- [Connector-V2] [Jdbc] Fix read from Oracle Date type value lose time (#5814)
+- [Connector-V2] [Pulsar] Fix: update IDENTIFIER = Pulsar for
pulsar-datasource on project:seatunnel-web (#5852)
+- [Connector-V2] [Jdbc] Fix Hive-Jdbc use krb5 overwrite kerberosKeytabPath
(#5891)
+- [Connector-V2] [InfluxDB] Resolve invalid SQL in initColumnsIndex method
caused by direct QUERY_LIMIT appendage with 'tz' function. (#4829)
+- [Connector-V2] [Jdbc] Fix cdc updates were not filtering same primary key
(#5923)
+- [Connector-V2] [File] Parquet reader parsing array type exception. (#4457)
+- [Connector-V2] [Http] Fix bug http config no schema option and improve e2e
test add case (#5939)
+- [Connector-V2] [Doris] Fix DorisCatalog not implement `name` method (#5988)
+- [Connector-V2] [TDengine] Fix the degree of multiple parallelism affects
driver loading (#6020)
+- [Connector-V2] [Jdbc] Fix jdbc setFetchSize error (#6005)
+- [Connector-V2] [CDC] Fix CDC job cannot consume incremental data After
restore run (#625) (#6094)
+- [Connector-V2] [File] Fix the Issue of Abnormal Data Reading from Excel
Files (#5932)
+- [Connector-V2] [CDC] Fix NPE caused by adding a table for restore job (#6145)
+- [Connector-V2] [Jdbc] Fix dameng catalog query table sql (#6141)
+- [Connector-V2] [Jdbc] update pgsql catalog for save mode (#6080)
+- [Connector-V2] [Jdbc] Fix Spliter Error in Case of Extensive Duplicate Data
(#6026)
+- [Connector-V2] [CDC] Fix added columns cannot be parsed after job restore
(#6118)
+- [Connector-V2] [CDC] Fix negative values in CDCRecordEmitDelay metric (#6259)
+- [Connector-V2] [Oracle CDC] Fix invalid split key when no primary key (#6251)
### Zeta(ST-Engine)
-- [Zeta] Fix LogicalDagGeneratorTest testcase (#4401)
-- [Zeta] Fix MultipleTableJobConfigParser parse only one transform (#4412)
-- [Zeta] Fix missing common plugin jars (#4448)
-- [Zeta] Fix handleCheckpointError be called while checkpoint already complete
(#4442)
-- [Zeta] Fix job error message is not right bug (#4463)
-- [Zeta] Fix finding TaskGroup deployment node bug (#4449)
-- [Zeta] Fix the bug of conf (#4488)
-- [Zeta] Fix Connector load logic from zeta (#4510)
-- [Zeta] Fix conflict dependency of hadoop-hdfs (#4509)
-- [Zeta] Fix TaskExecutionService synchronized lock will not release (#4886)
-- [Zeta] Fix TaskExecutionService will return not active ExecutionContext
(#4869)
-- [Zeta] Fix deploy operation timeout but task already finished bug (#4867)
-- [Zeta] Fix restoreComplete Future can't be completed when cancel task (#4863)
-- [Zeta] Fix IMap operation timeout bug (#4859)
-- [Zeta] fix pipeline state not right bug (#4823)
-- [Zeta] Fix the incorrect setting of transform parallelism (#4814)
-- [Zeta] Fix master active bug (#4855)
-- [Zeta] Fix completePendingCheckpoint concurrent action (#4854)
-- [Zeta] Fix engine runtime error (#4850)
-- [Zeta] Fix TaskGroupContext always hold classloader so classloader can't
recycle (#4849)
-- [Zeta] Fix task `notifyTaskStatusToMaster` failed when job not running or
failed before run (#4847)
-- [Zeta] Fix cpu load problem (#4828)
-- [zeta] Fix the deadlock issue with JDBC driver loading (#4878)
-- [zeta] dynamically replace the value of the variable at runtime (#4950)
-- [Zeta] Add from_unixtime function (#5462)
-- [zeta] Fix CDC task restore throw NPE (#5507)
+- [Zeta] Fix NotifyTaskRestoreOperation npe (#5362)
+- [Zeta] Fix Zeta will close task twice error (#5422)
+- [Zeta] Disable CheckpointTimeOutTest (#5438)
+- [Zeta] Fix CDC task restore throw NPE (#5507)
+- [Zeta] Multiple sink actions of the same type have the same name (#5499)
+- [Zeta] Checkpoint exception status messages exclude state data (#5547)
+- [Zeta] Fix memory leak issues related to checkpoints (#5539)
+- [Zeta] Fix the checkpoint be blocked with long time (#5695)
+- [Zeta] Fix the problem of unstable job status (#5450)
+- [Zeta] Fix submit job api (#5702)
+- [Zeta] Set default DeployMode to DeployMode.CLIENT (#5783)
+- [Zeta] rest api submit a job with chinese name return Garbled code name
(#5870)
+- [Zeta] Fix CheckpointCoordinator report NPE when ack not existed pending
checkpoint (#5909)
+- [Zeta] Fix submit job has the same job name error. (#6041)
+- [Zeta] Fixed the problem that the return list is empty due to no status
parameters (#6040)
+- [Zeta] Fix zeta scheduler bug (#6050)
+- [Zeta] Fix job can not restore when last checkpoint failed (#6193)
+- [Zeta] [Rest-API] Submit or stop job from an inactive master node (#6217)
### E2E
-- [E2E] [Kafka] Fix kafka e2e testcase (#4520)
-- [Container Version] Fix risk of unreproducible test cases #4591
-- [E2e] [Mysql-cdc] Removing the excess MySqlIncrementalSourceIT e2e reduces
the CI time (#4738)
- [E2E] [Common] Update test container version of seatunnel engine (#5323)
- [E2E] [Jdbc] Fix not remove docker image after test finish on jdbc suite
(#5586)
+- [E2E] [ClickHouse] Enhance ClickHouse E2E testing to trigger multiple
checkpoints (#5476)
+- [E2E] Fix not remove docker image after test finish on jdbc suite (#5586)
+- [E2E] Fix `ConnectorPackageServiceContainer` miss implement
getSavePointCommand/getRestoreCommand (#5780)
+- [E2E] Fix build failed cause by `JdbcHiveIT` and `SparkSinkTest` (#5798)
+- [E2E] Fix submit job case error (#6059)
+- [E2E] Fixed action related err (#6264)
+- [E2E] Lock mysql container version to 8.0 (#6263)
+
+### CI
+
+- [CI] Fix jindo oss connector name (#5385)
+- [Build] Fix error msg when fork repository not up to date. (#5497)
+- [CI] Fix file change not be check when CI run in fork repository (#5515)
+- [CI] remove jindo dependencies (#5528)
+- [CI] Fix phoenix ci error (#5530)
+- [Build] Update build version to 2.3.4-SNAPSHOT (#5619)
+- [Build] Ensure install-plugin.sh compatibility with sh on Debian #5630
(#5631)
+- [CI] [Chore] Remove useless sonar check script (#5665
+- [Chore] Remove DISCLAIMER file (#5673)
+- [CI] Fix CI unstable problem (#5896)
+- [Build] Fix empty line in config/plugin_config causes BUILD FAILURE (#5921)
+- [CI] Fix CI not run Kudu/AmazonSQS IT when not change api (#5955)
+- [CI] Split doris e2e into separate modules (#5999)
+- [CI] Fix Dead Links checker failure (#6016)
+- [CI] Fix e2e error (#6018)
+- [Build] Updated pom.xml (#6113)
+- [Build] Solve the problem of example running failure (#6173)
+- [Build] Fix build error (#6196)
+- [CI] Fix engine client not close (#6241)
+
+### Examples
+
+- [Examples] modify the transform URL link (#5298)
## Improve
-- [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict
keys (#4634)
+- [Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of
checkstyle. (#5260)
+- [Improve][CheckStyle] Adjust the phase of spotless plugin for release
plugin. (#5607)
### Core
-- [Core] [Spark] Push transform operation from Spark Driver to Executors
(#4503)
-- [Core] [Starter] Optimize code structure & remove redundant code (#4525)
-- [Core] [Translation] [Flink] Optimize code structure & remove redundant code
(#4527)
-- [Core] [Starter] Add check of sink and source config to avoid null pointer
exception. (#4734)
-- [Core] [Flink] Remove useless stage type related codes. (#5650)
+- [Core] [API] Remove CatalogTable field in CatalogTableUtil (#5521)
+- [Core] [API] Move get schema logic from Config to ReadonlyConfig (#5534)
+- [Starter] Throw IllegalArgumentException when find multiple connector jar
for one pluginIdentifier (#5551)
+- [Core] [API] Refactor CatalogTable and add
`SeaTunnelSource::getProducedCatalogTables` (#5562)
+- [Core] [API] Support config column/primaryKey/constraintKey in schema (#5564)
+- [Core] [API] Remove useless ReadonlyConfig flatten feature (#5612)
+- [Core] [Flink & Spark] Refactor Spark/Flink execution processor (#5595)
+- [Core] [API] Mark `SeaTunnelPluginLifeCycle` as deprecated (#5625
+- [Core] [API] Support config tableIdentifier for schema (#5628)
+- [Core] [Pom] Add junit4 to the root pom (#5611)
+- [Core] [API] Remove catalog tag for config file (#5645)
+- [Core] [API] Remove useless transform code come from `setTypeInfo` (#5647)
+- [Core] [API] Make sure CatalogTable options and partitionKeys are mutable
(#5681)
+- [Core] [API] Add default implement for `SeaTunnelSource::getProducedType`
(#5670)
+- [Core] [API] Add default implement for `SeaTunnelSink::setTypeInfo` (#5682)
+- [Core] [API] Add warning for use fall back keys (#5753)
+- [Core] [API] Adjust the sleep mode of flink and spark engine to be
consistent with zeta (#5698)
+- [Core] [API] Remove `Factory` option to avoid useless info (#5754)
+- [Core] [API] Add field name to `DataTypeConvertor` to improve error message
(#5782)
+- [Core] [API] Remove use `SeaTunnelSink::getConsumedType` method and mark it
as deprecated (#5755)
+- [Core] [Common] Remove assert key word. (#5915)
+- [Core] [Common] Clean flow control code (#5991)
+- [Core] [Common] Adapt `FILE_OPERATION_FAILED` to `CommonError` (#5928)
+- [Core] [API] Add `serialVersionUID` to Column
+- [Core] [Common] Extend `SupportResourceShare` to spark/flink (#5847)
+- [Core] [API] Remove checkpoint timeout check if checkpoint disabled (#6231)
### Formats
+- [Json]Use a static object mapper instead of creating it every time (#5460)
- [Json] Remove assert key word. (#5919)
- [Formats] Replace CommonErrorCodeDeprecated.JSON_OPERATION_FAILED. (#5948)
- [Formats] Refactor exception catch for `ignoreParseErrors`. (#6065)
+- [Formats] Using number format for Decimal type in
`seatunnel-format-compatible-debezium-json` (#5803)
+- [Text] add dateTimeFormatter to parse ISO8601 (#5974)
+- [Formats] Replace `CommonErrorCodeDeprecated.JSON_OPERATION_FAILED` (#5948)
### Connector-V2
-- [Connector-V2] [CDC] Improve startup.mode/stop.mode options (#4360)
-- [Connector-V2] [CDC] Optimize jdbc fetch-size options (#4352)
-- [Connector-V2] [CDC] Fix chunk start/end parameter type error (#4777)
-- [Connector-V2] [SQLServer] Fix sqlserver catalog (#4441)
-- [Connector-V2] [StarRocks] Improve StarRocks Serialize Error Message (#4458)
-- [Connector-V2] [Jdbc] add the log for sql and update some style (#4475)
-- [Connector-V2] [Jdbc] Fix the table name is not automatically obtained when
multiple tables (#4514)
-- [Connector-V2] [S3 & Kafka] Delete unavailable S3 & Kafka Catalogs (#4477)
-- [Connector-V2] [Pulsar] Support Canal Format
-- [Connector-V2] [CDC base] Implement Sample-based Sharding Strategy with
Configurable Sampling Rate (#4856)
-- [Connector-V2] [SelectDB] Add a jobId to the selectDB label to distinguish
between tasks (#4864)
-- [Connector-V2] [Doris] Add a jobId to the doris label to distinguish between
tasks (#4839) (#4853)
-- [Connector-v2] [Mongodb]Refactor mongodb connector (#4620)
-- [Connector-v2] [Jdbc] Populate primary key when jdbc sink is created using
CatalogTable (#4755)
-- [Connector-v2] [Neo4j] Supports neo4j sink batch write mode (#4835)
-- [Transform-V2] Optimize SQL Transform package and Fix Spark type conversion
bug of transform (#4490)
-- [Connector-V2] [Common] Remove assert key word (#5915)
-- [Connector-V2] Replace CommonErrorCodeDeprecated.JSON_OPERATION_FAILED.
(#5978)
+- [Connector-V2] [IoTDB] Remove scheduler in IoTDB sink (#5270)
+- [Connector-V2] [InfluxDB] Remove scheduler in InfluxDB sink (#5271)
+- [Connector-V2] [Dynamodb] Remove scheduler in Dynamodb sink (#5248)
+- [Connector-V2] [StarRocks] Remove scheduler in StarRocks sink (#5269)
+- [Connector-V2] [CDC] avoid listing tables under unnecessary databases (#5365)
+- [Connector-V2] [Jdbc] Refactor AbstractJdbcCatalog (#5096)
+- [Connector-V2] [CDC] Support flink running cdc job (#4918)
+- [Connector-V2] [Assert] support 'DECIMAL' type and Fix 'Number' type
precision issue (#5479)
+- [Connector-v2] [Redis] Redis support select db (#5570)
+- [Connector-v2] [CDC] Use Source to output the CatalogTable (#5626)
+- [Connector-v2] [SqlServer CDC] add dataType datetimeoffset (#5548)
+- [Connector-v2] [Jdbc] Support read multiple tables (#5581)
+- [Connector-v2] [SqlServer CDC] Unified sqlserver TypeUtils type conversion
mode (#5668)
+- [Connector-v2] [Http] improve http e2e test (#5655)
+- [Connector-v2] [AmazonDynamicDB] add amazondynamicdb source split (#5275)
+- [Connector-v2] [File] parquet use system timezone (#5605)
+- [Connector-v2] [Amazonsqs] Change `amazonsqs` to `AmazonSqs` as connector
identifier (#5742)
+- [Connector-v2] [File] unifiy option between file source/sink and update
document (#5680)
+- [Connector-v2] [AmazonDynamicDB] Code clean for AmazonDynamoDB connector
(#5791)
+- [Connector-v2] [MongoDB] Implement TableSourceFactory to create mongodb
source
+- [Connector-v2] [Jdbc] Optimize catalog-table metadata merge logic (#5828)
+- [Connector-v2] [Jdbc] Rename `getCountSql` to `getExistDataSql` (#5838)
+- [Connector-v2] [ClickHouse] Speed up ClickhouseFile Local generate a mmap
object (#5822)
+- [Connector-v2] [Jdbc] Improve Jdbc connector error message when datatype
unsupported (#5864)
+- [Connector-v2] [Jdbc] Reduce the time cost of getCatalogTable in jdbc (#5908)
+- [Connector-v2] [StarRocks] Support create varchar field type in StarRocks
(#5911)
+- [Connector-v2] [StarRocks] add http socket timeout. (#5918)
+- [Connector-v2] [File] Clean memory buffer of `JsonWriteStrategy` &
`ExcelWriteStrategy` (#5925)
+- [Connector-v2] [StarRocks] StarRocks support create table template with
unique key (#5905)
+- [Connector-v2] [CDC] Disable memory buffering when `exactly_once` is turned
off (#6017)
+- [Connector-v2] [Doris] Add batch flush in doris sink (#6024
+- [Connector-v2] [Paimon] Adaptation Paimon 0.6 Version (#6061)
+- [Connector-v2] [File] Make Oss implement source factory and sink factory
(#6062)
+- [Connector-v2] [File] Disable HDFSFileSystem cache (#6039)
+- [Connector-v2] [Jdbc] Shade hikari in jdbc connector (#6116)
+- [Connector-v2] [Jdbc] Supports Sqlserver Niche Data Types (#6122)
+- [Connector-v2] [Kafka] Remove useless code for kafka connector (#6157)
+- [Connector-v2] [Doris] Improve doris sink to random use be (#6132)
+- [Connector-v2] [Http] Increase custom configuration timeout. (#6223)
+- [Connector-v2] [Pulsar] Improve pulsar throughput performance. (#6234)
+- [Connector-v2] [SqlServer CDC] Support `int identity` type in sql server
(#6186)
+- [Connector-v2] [Doris] Doris stream load use FE instead of BE (#6235)
+- [Connector-v2] [Postgres CDC] Fix name typos (#6248)
+- [Connector-v2] [Tdengine] support read bool column from tdengine (#6025)
+- [Connector-v2] [Jdbc] Use PreparedStatement to sample data from column
(#6242)
+
### CI
-- [CI] Fix error repository name in ci config files (#4795)
+- [CI] update sql-udf document (#5197)
- [CI][E2E][Zeta] Increase Zeta checkpoint timeout to avoid
connector-file-sftp-e2e failed frequently (#5339)
+- [CI] Fix phoenix ci error
+- [Build] Put `seatunnel-hadoop3-3.1.4-uber.jar` into release binary package
(#5743)
+- [Test] Make sure the value in spark not be reused. (#5767)
+- [Test] Move MaxCompute test case file (#5786)
+- [CI] Always run all module unit test (#5800)
+- [Test] Change System.out.println to log output. (#5912)
+- [Test] Add some test case for command useage
+- [Test] Fix sql server catalog test case failed (#6128)
+- [Test] Fix JobMetricsTest unstable (#6152)
+- [Test] Fix ConnectorSpecificationCheckTest not work (#5820
+
+### E2E
+
+- [E2E] Remove unnecessary code to reduce disk pressure (#5613)
+- [E2E] Enable IT case for Oceanbase Mysql mode (#5697)
+- [E2E] load driver class from url on demand (#5712)
+- [E2E] Jdbc test checking data consistency. (#5734)
+- [E2E] Enable e2e log output and disable console sink log (#5879)
+- [E2E] Improve e2e log for all engines (#5936)
+- [E2E] Enhanced stability of Kudu E2E (#6258)
### Zeta(ST-Engine)
-- [Zeta] Support run the server through daemon mode (#4161)
-- [Zeta] Change ClassLoader To Improve the SDK compatibility of the client
(#4447)
-- [Zeta] Client Support Async Submit Job (#4456)
-- [Zeta] Add more detailed log output. (#4446)
-- [Zeta] Improve seatunnel-cluster.sh (#4435)
-- [Zeta] Reduce CPU Cost When Task Not Ready (#4479)
-- [Zeta] Add parser log (#4485)
-- [Zeta] Remove redundant code (#4489)
-- [Zeta] Remove redundancy code in validateSQL (#4506)
-- [Zeta] Improve JobMetrics fetch performance (#4467)
-- [Zeta] Reduce the operation count of imap_running_job_metrics (#4861)
-- [Zeta] Speed up listAllJob function (#4852)
-- [Zeta] async execute checkpoint trigger and other block method (#4846)
-- [Zeta] Reduce the number of IMAPs used by checkpointIdCounter (#4832)
-- [Zeta] Cancel pipeline add retry to avoid cancel failed. (#4792)
-- [Zeta] Improve Zeta operation max count and ignore NPE (#4787)
-- [Zeta] Remove serialize(deserialize) cost when use shuffle action (#4722)
-- [zeta] Checkpoint exception status messages exclude state data (#5547)
+- [Zeta] Optimize Test Cases
`CheckpointTimeOutTest.testJobLevelCheckpointTimeOut` (#5403)
+- [Zeta] Improve dependency packages (#5624)
+- [Zeta] Change hard code config key to reference (#5618)
+- [Zeta] Change class name for `RestJobExecutionEnvironment` implement (#5671)
+- [Zeta] Change default Zeta client JVM heap value (#5674)
+- [Zeta] Move generate_client_protocol.sh to engine module (#5667)
+- [Zeta] Optimize SeaTunnel Zeta engine Jar package upload logic (#5542)
+- [Zeta] Move `RestJobExecutionEnvironment` to rest package (#5764)
+- [Zeta] Remove `result_table_name` from action name(checkpoint state key)
(#5779)
+- [Zeta] Refactor jar package service module (#5763)
+- [Zeta] Expose client cluster-connect-timeout-millis to yaml (#5868)
+- [Zeta] Reduce checkpoint completed log (#5916)
- [Zeta] Remove assert key words (#5947)
+- [Zeta] Adjusting the log level for factory validation implementation. (#6153)
+- [Zeta] Ignore useless wrong target slot error (#6135)
+- [Zeta] Add restore when commit failed (#6101)
+
+### Transformer-V2
+
+- [All] Code clean for SeaTunnel transform (#5810)
+- [All] Add common error for transform (#5815)
## Feature
### Core
-- [Core] [API] Support convert strings as List<T> option (#4362)
-- [Core] [API] Add copy method to Catalog codes (#4414)
-- [Core] [API] Add options check before create source and sink and transform
in FactoryUtil (#4424)
-- [Core] [Shade] Add guava shade module (#4358)
-- [Core] [Spark] Support SeaTunnel Time Type (#5188)
+- [Core] [API] Add job-level configuration for checkpoint timeout. (#5222)
+- [Core] [API] Catalog add Case Conversion Definition (#5328)
+- [Core] [API] Add InMemoryCatalog for test and add new
getCatalogTableFromConfig method (#5485)
- [Core] [Flink] Support Decimal Type with configurable precision and scale
(#5419)
-- [Core] [API] Support hocon style declare row type in generic type (#6187)
+- [Core] [API] Add `init` and `restoreCommit` method in
`SinkAggregatedCommitter` (#5598)
+- [Core] [Flink] Support flow control in Flink (#5509)
+- [Core] [Spark] Support SeaTunnel Time Type. (#5188)
+- [Core] [Flink] Remove useless stageType. (#5650)
+- [Core] [API] Support multi-table sink (#5620)
+- [Core] [Spark] Support flow control in Spark (#5510)
+- [Core] [Flink] Add external configuration parameters (#5480)
+- [Core] [API] Remove all useless `prepare`, `getProducedType` method (#5741)
+- [Core] [Common] Introduce new error define rule (#5793)
+- [Core] [Common] Remove useless DeserializationFormatFactory and its
implement (#5880)
+- [Core] [API] Replace SeaTunnelRowType with TableSchema in the
JdbcRowConverter
+- [Core] [Flink] Upgrade flink source translation (#5100)
+- [Core] [API] Add unsupported datatype check for all catalog (#5890)
+- [Core] [Flink] Support record metrics in flink engine (#6035)
+
### Connector-V2
-- [Connector-V2] [CDC] [SQLServer] Support multi-table read (#4377)
-- [Connector-V2] [Kafka] Kafka source supports data deserialization failure
skipping (#4364)
-- [Connector-V2] [Jdbc] [TiDB] Add TiDB catalog (#4438)
-- [Connector-V2] [File] Add file excel sink and source (#4164)
-- [Connector-v2] [Snowflake] Add Snowflake Source&Sink connector (#4470)
-- [Connector-V2] [Pular] support read format for pulsar (#4111)
-- [Connector-V2] [Paimon] Introduce paimon connector (#4178)
-- [Connector V2] [Cassandra] Expose configurable options in Cassandra (#3681)
-- [Connector V2] [Jdbc] Supports GEOMETRY data type for PostgreSQL (#4673)
-- [Connector V2] [Jdbc] Supports Kingbase database (#4803)
-- [Transform-V2] Add UDF SPI and an example implement for SQL Transform plugin
(#4392)
-- [Transform-V2] Support copy field list (#4404)
-- [Transform-V2] Add support CatalogTable for FieldMapperTransform (#4423)
-- [Transform-V2] Add CatalogTable support for ReplaceTransform (#4411)
-- [Transform-V2] Add Catalog support for FilterRowKindTransform (#4420)
-- [Transform-V2] Add support CatalogTable for FilterFieldTransform (#4422)
-- [Transform-V2] Add catalog support for SQL Transform plugin (#4819)
-- [Connector-V2] [Assert] Support check the precision and scale of Decimal
type (#6110)
+- [Connector-V2] [SqlServer CDC] Support multi-table read (#4377)
+- [Connector-V2] [Jdbc] Jdbc database support identifier (#5089)
+- [Connector-V2] [Jdbc] jdbc connector supports Kingbase database (#4803)
+- [Connector-V2] [Jdbc] Add tidb datatype convertor (#5440)
+- [Connector-V2] [Jdbc] Add Dameng catalog (#5451)
+- [Connector-V2] [File] Supports writing column names when the output type is
file (CSV) (#5459)
+- [Connector-V2] [File] FILE_FORMAT_TYPE is text/csv ,add parameter
BaseSinkConfig.ENABLE_HEADER_WRITE: #5566 (#5567)
+- [Connector-V2] [CDC] Support for preferring numeric fields as split keys
(#5384)
+- [Connector-V2] [File] Support read empty directory (#5591)
+- [Connector-V2] [Fake&Assert] Add `table-names` from FakeSource/Assert to
produce/assert multi-table (#5604)
+- [Connector-V2] [Jdbc] Add OceanBase catalog (#5439)
+- [Connector-V2] [File] Support `LZO` compress on File Read (#5083)
+- [Connector-V2] [MongoDB CDC] Support MongoDB CDC running on flink (#5644)
+- [Connector-V2] [Jdbc] Supporting more ways to configure connection
parameters. (#5388)
+- [Connector-V2] [Kafka] KafkaSource use Factory to create source (#5635)
+- [Connector-V2] [Jdbc] Add connector amazonsqs (#5367)
+- [Connector-V2] [Jdbc] Support catalog in MaxCompute Source (#5283)
+- [Connector-V2] [Kudu] Refactor Kudu functionality and Sink support CDC data.
(#5437)
+- [Connector-V2] [Mysql CDC] Optimize the default value range of mysql
server-id to reduce conflicts. (#5550)
+- [Connector-V2] [Http] HTTP supports page increase #5477 (#5561)
+- [Connector-V2] [Jdbc] Add Save Mode function and Connector-JDBC (MySQL)
connector has been realized (#5663)
+- [Connector-V2] [Jdbc] Support XMLTYPE data integration #5716 (#5723)
+- [Connector-V2] [Jdbc] Support Hive JDBC Source Connector (#5424)
+- [Connector-V2] [Http] Http paramter support custom encryption (#5727)
+- [Connector-V2] [Kudu] Support TableSourceFactory/TableSinkFactory on kudu
(#5789)
+- [Connector-V2] [File] LocalFileSource support multiple table
+- [Connector-V2] [Fake] FakeSource support generate different CatalogTable for
MultipleTable (#5766)
+- [Connector-V2] [Kudu] Support multi-table source read (#5878
+- [Connector-V2] [Http] Support TableSourceFactory/TableSinkFactory on http
(#5816)
+- [Connector-V2] [Redis] Support TableSourceFactory/TableSinkFactory on redis
(#5901)
+- [Connector-V2] [Jdbc] Fix split key not support BigInteger type
+- [Connector-V2] [File] LocalFile sink support multiple table (#5931)
+- [Connector-V2] [Doris] Doris Catalog (#5175)
+- [Connector-V2] [Kudu] Support multi-table sink feature for kudu (#5951)
+- [Connector-V2] [File] Support using multiple hadoop account (#5903)
+- [Connector-V2] [File] Put Multiple Table File API to File Base Module (#6033)
+- [Connector-V2] [Paimon] Flink table store failed to prepare commit (#6057)
+- [Connector-V2] [File] Add multiple table file sink to base (#6049)
+- [Connector-V2] [Jdbc] add JDBC source support string type as partition key
(#6079)
+- [Connector-V2] [File] Support read .xls excel file (#6066)
+- [Connector-V2] [CDC] Support read no primary key table (#6098)
+- [Connector-V2] [Assert] Support check the precision and scale of Decimal
type. (#6110)
+- [Connector-V2] [Hbase] support array data. (#6100)
+- [Connector-V2] [File] FTP source/sink add ftp connection mode (#6077) (#6099)
+- [Connector-V2] [Jdbc] update sqlserver catalog for save mode (#6086)
+- [Connector-V2] [CDC] Support custom table primary key (#6106)
+- [Connector-V2] [Doris] Support SaveMode on Doris (#6085)
+- [Connector-V2] [Jdbc] update oracle catalog for save mode (#6092)
+- [Connector-V2] [ElasticSearch] add elasticsearch save_mode (#6046) (#6092)
+- [Connector-V2] [Jdbc] Improve get column sql compatibility (#5664)
+- [Connector-V2] [Jdbc] Improve get column sql compatibility (#5664)
+- [Connector-V2] [Pulsar] Add Pulsar Sink Connector. (#4382)
+- [Connector-V2] [StarRocks] add starrocks save_mode (#6029)
+- [Connector-V2] [Oracle CDC] Support for oracle cdc (#5196)
+- [Connector-V2] [Doris] Add Doris ConnectorV2 Source (#6161)
+- [Connector-V2] [Jdbc] Support `uuid` in postgres jdbc (#6185)
+- [Connector-V2] [Oracle CDC] Support read no primary key table (#6209)
+- [Connector-V2] [Oracle CDC] Fix jdbc setFetchSize error (#6210)
+- [Connector-V2] [Oracle CDC] Fix state recovery error when switching a single
table to multiple tables (#6211)
+- [Connector-V2] [Oracle CDC] Clean unused code (#6212)
+- [Connector-V2] [File] add s3file save mode function (#6131)
+- [Connector-V2] [Oracle CDC] Support custom table primary key (#6216)
+- [Connector-V2] [CDC] Add date type and float type column split support
(#6160)
+- [Connector-V2] [Postgres CDC] Support for Postgres cdc (#5986)
+- [Connector-V2] [Postgres CDC] Update jdbc fetchsize (#6245)
+- [Connector-V2] [CDC] Disable exactly_once by default to improve stability
(#6244)
+- [Connector-V2] [CDC] Support Short and Byte Type in spliter (#6027)
+- [Connector-V2] [Jdbc] Improve query Approximate Total Row Count of a Table
(#5972)
### Zeta(ST-Engine)
-- [Zeta] Support for mixing Factory and Plugin SPI (#4359)
-- [Zeta] Add get running job info by jobId rest api (#4140)
-- [Zeta] Add REST API To Get System Monitoring Information (#4315)
-- [Transform V2 & Zeta] Make SplitTransform Support CatalogTable And
CatalogTable Evolution (#4396)
-- [Zeta] Move driver into lib directory and change operation count (#4845)
-- [Zeta] Add Metaspace size default value to config file (#4848)
-- [Zeta] Reduce the frequency of fetching data from imap (#4851)
-- [Zeta] Add OSS support for Imap storage to cluster-mode type (#4683)
-- [Zeta] Improve local mode startup request ports (#4660)
+- [Zeta] Add the UNKNOWABLE job status. (#5303)
+- [Zeta] Support flow control in zeta (#5502)
+- [Zeta] [REST-API] Stop a running job. (#5512)
+- [Zeta] Support Zeta engine on Kubernetes (#5594)
+- [Zeta] In batch mode, checkpoint can be disabled (#5914)
+- [Zeta] Change skip trigger checkpoint log level to debug (#5954)
+- [Zeta] add new job status `DOING_SAVEPOINT` and `SAVEPOINT_DONE` (#5917)
+- [Zeta] Add waitForJobCompleteV2 api (#5965)
+- [Zeta] Can submit job with rest api to Zeta master node automaticly (#5950)
+- [Zeta] [REST-API] get finished jobs info (#5949)
+- [Zeta] Fix transform action return same name (#6034)
+- [Zeta] Unify job env parameters (#6003)
+- [Zeta] Add TaskGroupLocation to the thread name of the TaskExecutionService
(#6095)
+- [Zeta] Using G1 as defaut garbage colletor in zeta (#6114)
+- [Zeta] Fix start with savepoint with no checkpoint file error (#6215)
+- [Zeta] Support hocon style declare row type in generic type (#6187)
+
+### CI
+
+- [Bin] Add .bat script for all shell script. (#5445)
+- [INFRA] Move CI to running on fork repository container (#5495)
+- [Build] Remove `connector/seatunnel` directory (#5489)
+- [INFRA] Update PR template to add test and user change question (#5486)
+- [INFRA] Add log for notify_test_workflow.yml to trace error reason
+- [INFRA] Fix notify_test_workflow.yml not stable
+- [Test]Remove docker image after test finish on jdbc suite (#5568)
+- [Test] Add test for ResourceManager to keep task will be deployed in
different node (#5518)
+- [Chore] Remove useless `.scalafmt.conf` file (#5616)
+- [LICENSE] Add hadoop license (#6067)
+- [Build] Add seatunnel-spark-3-starter.jar into release package (#6044)
+- [Test] Reduce repeated catalog test times (#6207)
+- [CI] Make sure notify_test_workflow.yml error will be throwed (#6226)
+
+### Formats
+
+- [Ogg] Support read ogg format message #4201 (#4225)
+- [Json] Remove assert key word. (#5919)
+- [Avro] support avro format (#5084)
+- [Formats] Refactor exception catch for `ignoreParseErrors`. (#6065)
+- [Avro] improve avro format convert (#6082)
+
+### Transformer-V2
+
+- [All] add JsonPath transform (#5632)
+- [All] Support SqlTransform Not Like Expression (#5768)
+- [All] add from_unixtime function (#5462)
+- [All] Support case when Expression (#6123)
## Docs
-- [Docs] Optimizes part of the Doris and SelectDB connector documentation
(#4365)
-- [Docs] Fix docs code style (#4368)
-- [Docs] Update jdbc doc and kafka doc (#4380)
-- [Docs] Fix max_retries default value is 0. (#4383)
-- [Docs] Fix markdown syntax (#4426)
-- [Docs] Fix Kafka Doc Error Config Key "kafka." (#4427)
-- [Docs] Add Transform to Quick Start v2 (#4436)
-- [Docs] Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes
(#4788)
-- [Docs] Fix Mysql sink format doc (#4800)
-- [Docs] Add the generate sink sql parameter for the jdbc sink document (#4797)
-- [Docs] Add the generate sink sql parameter And example (#4769)
-- [Docs] Redshift add defaultRowFetchSize (#4616)
-- [Docs] Refactor connector-v2 docs using unified format Mysql (#4590)
-- [Docs] Add Value types in Java to Schema features (#5087)
-- [Docs] Replace username by user in the options of FtpFile (#5421)
-- [Docs] Add how to configure logging related parameters of SeaTunnel-E2E Test
(#5589)
-- [Docs] Remove redundant double quotation mark from an example code (#5845)
-- [Docs] Add Hive JDBC reference value (#5882)
+- [Docs] Refactor connector-v2 docs using unified format Feishu (#5343)
+- [Docs] Refactor IoTDB sink docs (#5306)
+- [Docs] Correct word errors (#5360)
+- [Docs] Improved iceberg documentation (#5335)
+- [Docs] Use short url https://s.apache.org/seatunnel-slack to replace long
URL (#5363)
+- [Docs] improve http doc param body des (#5368)
+- [Docs] Refactor connector-v2 docs using unified format Slack (#5344)
+- [Docs] update sql-udf document (#5197)
+- [Docs] Refactor MySQL-CDC docs (#5302)
+- [Docs] Replace username by user in the options description of FtpFile (#5421)
+- [Docs] update iotdb document (#5404)
+- [Docs] Added the mysql Connector Document version title Example pr (#5249)
+- [Docs] add Parallel parallelism (#5310)
+- [Docs] Http source option key poll_interval_ms different in source code
(#5430)
+- [Docs] Fix error examples of kafka sink (#5527)
+- [Docs] Improve Console Sink document (#5230)
+- [Docs] Add how to change logging configuration of e2e test (#5589)
+- [Docs] Add RocketMq connector (#5361)
+- [Docs] Fix build status in README.md not update (#5574)
+- [Docs] hdfsFile file_format changed to file_format_type (#5653)
+- [Docs] Improved README.md (#5662)
+- [Docs] Add FakeSource connector documentation (#5255)
+- [Docs] Introduce SeaTunnel web project in README.md (#5634)
+- [Docs] Added Table of Contents and FAQs to README (#5693)
+- [Docs] Update quick-start-spark.md (#5795)
+- [Docs] Add Socket connector doc #5255 (#5287)
+- [Docs] Improve file sink doc (#5799)
+- [Docs] Add SqlServer connector documentation (#5498)
+- [Docs] update (#5808)
+- [Docs] Add hive jdbc reference value (#5882)
+- [Docs] Checkpoint-Storage description is incorrect. (#5883)
+- [Docs] Reconstruct the OssFile connector document (#5233)
+- [Docs] Fix oss connector can not run bug (#6010)
+- [Docs] update doc for jdbc-connector (#5765)
+- [Docs] Add V2 connector jdbc document parameter can speed up data import PR
(#6176)
+- [Docs] Modify some document title specifications (#6237)
+- [Docs] Reconstruct the Socket Source and SftpFile connector document (#5386)
+- [Docs] Improve doc of driver put path
+- [Docs] Correct the introduction of array element types and map key types
(#6261)
+
+
