This is an automated email from the ASF dual-hosted git repository.
jackietien pushed a change to branch rc/2.0.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git
from 24c23ce09a8 Support invoke table function without TABLE keyword
new 01d4b73d586 Throw exception when wal entry is too large
new 2579128f4e1 Separate fully dirty files by partially dirty files task
(#15341)
new 1f4f6b5ab56 [RTO/RPO] Fix: NullPointer in ClusterTopology #15347
new 44e17ce5ee5 reopen discussion in .asf.yml config (#15335)
new 25186180ae7 Pipe: aggregate tablets with different measurements under
the same device before write into tsfile (#15193)
new 8c6c6eef316 Subscription: detect outdated subscription event for
better consumer exception control (#15326)
new 99685fb23bf Bump netty version to 4.1.119.Final
new d1f271e5f54 Load: check if the DataRegion is Deleted before loading
the new tsfile (#15346)
new c28897186e7 Pipe: Fixed the validation logic of OpcDaConnector &
Optimized the error message when the system is not Windows (#15339)
new e38636fb542 Pipe: use mem table to batch write tree data into tsfile
(#15276)
new 8996e718aa7 Pipe: Fix NPE caused by InsertNodeMemoryEstimator
estimating Binary array length (#15321)
new c4e10466942 Pipe: Fixed the IndexOutOfBound exception & Trimmed the
un-transferred schema region & Refactor for meta transfer (#15337)
new eb409d1b3a2 Load: Supports metrics in the Async Analyzer stage (#15312)
new 272adf852e8 Pipe/Load: Remove MD5 check for identically named TSFile
to speed up Active Load (#15311)
new b966bd014a5 [Bug] When using SubscriptionPullConsumer to consume data,
an error is reported when configuring ConsumerConstant.NODE_URLS_KEY as the
cluster address. (#15322)
new 98f6c933913 Make deserialize of AggregationDescriptor AttributesMap
use LinkedHashMap
new 12ce547b485 refactor: load max page&chunk configuration in
ReadOnlyMemchunk and M… (#15309)
new 6c1bbe5b7e8 perf: Adapt to Win11(wmic to powershell and compatibility
adjustment) (#15350)
new 4e92c775b63 Update release info after 2.0.2 and 1.3.4 released
new 5ad5a4760b0 update session source when have no permission in
ExportSchemaTable (#15355)
new a0aecfb7898 Fix process of last query if intermediate state of last
cache is read when query concurrently
new d282d695a58 Fix sort properties process of AggregationNode when
generate distribution plan
new fff83988bc6 fix an encrypt_key_import bug #15365
new 2fa12a6d024 Include thrift genarated python code in python client to
support AINode (#15348)
new bab8a51013e Pipe: switch to IdentityHashMap backed set to avoid
infinite loop in table tsfile builder (#15371)
new 5b78e3e8b29 Pipe: fix potential ClassCastException when casting
LocalDate[] in PipeTreeModelTsFileBuilderV2 (#15375)
new 74fd7c6d576 Load: Do not clear schema cache when last cache is
disabled (#15380)
new 0c2a3f09b48 Pipe: aggregate tablets with different measurements under
the same table before write into tsfile (#15372)
new 561ece9215f Subscription: obtain database name for message payload
subscribed by table model consumer (#15374)
new 2e0482c3f58 Pipe: use mem table to batch write table data into tsfile
(#15373)
new 02699186884 Fix error code when Row appears in query
new 798e0d96638 Subscription: implemented runtime permission check (#15376)
new 71bc3c44892 feat: disabled history for -e sql (#15389)
new 51a2d3e3fd2 Load: Fixed the bug that an empty table database may not
check privilege for auto-creation (#15366)
new d3cdaee6433 Pipe: Fixed the deadlock cause by terminate event
reporting & Added sink.batch.max-delay-ms to enable delicate control over batch
delay & Fixed the bug of premature halt in extractor snapshot mode cause by
real-time-first transfer & Stabilized the trigger of default batch sending &
Added "isNeedToReport" getter in PipeRawTabletInsertionEvent for user-defined
plugins & Reduce logs from PipeEventCommitter (#15377)
new a2ec01c1719 Load: Enhanced the table auto-creation logic (#15265)
new 8217b421bf3 Pipe: Fix batch type can not be changed (Introduced in
#15377) & Enhance close() in PipeTsFileResource (#15401)
new 57ec918d16a Load IT: Ignore unstable
testLoadAutoCreateWithoutInsertPermission (#15405)
new 508f592dec7 Load IT: Stabilized the
testLoadAutoCreateWithoutInsertPermission test (#15408)
new 4868bf7566f docker: support configuring JVM memory environment
variables in docker-compose file:IOTDB_JMX_OPTS for
datanode,CONFIGNODE_JMX_OPTS for confignode (#15413)
new 7e6a59bc1f8 Pipe: Implement hot loading of Pipe configuration
parameters (#15292)
new 0279c6b6ffa recover the place of setting ON_HEAP_MEMORY and
OFF_HEAP_MEMORY to fix atmos (#15421)
new 00bfdbc650c Fix memory leak in wal compressed buffer (#15418)
new e790d56d8e8 Fixed the NPE caused by concurrent template unset /
activation #15420
new 535485e9380 IoTV2: Try to fix tsfile corruption & receiver dir clean
(#15410)
new 97aef4a57db ShutdownNow all pipe connector subTask works to interrupt
all threads blocking in lock. (#15411)
new 6de079b6d86 fix: dockerfile adapter script (#15428)
new b21b1301d5f [remove datanode] GCR load balancing implement for
removing datanode (#15282)
new a31a18987df [remove datanode] Fix IoTDBRemoveDataNodeNormalIT #15429
new 92a884a5b66 IoTV2: Fix global client manager close by mistake when
running #15415
new 72083f31946 DAL: Fix thread leak when DN exit. #15395
new e4a6d0e62ac Procedure: Fix the concurrency error during
StateMachineProcedure snapshot (#15417)
new 12b8537044a bump ratis to 3.2.0-3247c7f-SNAPSHOT (#15427)
new 27bc6e0d9d8 Enhanced JDBC: allow tree model data view in DBeaver
(#15431)
new 3ecea88bb9a Append TTL logs for PartitionTableAutoCleaner #15437
new f606ec2f06a Invalidating partition cache for all datanodes except the
unknown ones when deleting database #15436
new 414d42b72c2 Preallocate array list & update TsFile version (#15434)
new 8134d919eb5 Pipe: Fixed the bug that disruptor queue close may block
forever by not waiting real-time tsFile close (#15464)
new 5a71566600e Fixed the cached TTL & time precision problem of
PartitionTableAutoCleaner (#15455)
new 3e22b7593a5 Use reference time position for PartitionTableAutoCleaner
The 60 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 3 +
.github/workflows/multi-language-client.yml | 6 +-
RELEASE_NOTES.md | 100 +++
.../DockerCompose/docker-compose-host-3c3d.yml | 2 +
.../DockerCompose/docker-compose-standalone.yml | 4 +-
.../main/DockerCompose/replace-conf-from-env.sh | 11 +-
docker/src/main/Dockerfile-1c1d | 2 +-
.../org/apache/iotdb/TableModelSessionExample.java | 15 +-
.../apache/iotdb/TableModelSessionPoolExample.java | 2 +-
.../TableModelSubscriptionSessionExample.java | 1 +
.../iotdb/it/utils/TsFileTableGenerator.java | 7 +-
.../partition/IoTDBPartitionTableAutoCleanIT.java | 7 +-
....java => IoTDBPartitionTableAutoCleanUSIT.java} | 34 +-
.../org/apache/iotdb/db/it/IoTDBLoadTsFileIT.java | 68 +-
.../iotdb/db/it/aggregation/IoTDBCountIfIT.java | 13 +
.../org/apache/iotdb/db/it/utils/TestUtils.java | 21 +
.../pipe/it/dual/tablemodel/TableModelUtils.java | 105 +--
.../manual/basic/IoTDBPipeWithLoadIT.java | 107 +++
.../IoTDBPipeTypeConversionISessionIT.java | 18 +-
.../manual/IoTDBPipeTypeConversionISessionIT.java | 5 +-
.../it/db/it/IoTDBAuthenticationTableIT.java | 2 +-
.../relational/it/db/it/IoTDBInsertTableIT.java | 2 +-
.../it/query/old/orderBy/IoTDBOrderByTableIT.java | 33 +-
.../it/query/recent/IoTDBComplexQueryIT.java | 84 +++
.../it/query/recent/IoTDBTableAggregationIT.java | 8 +
.../iotdb/relational/it/schema/IoTDBTableIT.java | 8 +-
.../it/session/IoTDBSessionRelationalIT.java | 2 +-
.../pool/IoTDBInsertTableSessionPoolIT.java | 2 +-
.../it/local/IoTDBSubscriptionPermissionIT.java | 236 +++++++
...TDBConsumer2With1TopicShareProcessTsfileIT.java | 23 +-
.../pattern/IoTDBTSPatternPullConsumeTsfileIT.java | 2 -
...TDBConsumer2With1TopicShareProcessTsfileIT.java | 8 +-
.../IoTDBTSPatternTsfilePushConsumerIT.java | 2 +-
.../cli/src/assembly/resources/sbin/start-cli.bat | 3 +
.../cli/src/assembly/resources/tools/backup.bat | 7 +-
.../src/main/java/org/apache/iotdb/cli/Cli.java | 1 +
.../apache/iotdb/tool/data/AbstractDataTool.java | 6 +-
.../apache/iotdb/tool/data/ImportDataTable.java | 9 +-
.../iotdb/tool/schema/ExportSchemaTable.java | 19 +-
.../iotdb/tool/tsfile/ImportTsFileRemotely.java | 6 +-
iotdb-client/client-py/pom.xml | 13 +-
.../org/apache/iotdb/jdbc/IoTDBConnection.java | 2 -
.../apache/iotdb/jdbc/IoTDBDatabaseMetadata.java | 4 +-
.../java/org/apache/iotdb/rpc/TSStatusCode.java | 1 +
.../rpc/subscription/annotation/TableModel.java | 12 +-
.../rpc/subscription/config/ConsumerConfig.java | 8 +
.../subscription/payload/poll/ErrorPayload.java | 4 +
.../subscription/payload/poll/FileSealPayload.java | 27 +-
.../subscription/payload/poll/TabletsPayload.java | 41 +-
.../response/PipeSubscribeHeartbeatResp.java | 48 +-
.../SubscriptionSessionConnection.java | 32 -
.../base/AbstractSubscriptionConsumer.java | 47 +-
.../base/AbstractSubscriptionProvider.java | 13 +-
.../base/AbstractSubscriptionProviders.java | 6 +-
.../payload/SubscriptionFileHandler.java | 7 -
.../subscription/payload/SubscriptionMessage.java | 29 +-
.../payload/SubscriptionMessageHandler.java | 7 +-
.../payload/SubscriptionSessionDataSet.java | 12 +-
.../SubscriptionSessionDataSetsHandler.java | 56 +-
.../payload/SubscriptionTsFileHandler.java | 14 +-
.../iotdb/session/SessionCacheLeaderTest.java | 2 +-
.../src/assembly/resources/conf/confignode-env.bat | 14 +-
.../src/assembly/resources/conf/confignode-env.sh | 21 +-
.../assembly/resources/sbin/start-confignode.bat | 2 +-
.../iotdb/confignode/manager/ConfigManager.java | 2 +-
.../iotdb/confignode/manager/ProcedureManager.java | 6 +-
.../iotdb/confignode/manager/TTLManager.java | 9 +-
.../region/GreedyCopySetRegionGroupAllocator.java | 356 +++++++++-
.../region/GreedyRegionGroupAllocator.java | 13 +
.../balancer/region/IRegionGroupAllocator.java | 21 +
.../PartiteGraphPlacementRegionGroupAllocator.java | 13 +
.../persistence/schema/ClusterSchemaInfo.java | 15 +-
.../procedure/PartitionTableAutoCleaner.java | 29 +-
.../procedure/env/ConfigNodeProcedureEnv.java | 52 +-
.../procedure/env/RemoveDataNodeHandler.java | 186 ++++++
.../procedure/impl/StateMachineProcedure.java | 11 +-
.../impl/node/RemoveDataNodesProcedure.java | 6 +-
.../subscription/CreateSubscriptionProcedure.java | 21 +-
.../GreedyCopySetRemoveNodeReplicaSelectTest.java | 342 ++++++++++
.../pipe/consensuspipe/ConsensusPipeReceiver.java | 3 +-
.../src/assembly/resources/conf/datanode-env.bat | 14 +-
.../src/assembly/resources/conf/datanode-env.sh | 24 +-
.../src/assembly/resources/sbin/start-datanode.bat | 2 +-
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 32 +-
.../schemaregion/SchemaExecutionVisitor.java | 15 +
.../db/pipe/agent/task/PipeDataNodeTaskAgent.java | 30 +-
.../agent/task/builder/PipeDataNodeBuilder.java | 4 +-
.../task/builder/PipeDataNodeTaskBuilder.java | 51 +-
.../task/execution/PipeSubtaskExecutorManager.java | 7 +
.../subtask/connector/PipeConnectorSubtask.java | 13 +-
.../PipeRealtimePriorityBlockingQueue.java | 26 +-
.../IoTDBDataNodeCacheLeaderClientManager.java | 12 +-
.../batch/PipeTabletEventTsFileBatch.java | 18 +-
.../batch/PipeTransferBatchReqBuilder.java | 41 +-
.../request/PipeTransferSchemaSnapshotSealReq.java | 5 +-
.../connector/protocol/opcda/OpcDaConnector.java | 8 +-
.../connector/protocol/opcua/OpcUaNameSpace.java | 6 +-
.../pipeconsensus/PipeConsensusAsyncConnector.java | 157 +++--
.../pipeconsensus/PipeConsensusSyncConnector.java | 3 -
.../PipeConsensusTransferBatchReqBuilder.java | 19 +-
.../async/IoTDBDataRegionAsyncConnector.java | 31 +-
.../thrift/sync/IoTDBSchemaRegionConnector.java | 8 +-
...ilder.java => PipeTableModelTsFileBuilder.java} | 121 +++-
.../builder/PipeTableModelTsFileBuilderV2.java | 259 ++++++++
.../util/builder/PipeTreeModelTsFileBuilder.java | 68 +-
.../util/builder/PipeTreeModelTsFileBuilderV2.java | 198 ++++++
.../connector/util/builder/PipeTsFileBuilder.java | 24 +-
.../deletion/persist/PageCacheDeletionBuffer.java | 12 +-
.../schema/PipeSchemaRegionSnapshotEvent.java | 4 +-
.../common/tablet/PipeRawTabletInsertionEvent.java | 5 +
.../tablet/parser/TabletInsertionEventParser.java | 19 +-
.../event/common/terminate/PipeTerminateEvent.java | 6 +-
.../scan/TsFileInsertionEventScanParser.java | 5 +-
...ileInsertionEventTableParserTabletIterator.java | 31 +-
...oricalDataRegionTsFileAndDeletionExtractor.java | 30 +-
.../PipeRealtimeDataRegionHybridExtractor.java | 5 +-
.../realtime/assigner/PipeDataRegionAssigner.java | 9 +-
.../schemaregion/SchemaRegionListeningFilter.java | 15 +
.../pipe/processor/twostage/combiner/Combiner.java | 5 +-
.../pipeconsensus/PipeConsensusReceiver.java | 327 +++++-----
.../pipeconsensus/PipeConsensusReceiverAgent.java | 61 +-
.../protocol/thrift/IoTDBDataNodeReceiver.java | 10 +-
.../resource/memory/InsertNodeMemoryEstimator.java | 4 +
.../db/pipe/resource/memory/PipeMemoryManager.java | 54 +-
.../pipe/resource/memory/PipeMemoryWeightUtil.java | 3 +-
.../pipe/resource/tsfile/PipeTsFileResource.java | 14 +-
.../resource/tsfile/PipeTsFileResourceManager.java | 2 +-
.../v1/handler/StatementConstructionHandler.java | 4 +-
.../execution/executor/RegionReadExecutor.java | 17 +-
.../relational/LastQueryAggTableScanOperator.java | 10 +-
.../relational/ColumnTransformerBuilder.java | 2 +-
.../iotdb/db/queryengine/plan/ClusterTopology.java | 4 +
.../analyze/cache/schema/DataNodeTTLCache.java | 2 +-
.../plan/analyze/load/LoadTsFileAnalyzer.java | 133 ++--
.../analyze/load/LoadTsFileTableSchemaCache.java | 78 ++-
.../config/executor/ClusterConfigTaskExecutor.java | 4 +-
.../plan/parser/StatementGenerator.java | 2 +-
.../plan/planner/TableOperatorGenerator.java | 2 +
.../plan/parameter/AggregationDescriptor.java | 2 +-
.../plan/relational/metadata/TableSchema.java | 2 +-
.../distribute/TableDistributedPlanGenerator.java | 42 +-
.../TransformAggregationToStreamable.java | 33 +-
.../plan/relational/sql/ast/Expression.java | 3 +
.../plan/relational/sql/ast/LoadTsFile.java | 10 +-
.../queryengine/plan/relational/sql/ast/Row.java | 33 +
.../relational/sql/ast/TableExpressionType.java | 3 +-
.../dag/column/leaf/ConstantColumnTransformer.java | 2 +-
.../db/schemaengine/table/DataNodeTableCache.java | 18 +-
.../iotdb/db/service/DataNodeShutdownHook.java | 4 +
.../iotdb/db/service/metrics/WritingMetrics.java | 38 +-
.../metrics/memory/StorageEngineMemoryMetrics.java | 126 ++++
.../iotdb/db/storageengine/StorageEngine.java | 1 +
.../db/storageengine/dataregion/DataRegion.java | 15 +-
.../execute/utils/CompactionTableSchema.java | 2 +-
.../compaction/io/CompactionTsFileWriter.java | 2 +-
.../compaction/schedule/CompactionScheduler.java | 16 +-
.../selector/impl/SettleSelectorImpl.java | 103 ++-
.../dataregion/flush/MemTableFlushTask.java | 12 +-
.../memtable/AlignedReadOnlyMemChunk.java | 6 +-
.../memtable/AlignedWritableMemChunk.java | 52 +-
.../dataregion/memtable/ReadOnlyMemChunk.java | 19 +-
.../dataregion/memtable/TsFileProcessor.java | 38 +-
.../dataregion/memtable/WritableMemChunk.java | 19 +-
.../dataregion/wal/buffer/WALBuffer.java | 9 +-
.../dataregion/wal/io/WALInputStream.java | 2 +
.../wal/utils/MemoryControlledWALEntryQueue.java | 14 +
.../load/metrics/LoadTsFileCostMetricsSet.java | 14 +-
.../rescon/memory/PrimitiveArrayManager.java | 7 +
.../agent/SubscriptionBrokerAgent.java | 9 +
.../db/subscription/broker/SubscriptionBroker.java | 10 +
.../broker/SubscriptionPrefetchingQueue.java | 21 +
.../broker/SubscriptionPrefetchingTabletQueue.java | 10 +
.../broker/SubscriptionPrefetchingTsFileQueue.java | 13 +
.../db/subscription/event/SubscriptionEvent.java | 4 +-
.../batch/SubscriptionPipeTabletEventBatch.java | 27 +-
.../batch/SubscriptionPipeTsFileEventBatch.java | 5 +-
.../response/SubscriptionEventTabletResponse.java | 20 +-
.../response/SubscriptionEventTsFileResponse.java | 9 +-
.../receiver/SubscriptionReceiverV1.java | 76 ++-
.../apache/iotdb/db/utils/ModificationUtils.java | 23 +-
.../db/utils/datastructure/AlignedTVList.java | 40 +-
.../db/utils/datastructure/BatchEncodeInfo.java | 16 +-
.../datastructure/MemPointIteratorFactory.java | 194 ++++--
.../MergeSortMultiAlignedTVListIterator.java | 27 +-
.../MergeSortMultiTVListIterator.java | 15 +-
.../datastructure/MultiAlignedTVListIterator.java | 14 +-
.../utils/datastructure/MultiTVListIterator.java | 13 +-
.../OrderedMultiAlignedTVListIterator.java | 6 +-
.../datastructure/OrderedMultiTVListIterator.java | 5 +-
.../iotdb/db/utils/datastructure/TVList.java | 63 +-
.../pipe/connector/PipeTabletEventSorterTest.java | 21 +-
.../plan/parser/StatementGeneratorTest.java | 9 +-
.../compaction/AbstractCompactionTest.java | 4 +-
.../settle/SettleCompactionSelectorTest.java | 20 +-
.../CompactionTableModelTestFileWriter.java | 2 +-
.../CompactionTableSchemaCollectorTest.java | 2 +-
.../reader/chunk/MemAlignedChunkLoaderTest.java | 5 +-
.../read/reader/chunk/MemChunkLoaderTest.java | 19 +-
.../wal/compression/WALCompressionTest.java | 45 +-
.../src/assembly/resources/sbin/health_check.bat | 33 +-
.../apache/iotdb/commons/conf/CommonConfig.java | 445 +++++++++++++
.../iotdb/commons/conf/CommonDescriptor.java | 486 +-------------
.../task/connection/BlockingPendingQueue.java | 13 +-
.../agent/task/execution/PipeSubtaskExecutor.java | 2 +-
.../agent/task/execution/PipeSubtaskScheduler.java | 15 +-
.../agent/task/progress/PipeEventCommitter.java | 28 +-
.../iotdb/commons/pipe/config/PipeConfig.java | 5 +-
.../config/PipeDescriptor.java} | 726 +++++----------------
.../config/constant/PipeConnectorConstant.java | 3 +
.../pipe/connector/protocol/IoTDBConnector.java | 10 +
.../pipe/datastructure/pattern/TablePattern.java | 2 +
.../commons/pipe/receiver/IoTDBFileReceiver.java | 18 +-
.../apache/iotdb/commons/schema/table/TsTable.java | 15 +-
.../schema/table/column/TsTableColumnCategory.java | 2 +-
.../iotdb/commons/service/metric/enums/Metric.java | 4 +
.../meta/consumer/ConsumerGroupMeta.java | 21 +
.../subscription/meta/consumer/ConsumerMeta.java | 12 +
.../commons/subscription/meta/topic/TopicMeta.java | 9 +-
.../org/apache/iotdb/commons/utils/FileUtils.java | 114 +++-
.../iotdb/commons/utils/TimePartitionUtils.java | 13 -
iotdb-doap.rdf | 16 +
pom.xml | 6 +-
222 files changed, 5386 insertions(+), 2582 deletions(-)
copy
integration-test/src/test/java/org/apache/iotdb/confignode/it/partition/{IoTDBPartitionTableAutoCleanIT.java
=> IoTDBPartitionTableAutoCleanUSIT.java} (87%)
create mode 100644
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/IoTDBComplexQueryIT.java
create mode 100644
integration-test/src/test/java/org/apache/iotdb/subscription/it/local/IoTDBSubscriptionPermissionIT.java
copy
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/node/utils/MNodeFactory.java
=>
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/subscription/annotation/TableModel.java
(78%)
create mode 100644
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/load/balancer/region/GreedyCopySetRemoveNodeReplicaSelectTest.java
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/util/builder/{PipeTableModeTsFileBuilder.java
=> PipeTableModelTsFileBuilder.java} (64%)
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/util/builder/PipeTableModelTsFileBuilderV2.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/util/builder/PipeTreeModelTsFileBuilderV2.java
copy
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/{conf/CommonDescriptor.java
=> pipe/config/PipeDescriptor.java} (55%)
