This is an automated email from the ASF dual-hosted git repository.
tanxinyu pushed a change to branch wal_compression
in repository https://gitbox.apache.org/repos/asf/iotdb.git
discard 4cf96171810 fix bug
discard 47070d0046b Allow using python 3.6 and enable CI (#12792)
discard 89b10fe1ace Modify ttl (#12811)
discard 96c6bcb2a0c Pipe/PipeConsensus: support progressIndex for
InsertRowsNode (#12806)
discard 08a237e431a use template file start 1c1d (#12808)
discard bd032f781eb Add flush cause metric (#12803)
discard b91bc23a752 Specify error msg when DriverTask is aborted by
MemoryNotEnoughException
discard 4420e4505c0 Pipe: Added collect invocation count in the caculation of
data region extractor events for data node remaining time (#12799)
discard bbfcbb437f9 Pipe: Fix remaining time metrics is not dropped after
dropPipe(pipeName) & alter pipe does not take effect in non-data regions
(#12795)
discard dcd5780f75c PipeConsensus: fix concurrent problems regarding to
initiate receiver and progressIndex & optimize receiver logs & fix npe (#12790)
discard a9b946448c1 enhance test
discard 02087197a38 [IOTDB-6343] Fix the device path construction bug in
visitSingleDeviceViewNode of AggragationPushDown
discard 336553a9168 Pipe: Enable `'sink.format'='tsfile'` to use tsFile as
tablet event batch (#12737)
discard 08109f16a80 Fix WAL resource leak when using IoT series consensus
#12796
discard 8e7a8442562 Try to fix set cluster name integration test (#12794)
discard b6341e0e335 Fix iotdb-system.properties cluster_name comments #12791
discard a62b5c16bdd Use milliseconds to express TTL (#12757)
discard a7df9aac243 Enhance leader quantity metric for load scenario (#12785)
discard 5bfebdae13d Load: change the metric type of LoadDiskIO metric from
RATE to COUNTER (#12787)
discard f6969721d65 remove direct buffer
discard 095c881c46d bind cluster name to metrics (#12786)
discard 88794504579 Classification properties (#12764)
discard 2fbc789c9c8 Add memory control for MergeReader
discard b816f7a81d6 update tsfile version
discard de5ee16c0ff load-tsfile: Add retry logic for load memory controls
(#12782)
discard 6879ffab28d remove wal files when drop database (#12778)
discard cef8d4e7f80 Support editing cluster name (#12776)
discard e87c4084504 Load: Avoid potential NPE in warn log when
LoadRuntimeOutOfMemoryException occurs & load-tsfile: Polish logs (#12777)
discard 68b3fd0ff35 Update tsfile version (#12774)
discard e06cb7180da Load: Add some load metrics of time cost, write point and
disk throughput (#12735)
discard beb4a9cf80b Pipe: Avoid event being continuously rate-limited even its
pipe is dropped (#12753)
discard bd76aeaccac Pipe Consensus: Fix events in connector are not in order
because of parameter `'realtime-first' = 'true'` (#12773)
discard b6f9733f6d6 SystemPropertiesHandler close reader and writer #12772
discard a4a457ce5e1 Synchronize takeSnapshotAsync on RaftGroupID (#12767)
discard 6834c40cb98 Pipe: Adjusted some loggers of metrics to avoid
unnecessary warns & Include "lastEvent" into pipe's event count metrics &
Removed the "userConflict" judgment to data sync failure caused by
METADATA_ERROR (#12758)
discard d81cb80b210 Pipe: Filter devices by pattern before reading device
metadata from TsFile (#12765)
discard 0e03d697f51 SystemPropertiesHandler use Files::move instead of
File::renameTo (#12770)
discard cf22dd7e3b9 Add clean logic for FragmentInstance in case that callback
is not added. (#12768)
discard b96dd0ee438 Subscription: support topic loose range for path and time
(#12760)
discard 32f98c146dd Fix overlap between tsfile is not correctly marked (#12748)
discard f1d8e8a2777 Refactor node registration check (#12766)
discard b82800b4464 [IOTDB-6342] Clear
FragmentInstanceManager.dataNodeQueryContextMap while encountering
MemoryNotEnough Exception (#12762)
discard 5088a4c6796 [IOTDB-6340] Clear Coordinator.queryExecutionMap while
encountering MemoryNotEnough Exception
discard b1309ce6ca4 Optimize node startup process & Normalize
system.properties management (#12752)
add 16bbc224679 remove direct buffer
add 8008fff7ab6 enhance test
add 94f074f61e2 fix bug
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4cf96171810)
\
N -- N -- N refs/heads/wal_compression (94f074f61e2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/multi-language-client.yml | 15 +-
distribution/src/assembly/all.xml | 9 +
distribution/src/assembly/confignode.xml | 9 +
distribution/src/assembly/datanode.xml | 9 +
.../iotdb/it/env/cluster/ClusterConstant.java | 3 +
.../iotdb/it/env/cluster/env/AbstractEnv.java | 13 +-
.../it/env/cluster/node/ConfigNodeWrapper.java | 4 +-
.../iotdb/it/env/cluster/node/DataNodeWrapper.java | 4 +-
.../iotdb/it/env/remote/env/RemoteServerEnv.java | 6 -
.../java/org/apache/iotdb/itbase/env/BaseEnv.java | 3 -
.../org/apache/iotdb/db/it/IoTDBRecoverIT.java | 2 +-
.../org/apache/iotdb/db/it/IoTDBRestServiceIT.java | 15 +-
.../iotdb/db/it/IoTDBSetConfigurationIT.java | 90 +-
...oTDBAlignByDeviceWithTemplateAggregationIT.java | 29 -
.../iotdb/db/it/auth/IoTDBSeriesPermissionIT.java | 2 +-
.../org/apache/iotdb/db/it/utils/TestUtils.java | 36 -
.../pipe/it/autocreate/IoTDBPipeDataSinkIT.java | 91 +-
.../pipe/it/autocreate/IoTDBPipeExtractorIT.java | 17 +-
.../pipe/it/manual/IoTDBPipeMetaHistoricalIT.java | 6 +-
.../it/IoTDBSubscriptionITConstant.java | 2 +-
.../it/dual/IoTDBSubscriptionTopicIT.java | 135 --
.../java/org/apache/iotdb/tool/ImportTsFile.java | 31 +-
.../client-py/iotdb/utils/IoTDBRpcDataSet.py | 2 +-
iotdb-client/client-py/requirements.txt | 4 +-
iotdb-client/client-py/requirements_dev.txt | 8 +-
iotdb-client/client-py/resources/setup.py | 6 +-
iotdb-client/client-py/tests/test_todf.py | 6 +-
.../iotdb/rpc/subscription/config/TopicConfig.java | 34 +-
.../rpc/subscription/config/TopicConstant.java | 6 -
.../org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4 | 2 +-
.../handlers/ConfigNodeAsyncRequestContext.java | 6 -
.../iotdb/confignode/conf/ConfigNodeConfig.java | 12 +
.../confignode/conf/ConfigNodeDescriptor.java | 11 +-
.../confignode/conf/ConfigNodeStartupCheck.java | 3 +-
.../conf/ConfigNodeSystemPropertiesHandler.java | 54 -
.../confignode/conf/SystemPropertiesUtils.java | 163 +-
.../response/datanode/DataNodeRegisterResp.java | 6 +-
.../iotdb/confignode/manager/ConfigManager.java | 26 +-
.../apache/iotdb/confignode/manager/IManager.java | 2 -
.../iotdb/confignode/manager/TTLManager.java | 12 +
.../manager/node/ClusterNodeStartUtils.java | 133 +-
.../iotdb/confignode/manager/node/NodeManager.java | 26 +-
.../protocol/IoTDBConfigRegionAirGapConnector.java | 7 +-
.../protocol/IoTDBConfigRegionConnector.java | 16 +-
.../metric/PipeConfigNodeRemainingTimeMetrics.java | 10 +-
.../metric/PipeConfigRegionConnectorMetrics.java | 2 +-
.../manager/schema/ClusterSchemaManager.java | 5 +
.../impl/pipe/task/AlterPipeProcedureV2.java | 44 +-
.../thrift/ConfigNodeRPCServiceProcessor.java | 5 -
.../iotdb/confignode/persistence/NodeInfoTest.java | 3 +-
.../org/apache/iotdb/consensus/IConsensus.java | 8 -
.../apache/iotdb/consensus/iot/IoTConsensus.java | 6 -
.../apache/iotdb/consensus/pipe/PipeConsensus.java | 6 -
.../pipe/consensuspipe/ConsensusPipeManager.java | 2 -
.../iotdb/consensus/ratis/RatisConsensus.java | 32 +-
.../iotdb/consensus/simple/SimpleConsensus.java | 5 -
.../apache/iotdb/consensus/iot/StabilityTest.java | 4 -
.../iotdb/consensus/ratis/RatisConsensusTest.java | 7 -
.../consensus/simple/SimpleConsensusTest.java | 6 +-
iotdb-core/datanode/pom.xml | 4 +-
.../db/conf/DataNodeSystemPropertiesHandler.java | 54 -
.../java/org/apache/iotdb/db/conf/IoTDBConfig.java | 14 +-
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 6 +-
.../org/apache/iotdb/db/conf/IoTDBStartCheck.java | 181 +-
.../db/consensus/DataRegionConsensusImpl.java | 9 +-
.../schemaregion/SchemaExecutionVisitor.java | 4 +-
.../schemaregion/SchemaRegionStateMachine.java | 16 +-
.../{PipeDataNodeAgent.java => PipeAgent.java} | 20 +-
.../db/pipe/agent/runtime/PipeAgentLauncher.java | 6 +-
.../agent/runtime/PipeDataNodeRuntimeAgent.java | 10 +-
.../db/pipe/agent/task/PipeDataNodeTaskAgent.java | 23 +-
.../IoTDBDataNodeCacheLeaderClientManager.java | 21 +-
.../evolvable/batch/PipeTabletEventBatch.java | 133 --
.../batch/PipeTabletEventTsFileBatch.java | 365 ----
.../PipeEventBatch.java} | 129 +-
.../PipeTransferBatchReqBuilder.java | 96 +-
.../request/PipeTransferTabletRawReq.java | 4 +-
.../airgap/IoTDBDataNodeAirGapConnector.java | 1 -
.../airgap/IoTDBDataRegionAirGapConnector.java | 16 +-
.../airgap/IoTDBSchemaRegionAirGapConnector.java | 6 +-
.../protocol/legacy/IoTDBLegacyPipeConnector.java | 11 +-
.../pipeconsensus/PipeConsensusAsyncConnector.java | 16 +-
.../PipeConsensusTransferBatchReqBuilder.java | 8 +-
.../async/IoTDBDataRegionAsyncConnector.java | 228 +--
.../PipeTransferTabletBatchEventHandler.java | 40 +-
.../PipeTransferTabletInsertionEventHandler.java | 22 +-
...> PipeTransferTsFileInsertionEventHandler.java} | 130 +-
.../thrift/sync/IoTDBDataNodeSyncConnector.java | 1 -
.../thrift/sync/IoTDBDataRegionSyncConnector.java | 132 +-
.../thrift/sync/IoTDBSchemaRegionConnector.java | 15 +-
.../protocol/writeback/WriteBackConnector.java | 4 +-
.../consensus/ConsensusPipeDataNodeDispatcher.java | 4 +-
.../ConsensusPipeDataNodeRuntimeAgentGuardian.java | 4 +-
.../consensus/ProgressIndexDataNodeManager.java | 24 +-
.../tablet/PipeInsertNodeTabletInsertionEvent.java | 18 +-
.../event/common/terminate/PipeTerminateEvent.java | 4 +-
.../tsfile/TsFileInsertionDataContainer.java | 83 +-
.../realtime/PipeRealtimeDataRegionExtractor.java | 8 +-
.../PipeRealtimeDataRegionHybridExtractor.java | 8 +-
.../PipeRealtimeDataRegionLogExtractor.java | 11 +-
.../PipeRealtimeDataRegionTsFileExtractor.java | 8 +-
.../listener/PipeInsertionDataNodeListener.java | 4 +-
.../schemaregion/IoTDBSchemaRegionExtractor.java | 13 +-
.../PipeDataNodeRemainingEventAndTimeMetrics.java | 25 -
.../PipeDataNodeRemainingEventAndTimeOperator.java | 41 +-
.../metric/PipeDataRegionConnectorMetrics.java | 4 +-
.../metric/PipeDataRegionExtractorMetrics.java | 8 +-
.../iotdb/db/pipe/metric/PipeProcessorMetrics.java | 48 +-
.../metric/PipeSchemaRegionConnectorMetrics.java | 2 +-
.../processor/aggregate/AggregateProcessor.java | 4 +-
.../downsampling/PartialPathLastObjectCache.java | 16 +-
.../twostage/combiner/PipeCombineHandler.java | 4 +-
.../combiner/PipeCombineHandlerManager.java | 4 +-
.../protocol/airgap/IoTDBAirGapReceiver.java | 6 +-
.../pipeconsensus/PipeConsensusReceiver.java | 71 +-
.../pipeconsensus/PipeConsensusReceiverAgent.java | 53 +-
.../protocol/thrift/IoTDBDataNodeReceiver.java | 4 +-
.../visitor/PipeStatementTSStatusVisitor.java | 12 +-
.../db/pipe/resource/memory/PipeMemoryManager.java | 67 +-
...oryWeightUtil.java => PipeMemoryWeighUtil.java} | 63 +-
.../pipe/resource/tsfile/PipeTsFileResource.java | 9 +-
.../resource/tsfile/PipeTsFileResourceManager.java | 4 +-
.../pipe/resource/wal/PipeWALResourceManager.java | 4 +-
.../pipe/task/connection/PipeEventCollector.java | 8 +-
.../db/pipe/task/stage/PipeTaskExtractorStage.java | 6 +-
.../db/pipe/task/stage/PipeTaskProcessorStage.java | 6 +-
.../subtask/connector/PipeConnectorSubtask.java | 24 +-
.../connector/PipeConnectorSubtaskManager.java | 8 +-
.../subtask/processor/PipeProcessorSubtask.java | 19 +-
.../processor/PipeProcessorSubtaskWorker.java | 21 +-
.../iotdb/db/protocol/client/ConfigNodeClient.java | 9 +-
.../iotdb/db/protocol/client/ConfigNodeInfo.java | 66 +-
.../protocol/thrift/impl/ClientRPCServiceImpl.java | 27 +-
.../impl/DataNodeInternalRPCServiceImpl.java | 24 +-
.../thrift/impl/DataNodeRegionManager.java | 4 +-
.../db/queryengine/common/MPPQueryContext.java | 51 +-
.../common/header/ColumnHeaderConstant.java | 4 +-
.../execution/exchange/MPPDataExchangeManager.java | 15 +-
.../fragment/FragmentInstanceContext.java | 31 +-
.../fragment/FragmentInstanceExecution.java | 4 +-
.../fragment/FragmentInstanceManager.java | 10 -
.../execution/load/LoadTsFileManager.java | 40 +-
.../execution/load/LoadTsFileRateLimiter.java | 45 +-
.../queryengine/execution/memory/MemoryPool.java | 9 +-
.../operator/schema/source/DeviceSchemaSource.java | 2 +-
.../AbstractSeriesAggregationScanOperator.java | 2 +-
.../operator/source/AlignedSeriesScanOperator.java | 4 +-
.../operator/source/SeriesScanOperator.java | 3 +-
.../execution/operator/source/SeriesScanUtil.java | 3 +-
.../execution/schedule/AbstractDriverThread.java | 12 +-
.../schedule/DriverTaskAbortedException.java | 3 -
.../load/LoadTsFileAnalyzeSchemaMemoryBlock.java | 2 +-
.../metric/{load => }/LoadTsFileMemMetricSet.java | 2 +-
.../metric/load/LoadTsFileCostMetricsSet.java | 131 --
.../iotdb/db/queryengine/plan/Coordinator.java | 2 +-
.../queryengine/plan/analyze/AnalyzeVisitor.java | 8 -
.../plan/analyze/LoadTsfileAnalyzer.java | 4 +-
.../analyze/cache/schema/DataNodeTTLCache.java | 31 +-
.../queryengine/plan/execution/QueryExecution.java | 3 +-
.../config/executor/ClusterConfigTaskExecutor.java | 17 +-
.../plan/optimization/AggregationPushDown.java | 33 +-
.../plan/planner/LocalExecutionPlanner.java | 13 +-
.../planner/memory/MemoryReservationManager.java | 46 -
.../NotThreadSafeMemoryReservationManager.java | 94 -
.../memory/ThreadSafeMemoryReservationManager.java | 51 -
.../plan/planner/plan/node/PlanGraphPrinter.java | 5 -
.../planner/plan/node/write/InsertRowsNode.java | 11 -
.../scheduler/load/LoadTsFileDispatcherImpl.java | 6 +-
.../plan/scheduler/load/LoadTsFileScheduler.java | 68 +-
.../statement/metadata/ShowDatabaseStatement.java | 37 +-
.../java/org/apache/iotdb/db/service/DataNode.java | 63 +-
.../db/service/metrics/DataNodeMetricsHelper.java | 6 +-
.../iotdb/db/service/metrics/WritingMetrics.java | 186 +-
.../iotdb/db/storageengine/StorageEngine.java | 6 +
.../db/storageengine/dataregion/DataRegion.java | 20 +-
.../CompactionValidationFailedException.java | 14 +-
.../execute/task/AbstractCompactionTask.java | 17 +-
.../writer/RepairUnsortedFileCompactionWriter.java | 18 +-
.../dataregion/memtable/AbstractMemTable.java | 32 +-
.../dataregion/memtable/TsFileProcessor.java | 11 +-
.../read/reader/common/PriorityMergeReader.java | 34 +-
.../storageengine/dataregion/wal/WALManager.java | 7 +-
.../storageengine/dataregion/wal/node/WALNode.java | 2 +-
.../file/AbstractTsFileRecoverPerformer.java | 4 +-
.../file/UnsealedTsFileRecoverPerformer.java | 4 +-
.../dataregion/wal/utils/WALEntryPosition.java | 1 -
.../dataregion/wal/utils/WALInsertNodeCache.java | 43 +-
.../broker/SubscriptionPrefetchingQueue.java | 6 +-
.../SubscriptionPrefetchingTabletsQueue.java | 6 +-
.../event/SubscriptionEventBinaryCache.java | 8 +-
.../SubscriptionConnectorSubtaskManager.java | 4 +-
.../extractor/SchemaRegionListeningQueueTest.java | 22 +-
.../resource/PipeTsFileResourceManagerTest.java | 8 +-
.../execution/operator/OperatorMemoryTest.java | 19 +-
.../plan/planner/node/PlanGraphPrinterTest.java | 2 +-
.../repair/RepairUnsortedFileCompactionTest.java | 25 -
.../iotdb/metrics/core/type/IoTDBCounter.java | 4 +-
.../apache/iotdb/metrics/config/MetricConfig.java | 8 +-
iotdb-core/node-commons/pom.xml | 73 +-
.../resources/conf/generate_properties.bat | 65 +
.../assembly/resources/conf/generate_properties.sh | 59 +
.../resources/conf/iotdb-system.properties | 1859 ++++++++++++++++++-
.../conf/iotdb-system.properties.template | 1878 --------------------
.../apache/iotdb/commons/conf/CommonConfig.java | 11 -
.../iotdb/commons/conf/CommonDescriptor.java | 5 -
.../iotdb/commons/conf/ConfigurationFileUtils.java | 71 +-
.../apache/iotdb/commons/conf/IoTDBConstant.java | 1 +
.../consensus/index/impl/RecoverProgressIndex.java | 5 +-
.../iotdb/commons/file/SystemFileFactory.java | 4 +
.../commons/file/SystemPropertiesHandler.java | 200 ---
.../commons/pipe/agent/task/PipeTaskAgent.java | 9 -
.../iotdb/commons/pipe/config/PipeConfig.java | 7 -
.../config/constant/PipeConnectorConstant.java | 12 +-
.../pipe/connector/limiter/GlobalRateLimiter.java | 43 +-
.../connector/limiter/PipeEndPointRateLimiter.java | 39 +-
.../connector/protocol/IoTDBAirGapConnector.java | 38 +-
.../pipe/connector/protocol/IoTDBConnector.java | 55 +-
.../connector/protocol/IoTDBSslSyncConnector.java | 18 +-
.../queue/ConcurrentIterableLinkedQueue.java | 27 +-
.../iotdb/commons/pipe/event/EnrichedEvent.java | 2 +-
.../commons/pipe/task/meta/PipeTemporaryMeta.java | 8 +-
.../task/subtask/PipeAbstractConnectorSubtask.java | 13 +-
.../apache/iotdb/commons/schema/ttl/TTLCache.java | 18 +-
.../iotdb/commons/service/metric/enums/Metric.java | 14 +-
.../commons/subscription/meta/topic/TopicMeta.java | 2 -
.../iotdb/commons/utils/CommonDateTimeUtils.java | 10 -
.../src/main/thrift/confignode.thrift | 6 +-
pom.xml | 2 +-
228 files changed, 3844 insertions(+), 5898 deletions(-)
delete mode 100644
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/ConfigNodeSystemPropertiesHandler.java
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/DataNodeSystemPropertiesHandler.java
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/{PipeDataNodeAgent.java
=> PipeAgent.java} (81%)
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/batch/PipeTabletEventBatch.java
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/batch/PipeTabletEventTsFileBatch.java
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/{batch/PipeTabletEventPlainBatch.java
=> builder/PipeEventBatch.java} (57%)
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/{batch
=> builder}/PipeTransferBatchReqBuilder.java (55%)
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/async/handler/{PipeTransferTsFileHandler.java
=> PipeTransferTsFileInsertionEventHandler.java} (66%)
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/{PipeMemoryWeightUtil.java
=> PipeMemoryWeighUtil.java} (56%)
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/metric/{load
=> }/LoadTsFileMemMetricSet.java (98%)
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/metric/load/LoadTsFileCostMetricsSet.java
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/memory/MemoryReservationManager.java
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/memory/NotThreadSafeMemoryReservationManager.java
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/memory/ThreadSafeMemoryReservationManager.java
create mode 100644
iotdb-core/node-commons/src/assembly/resources/conf/generate_properties.bat
create mode 100644
iotdb-core/node-commons/src/assembly/resources/conf/generate_properties.sh
delete mode 100644
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template
delete mode 100644
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/file/SystemPropertiesHandler.java
