Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
DanielWang2035 commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568333531
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/event/EnrichedEvent.java:
##
@@ -284,6 +289,21 @@ public void onCommitted() {
}
}
+ /** Used for pipeConsensus */
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+EnrichedEvent otherEvent = (EnrichedEvent) o;
+return committerKey.equals(otherEvent.committerKey)
Review Comment:
Maybe use Objects.equals(committerKey, otherEvent.committerKey)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
DanielWang2035 commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568335911
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.connector.protocol.pipeconsensus;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import
org.apache.iotdb.db.pipe.connector.protocol.thrift.async.IoTDBDataRegionAsyncConnector;
+import org.apache.iotdb.pipe.api.event.Event;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TabletInsertionEvent;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TsFileInsertionEvent;
+import org.apache.iotdb.pipe.api.exception.PipeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.TimeUnit;
+
+public class PipeConsensusAsyncConnector extends IoTDBDataRegionAsyncConnector
{
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeConsensusAsyncConnector.class);
+
+ private static final String ENQUEUE_EXCEPTION_MSG =
+ "Timeout: PipeConsensusConnector offers an event into transferBuffer
failed, because transferBuffer is full";
+
+ private static final CommonConfig COMMON_CONFIG =
CommonDescriptor.getInstance().getConfig();
+
+ private final BlockingQueue transferBuffer =
+ new
LinkedBlockingDeque<>(COMMON_CONFIG.getPipeConsensusEventBufferSize());
+
+ /** Add an event to transferBuffer, whose events will be asynchronizedly
transfer to receiver. */
+ private boolean addEvent2Buffer(Event event) {
Review Comment:
Do we need to increase the reference count of the event when adding it to
buffer? And also decrease reference count when removing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Pipe: Fixed the CPU consuming problem when configNode has nothing to sync [iotdb]
SteveYurongSu merged PR #12359: URL: https://github.com/apache/iotdb/pull/12359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] iotdb 压缩线程处理异常之后，会无限处理错误数据。手动删除对应的序列也没用 [iotdb]
lintao666 opened a new issue, #12360: URL: https://github.com/apache/iotdb/issues/12360 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar. ### Version iotdb:1.3.0 ### Describe the bug and provide the minimal reproduce step 1. 发现iotdb错误日志中有一些时间序列异常`org.apache.iotdb.commons.exception.IllegalPathException` 2. 删除对应的序列之后，也没有重新写入数据。还是一直报之前的错误。 3. 重启服务之后，还是一直报相同的错误。 错误日志节选 [error.log](https://github.com/apache/iotdb/files/15007515/error.log) ### What did you expect to see? 删除对应的时间序列之后，这些错误数据应该同时被删除。 ### What did you see instead? 仔细观察日志后发现压缩线程池一直在处理相同的错误数据 ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] iotdb 压缩线程处理异常之后，会无限处理错误数据。手动删除对应的序列也没用 [iotdb]
github-actions[bot] commented on issue #12360: URL: https://github.com/apache/iotdb/issues/12360#issuecomment-2060598821 Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix convertMillisecondToDurationStr and add UT [iotdb]
liyuheng5 opened a new pull request, #12361: URL: https://github.com/apache/iotdb/pull/12361 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Pipe: Added debug logger after schema/config event is successfully transferred [iotdb]
Caideyipi opened a new pull request, #12362: URL: https://github.com/apache/iotdb/pull/12362 ## Description As the title said. This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix sonar analysis warning [iotdb]
HTHou opened a new pull request, #12363: URL: https://github.com/apache/iotdb/pull/12363 ## Description https://github.com/apache/iotdb/assets/25913899/c91180c6-a7f2-47f5-984a-da61322f2a0a";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Region status Adding and Removing [iotdb]
CRZbulabula commented on code in PR #12342: URL: https://github.com/apache/iotdb/pull/12342#discussion_r1568441311 ## iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/AddRegionPeerProcedure.java: ## @@ -92,6 +94,7 @@ protected Flow executeFromState(ConfigNodeProcedureEnv env, AddRegionPeerState s setNextState(AddRegionPeerState.DO_ADD_REGION_PEER); break; case DO_ADD_REGION_PEER: + handler.updateRegionCache(consensusGroupId, destDataNode, RegionStatus.Adding); Review Comment: Why should update RegionStatus to Adding twice? ## iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/RemoveRegionPeerProcedure.java: ## @@ -100,6 +102,7 @@ protected Flow executeFromState(ConfigNodeProcedureEnv env, RemoveRegionPeerStat setNextState(DELETE_OLD_REGION_PEER); break; case DELETE_OLD_REGION_PEER: + handler.updateRegionCache(consensusGroupId, targetDataNode, RegionStatus.Removing); Review Comment: Why should update RegionStatus to Removing twice? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix sonar analysis warning [iotdb]
sonarcloud[bot] commented on PR #12363: URL: https://github.com/apache/iotdb/pull/12363#issuecomment-2060710912 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12363) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=12363&resolved=false&inNewCodePeriod=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12363&metric=new_accepted_issues&view=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=12363&resolved=false&inNewCodePeriod=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12363) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Region status Adding and Removing [iotdb]
liyuheng5 commented on code in PR #12342: URL: https://github.com/apache/iotdb/pull/12342#discussion_r1568447182 ## iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/AddRegionPeerProcedure.java: ## @@ -92,6 +94,7 @@ protected Flow executeFromState(ConfigNodeProcedureEnv env, AddRegionPeerState s setNextState(AddRegionPeerState.DO_ADD_REGION_PEER); break; case DO_ADD_REGION_PEER: + handler.updateRegionCache(consensusGroupId, destDataNode, RegionStatus.Adding); Review Comment: This is for robustness. The procedure may recover to different state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Region status Adding and Removing [iotdb]
liyuheng5 commented on code in PR #12342: URL: https://github.com/apache/iotdb/pull/12342#discussion_r1568447458 ## iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/RemoveRegionPeerProcedure.java: ## @@ -100,6 +102,7 @@ protected Flow executeFromState(ConfigNodeProcedureEnv env, RemoveRegionPeerStat setNextState(DELETE_OLD_REGION_PEER); break; case DELETE_OLD_REGION_PEER: + handler.updateRegionCache(consensusGroupId, targetDataNode, RegionStatus.Removing); Review Comment: same as above -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix sonar analysis warning [iotdb]
HTHou merged PR #12363: URL: https://github.com/apache/iotdb/pull/12363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] IoTConsensus Transit Snapshot Rate Limiter [iotdb]
OneSizeFitsQuorum merged PR #12348: URL: https://github.com/apache/iotdb/pull/12348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix convertMillisecondToDurationStr and add UT [iotdb]
OneSizeFitsQuorum merged PR #12361: URL: https://github.com/apache/iotdb/pull/12361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
Pengzna commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568517420
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.connector.protocol.pipeconsensus;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import
org.apache.iotdb.db.pipe.connector.protocol.thrift.async.IoTDBDataRegionAsyncConnector;
+import org.apache.iotdb.pipe.api.event.Event;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TabletInsertionEvent;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TsFileInsertionEvent;
+import org.apache.iotdb.pipe.api.exception.PipeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.TimeUnit;
+
+public class PipeConsensusAsyncConnector extends IoTDBDataRegionAsyncConnector
{
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeConsensusAsyncConnector.class);
+
+ private static final String ENQUEUE_EXCEPTION_MSG =
+ "Timeout: PipeConsensusConnector offers an event into transferBuffer
failed, because transferBuffer is full";
+
+ private static final CommonConfig COMMON_CONFIG =
CommonDescriptor.getInstance().getConfig();
+
+ private final BlockingQueue transferBuffer =
+ new
LinkedBlockingDeque<>(COMMON_CONFIG.getPipeConsensusEventBufferSize());
+
+ /** Add an event to transferBuffer, whose events will be asynchronizedly
transfer to receiver. */
+ private boolean addEvent2Buffer(Event event) {
Review Comment:
> Do we need to increase the reference count of the event when adding it to
buffer? And also decrease reference count when removing?
@DanielWang2035 In current design, we will increase the reference count when
transfer method and decrease it in callback method of AsyncThrift RPC. but I
think it's better to manage reference count in the way you said, since it's
semantically clearer. I will try to fix it as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
Pengzna commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568157024
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java:
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.connector.protocol.pipeconsensus;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import
org.apache.iotdb.db.pipe.connector.protocol.thrift.async.IoTDBDataRegionAsyncConnector;
+import org.apache.iotdb.pipe.api.event.Event;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TabletInsertionEvent;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TsFileInsertionEvent;
+import org.apache.iotdb.pipe.api.exception.PipeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.TimeUnit;
+
+public class PipeConsensusAsyncConnector extends IoTDBDataRegionAsyncConnector
{
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeConsensusAsyncConnector.class);
+
+ private static final String ENQUEUE_EXCEPTION_MSG =
+ "Timeout: PipeConsensusConnector offers an event into transferBuffer
failed, because transferBuffer is full";
+
+ private static final CommonConfig COMMON_CONFIG =
CommonDescriptor.getInstance().getConfig();
+
+ private final BlockingQueue transferBuffer =
+ new
LinkedBlockingDeque<>(COMMON_CONFIG.getPipeConsensusEventBufferSize());
+
+ /** Add an event to transferBuffer, whose events will be asynchronizedly
transfer to receiver. */
+ private boolean addEvent2Buffer(Event event) {
+try {
+ LOGGER.debug(
Review Comment:
> It's better to print the debug message after `LOGGER.isDebugEnabled()`.
Since the `transferBuffer.size()` and
`COMMON_CONFIG.getPipeConsensusEventBufferSize()` will be computed and
concatenated to the log string otherwise. See
https://stackoverflow.com/questions/963492/in-log4j-does-checking-isdebugenabled-before-logging-improve-performance.
nice catch, fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
Pengzna commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568517420
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.connector.protocol.pipeconsensus;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import
org.apache.iotdb.db.pipe.connector.protocol.thrift.async.IoTDBDataRegionAsyncConnector;
+import org.apache.iotdb.pipe.api.event.Event;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TabletInsertionEvent;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TsFileInsertionEvent;
+import org.apache.iotdb.pipe.api.exception.PipeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.TimeUnit;
+
+public class PipeConsensusAsyncConnector extends IoTDBDataRegionAsyncConnector
{
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeConsensusAsyncConnector.class);
+
+ private static final String ENQUEUE_EXCEPTION_MSG =
+ "Timeout: PipeConsensusConnector offers an event into transferBuffer
failed, because transferBuffer is full";
+
+ private static final CommonConfig COMMON_CONFIG =
CommonDescriptor.getInstance().getConfig();
+
+ private final BlockingQueue transferBuffer =
+ new
LinkedBlockingDeque<>(COMMON_CONFIG.getPipeConsensusEventBufferSize());
+
+ /** Add an event to transferBuffer, whose events will be asynchronizedly
transfer to receiver. */
+ private boolean addEvent2Buffer(Event event) {
Review Comment:
> Do we need to increase the reference count of the event when adding it to
buffer? And also decrease reference count when removing?
@DanielWang2035 In current design, we will increase the reference count when
transfer method and decrease it in callback method of AsyncThrift RPC. but I
think it's better to manage reference count in the way you said, since it's
semantically clearer. I will try to fix it as you suggested~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Optimize priority map lock structure [iotdb]
OneSizeFitsQuorum merged PR #12351: URL: https://github.com/apache/iotdb/pull/12351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
Pengzna commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568516568
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.connector.protocol.pipeconsensus;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import
org.apache.iotdb.db.pipe.connector.protocol.thrift.async.IoTDBDataRegionAsyncConnector;
+import org.apache.iotdb.pipe.api.event.Event;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TabletInsertionEvent;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TsFileInsertionEvent;
+import org.apache.iotdb.pipe.api.exception.PipeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.TimeUnit;
+
+public class PipeConsensusAsyncConnector extends IoTDBDataRegionAsyncConnector
{
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeConsensusAsyncConnector.class);
+
+ private static final String ENQUEUE_EXCEPTION_MSG =
+ "Timeout: PipeConsensusConnector offers an event into transferBuffer
failed, because transferBuffer is full";
+
+ private static final CommonConfig COMMON_CONFIG =
CommonDescriptor.getInstance().getConfig();
+
+ private final BlockingQueue transferBuffer =
+ new
LinkedBlockingDeque<>(COMMON_CONFIG.getPipeConsensusEventBufferSize());
+
+ /** Add an event to transferBuffer, whose events will be asynchronizedly
transfer to receiver. */
+ private boolean addEvent2Buffer(Event event) {
Review Comment:
In current design, we will increase the reference count when transfer method
and decrease it in callback method of AsyncThrift RPC. but I think it's better
to manage reference count in the way you said, since it's semantically clearer.
I will try to fix it as you suggested~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
Pengzna commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568514855
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/event/EnrichedEvent.java:
##
@@ -284,6 +289,21 @@ public void onCommitted() {
}
}
+ /** Used for pipeConsensus */
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+EnrichedEvent otherEvent = (EnrichedEvent) o;
+return committerKey.equals(otherEvent.committerKey)
Review Comment:
Nice catch~ fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] PipeConsensus: initialize thrift service & connector and receiver logic [iotdb]
Pengzna commented on code in PR #12355:
URL: https://github.com/apache/iotdb/pull/12355#discussion_r1568516568
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/pipeconsensus/PipeConsensusAsyncConnector.java:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.connector.protocol.pipeconsensus;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import
org.apache.iotdb.db.pipe.connector.protocol.thrift.async.IoTDBDataRegionAsyncConnector;
+import org.apache.iotdb.pipe.api.event.Event;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TabletInsertionEvent;
+import org.apache.iotdb.pipe.api.event.dml.insertion.TsFileInsertionEvent;
+import org.apache.iotdb.pipe.api.exception.PipeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.TimeUnit;
+
+public class PipeConsensusAsyncConnector extends IoTDBDataRegionAsyncConnector
{
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeConsensusAsyncConnector.class);
+
+ private static final String ENQUEUE_EXCEPTION_MSG =
+ "Timeout: PipeConsensusConnector offers an event into transferBuffer
failed, because transferBuffer is full";
+
+ private static final CommonConfig COMMON_CONFIG =
CommonDescriptor.getInstance().getConfig();
+
+ private final BlockingQueue transferBuffer =
+ new
LinkedBlockingDeque<>(COMMON_CONFIG.getPipeConsensusEventBufferSize());
+
+ /** Add an event to transferBuffer, whose events will be asynchronizedly
transfer to receiver. */
+ private boolean addEvent2Buffer(Event event) {
Review Comment:
In current design, we will increase the reference count when transfer method
and decrease it in callback method of AsyncThrift RPC. but I think it's better
to manage reference count in the way you said, since it's semantically clearer.
I will try to fix it as you suggested~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Fix compaction worker stopped after drop database [iotdb]
OneSizeFitsQuorum commented on code in PR #12357:
URL: https://github.com/apache/iotdb/pull/12357#discussion_r1568763061
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/schedule/CompactionWorker.java:
##
@@ -50,14 +50,22 @@ public CompactionWorker(
@SuppressWarnings("squid:S2142")
@Override
public void run() {
-while (!Thread.currentThread().isInterrupted()) {
+while (true) {
+ if (Thread.currentThread().isInterrupted()) {
+// If the interrupt is caused by `drop database`, clear the status
+if (!CompactionTaskManager.getInstance().isStopAllCompactionWorker()) {
+ Thread.interrupted();
Review Comment:
What's this as it's result is not used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] feature add iotdb backup [iotdb]
CloudWise-Lukemiao opened a new pull request, #12365: URL: https://github.com/apache/iotdb/pull/12365 feature add iotdb backup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix compaction worker stopped after drop database [iotdb]
OneSizeFitsQuorum merged PR #12357: URL: https://github.com/apache/iotdb/pull/12357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix error when the wal dir is not exist [iotdb]
OneSizeFitsQuorum merged PR #12339: URL: https://github.com/apache/iotdb/pull/12339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Region status Adding and Removing [iotdb]
OneSizeFitsQuorum merged PR #12342: URL: https://github.com/apache/iotdb/pull/12342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] iotdb 压缩线程处理异常之后，会无限处理错误数据。手动删除对应的序列也没用 [iotdb]
HTHou commented on issue #12360: URL: https://github.com/apache/iotdb/issues/12360#issuecomment-2062858071 这个问题已经在 1.3.1 版本中修复了，可以编译 rc/1.3.1 分支的代码进行一下版本升级，或者等待 1.3.1 正式版本发布。 https://github.com/apache/iotdb/pull/11579 https://github.com/apache/iotdb/pull/11547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] How To backup/restore cluster [iotdb]
HTHou commented on issue #12354: URL: https://github.com/apache/iotdb/issues/12354#issuecomment-2062863799 We are developing a tool to support it. https://github.com/apache/iotdb/pull/12365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] How To backup/restore cluster [iotdb]
HTHou commented on issue #12354: URL: https://github.com/apache/iotdb/issues/12354#issuecomment-2062864428 We are developing a tool to support it. https://github.com/apache/iotdb/pull/12365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] How To backup/restore cluster [iotdb]
HTHou commented on issue #12354: URL: https://github.com/apache/iotdb/issues/12354#issuecomment-2062866986 We are developing a tool to support it. https://github.com/apache/iotdb/pull/12365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Lazy load chunk and page in fast compaction [iotdb]
OneSizeFitsQuorum closed pull request #11503: Lazy load chunk and page in fast compaction URL: https://github.com/apache/iotdb/pull/11503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Two stage aggregate counter [iotdb]
sonarcloud[bot] commented on PR #12328: URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2062959674 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12328) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12328&metric=new_coverage&view=list) (required ≥ 80%)  [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12328) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12328) ##  Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Two stage aggregate counter [iotdb]
sonarcloud[bot] commented on PR #12328: URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2062979104 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12328) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12328&metric=new_coverage&view=list) (required ≥ 80%)  [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12328) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12328) ##  Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [IOTDB-6322] Let show variables can be executed by anyone [iotdb]
JackieTien97 opened a new pull request, #12367: URL: https://github.com/apache/iotdb/pull/12367 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6322] Let show variables can be executed by anyone [iotdb]
sonarcloud[bot] commented on PR #12367: URL: https://github.com/apache/iotdb/pull/12367#issuecomment-2062987020 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12367) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=12367&resolved=false&inNewCodePeriod=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12367&metric=new_accepted_issues&view=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=12367&resolved=false&inNewCodePeriod=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12367) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6322] Let show variables can be executed by anyone [iotdb]
codecov[bot] commented on PR #12367: URL: https://github.com/apache/iotdb/pull/12367#issuecomment-2063034427 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/12367?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `1 lines` in your changes are missing coverage. Please review. > Project coverage is 46.40%. Comparing base [(`18df9a9`)](https://app.codecov.io/gh/apache/iotdb/commit/18df9a986be234960b12f44dd571620ba30a3978?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`55e2158`)](https://app.codecov.io/gh/apache/iotdb/pull/12367?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). | [Files](https://app.codecov.io/gh/apache/iotdb/pull/12367?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...lan/statement/metadata/ShowVariablesStatement.java](https://app.codecov.io/gh/apache/iotdb/pull/12367?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fstatement%2Fmetadata%2FShowVariablesStatement.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9zdGF0ZW1lbnQvbWV0YWRhdGEvU2hvd1ZhcmlhYmxlc1N0YXRlbWVudC5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/12367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@Coverage Diff@@ ## master #12367 +/- ## = Coverage 46.39% 46.40% Complexity 69 69 = Files 3380 3380 Lines207677 207674-3 Branches 2496624966 = + Hits 9635996375 +16 + Misses 111318 111299 -19 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/12367?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Increase region migration pool size on DataNode [iotdb]
OneSizeFitsQuorum merged PR #12366: URL: https://github.com/apache/iotdb/pull/12366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6322] Let show variables can be executed by anyone [iotdb]
sonarcloud[bot] commented on PR #12367: URL: https://github.com/apache/iotdb/pull/12367#issuecomment-2063101540 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12367) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=12367&resolved=false&inNewCodePeriod=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=12367&metric=new_accepted_issues&view=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=12367&resolved=false&inNewCodePeriod=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=12367) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Fixed the bug that mod file may not close in async transferring [iotdb]
SteveYurongSu merged PR #12347: URL: https://github.com/apache/iotdb/pull/12347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
