[GitHub] [iotdb] liyuheng55555 commented on a diff in pull request #10988: [IOTDB-6129] ConfigNode restarts without relying on Seed-ConfigNode
liyuheng5 commented on code in PR #10988:
URL: https://github.com/apache/iotdb/pull/10988#discussion_r1309789997
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterRestartIT.java:
##
@@ -50,15 +51,14 @@
@RunWith(IoTDBTestRunner.class)
@Category({ClusterIT.class})
public class IoTDBClusterRestartIT {
+ private static final Logger logger = IoTDBTestLogger.logger;
private static final String ratisConsensusProtocolClass =
"org.apache.iotdb.consensus.ratis.RatisConsensus";
- private static final int testConfigNodeNum = 2;
- private static final int testDataNodeNum = 2;
+
private static final int testReplicationFactor = 2;
- @Before
- public void setUp() throws Exception {
+ public void setUp(int configNodeNum, int dataNodeNum) {
Review Comment:
Agree, it's ok that all 3 tests use 3c2d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] liyuheng55555 commented on a diff in pull request #10988: [IOTDB-6129] ConfigNode restarts without relying on Seed-ConfigNode
liyuheng5 commented on code in PR #10988:
URL: https://github.com/apache/iotdb/pull/10988#discussion_r1309790204
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterRestartIT.java:
##
@@ -70,7 +70,7 @@ public void setUp() throws Exception {
.setDataReplicationFactor(testReplicationFactor);
// Init 2C2D cluster environment
Review Comment:
Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] liyuheng55555 commented on a diff in pull request #10988: [IOTDB-6129] ConfigNode restarts without relying on Seed-ConfigNode
liyuheng5 commented on code in PR #10988:
URL: https://github.com/apache/iotdb/pull/10988#discussion_r1309794219
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/SystemPropertiesUtils.java:
##
@@ -401,7 +401,7 @@ private static synchronized Properties
getSystemProperties() throws IOException
private static synchronized void storeSystemProperties(Properties
systemProperties)
throws IOException {
try (FileOutputStream fileOutputStream = new
FileOutputStream(systemPropertiesFile)) {
- systemProperties.store(fileOutputStream, "");
+ systemProperties.store(fileOutputStream, " DO NOT MODIFY THIS FILE !!!");
Review Comment:
That's more accurate !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] JackieTien97 commented on a diff in pull request #10939: IOTDB. auth refine.
JackieTien97 commented on code in PR #10939:
URL: https://github.com/apache/iotdb/pull/10939#discussion_r1307423753
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/auth/user/LocalFileUserManager.java:
##
@@ -33,6 +35,9 @@ public LocalFileUserManager(String userDirPath) throws
AuthException {
@Override
public boolean processTakeSnapshot(File snapshotDir) throws TException,
IOException {
+for (Map.Entry entry : userMap.entrySet()) {
+ accessor.saveUser(entry.getValue());
+}
Review Comment:
you can put this call into snapshot take.
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ConfigManager.java:
##
@@ -992,6 +994,46 @@ public TPermissionInfoResp checkUserPrivileges(
}
}
+ public TAuthizedPatternTreeResp fetchAuthizedPatternTree(String username,
int permission) {
+TSStatus status = confirmLeader();
+if (status.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ try {
+return permissionManager.fetchAuthizedPTree(username, permission);
+ } catch (AuthException e) {
+TAuthizedPatternTreeResp resp = new TAuthizedPatternTreeResp();
+status
+.setCode(TSStatusCode.EXECUTE_STATEMENT_ERROR.getStatusCode())
+.setMessage(e.getMessage());
+resp.setStatus(status);
+return resp;
+ }
+} else {
+ TAuthizedPatternTreeResp resp = new TAuthizedPatternTreeResp();
+ resp.setStatus(status);
+ return resp;
+}
+ }
+
+ public TPermissionInfoResp checkUserPrivilegeGrantOpt(
+ String username, PartialPath path, int permission) {
+TSStatus status = confirmLeader();
+TPermissionInfoResp resp = new TPermissionInfoResp();
+if (status.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ try {
+resp = permissionManager.checkUserPrivilegeGrantOpt(username, path,
permission);
+ } catch (AuthException e) {
+status
+.setCode(TSStatusCode.EXECUTE_STATEMENT_ERROR.getStatusCode())
Review Comment:
use the statusCode in AuthException
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/state/AuthOperationProcedureState.java:
##
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.state;
+
+public enum AuthOperationProcedureState {
+ INIT,
+ CONFIGNODE_AUTH_ALTER,
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] JackieTien97 commented on a diff in pull request #10939: IOTDB. auth refine.
JackieTien97 commented on code in PR #10939:
URL: https://github.com/apache/iotdb/pull/10939#discussion_r1309802630
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/auth/user/LocalFileUserAccessor.java:
##
@@ -292,4 +376,9 @@ public void reset() {
public String getDirPath() {
return userDirPath;
}
+
+ @Override
+ public void cleanUserFolder() {
+SystemFileFactory.INSTANCE.getFile(userDirPath).delete();
Review Comment:
```suggestion
FileUtils.cleanDirectory(SystemFileFactory.INSTANCE.getFile(userDirPath));
```
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/auth/entity/Role.java:
##
@@ -36,62 +37,192 @@
public class Role {
private String name;
- private List privilegeList;
+ private List pathPrivilegeList;
+
+ private Set sysPrivilegeSet;
+
+ private Set sysPriGrantOpt;
+
+ private static final int SYS_PRI_SIZE = PrivilegeType.getSysPriCount();
public Role() {
// empty constructor
}
public Role(String name) {
this.name = name;
-this.privilegeList = new ArrayList<>();
+this.pathPrivilegeList = new ArrayList<>();
+this.sysPrivilegeSet = new HashSet<>();
+this.sysPriGrantOpt = new HashSet<>();
}
+ /** - get func -* */
public String getName() {
return name;
}
- public void setName(String name) {
-this.name = name;
+ public List getPathPrivilegeList() {
+return pathPrivilegeList;
}
- public List getPrivilegeList() {
-return privilegeList;
+ public Set getSysPrivilege() {
+return sysPrivilegeSet;
}
- public void setPrivilegeList(List privilegeList) {
-this.privilegeList = privilegeList;
+ public Set getPathPrivileges(PartialPath path) throws AuthException
{
+return AuthUtils.getPrivileges(path, pathPrivilegeList);
}
- public boolean hasPrivilege(PartialPath path, int privilegeId) {
-return AuthUtils.hasPrivilege(path, privilegeId, privilegeList);
+ public Set getSysPriGrantOpt() {
+return sysPriGrantOpt;
}
- public void addPrivilege(PartialPath path, int privilegeId) {
-AuthUtils.addPrivilege(path, privilegeId, privilegeList);
+ public int getAllSysPrivileges() {
+int privs = 0;
+for (Integer sysPri : sysPrivilegeSet) {
+ privs |= (0b1 << sysPriTopos(sysPri));
+}
+for (Integer sysPriGrantOpt : sysPriGrantOpt) {
+ privs |= 0b1 << (sysPriTopos(sysPriGrantOpt) + 16);
Review Comment:
```suggestion
privs |= 1 << (sysPriTopos(sysPriGrantOpt) + 16);
```
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/auth/authorizer/BasicAuthorizer.java:
##
@@ -317,20 +302,109 @@ public boolean checkUserPrivileges(String username,
PartialPath path, int privil
throw new AuthException(
TSStatusCode.USER_NOT_EXIST, String.format(NO_SUCH_USER_EXCEPTION,
username));
}
-// get privileges of the user
-if (user.checkPrivilege(path, privilegeId)) {
- return true;
-}
-// merge the privileges of the roles of the user
-for (String roleName : user.getRoleList()) {
- Role role = roleManager.getRole(roleName);
- if (role.checkPrivilege(path, privilegeId)) {
+if (path != null) {
+ // get privileges of the user
+ if (user.checkPathPrivilege(path, privilegeId)) {
+return true;
+ }
+ // merge the privileges of the roles of the user
+ for (String roleName : user.getRoleList()) {
+Role role = roleManager.getRole(roleName);
+if (role.checkPathPrivilege(path, privilegeId)) {
+ return true;
+}
+ }
+} else {
+ if (user.checkSysPrivilege(privilegeId)) {
return true;
}
+ for (String roleName : user.getRoleList()) {
+Role role = roleManager.getRole(roleName);
+if (role.checkSysPrivilege(privilegeId)) {
+ return true;
+}
+ }
}
+
return false;
}
+ public boolean checkUserPrivilegeGrantOpt(String username, PartialPath path,
int privilegeId)
+ throws AuthException {
+User user = userManager.getUser(username);
+if (user == null) {
+ throw new AuthException(
+ TSStatusCode.USER_NOT_EXIST, String.format(NO_SUCH_USER_EXCEPTION,
username));
+}
+if (path == null) {
+ if (user.checkSysPrivilege(privilegeId)) {
+if (user.getSysPriGrantOpt().contains(privilegeId)) {
+ return true;
+}
+ }
+ if (user.getRoleList().isEmpty()) {
+throw new AuthException(
+TSStatusCode.NOT_HAS_PRIVILEGE,
+String.format(
+"Dont have privilege: %s to grant",
+PrivilegeType.values()[privilegeId].toString()));
+ }
+ for (String roleName : user.getRoleList()) {
+Role role = roleManager.getRole(roleName);
+if (role.checkSysPrivilege(privilegeId) &&
role.getSysPriGrantOpt().contains(
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1309835948
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ModelManager.java:
##
@@ -63,15 +81,41 @@ public ModelInfo getModelInfo() {
}
public TSStatus createModel(TCreateModelReq req) {
-ModelInformation modelInformation =
-new ModelInformation(
-req.getModelId(),
-req.getModelTask(),
-req.getModelType(),
-req.isIsAuto(),
-req.getQueryExpressions(),
-req.getQueryFilter());
-return configManager.getProcedureManager().createModel(modelInformation,
req.getModelConfigs());
+TaskType taskType = req.getTaskType();
+
+Map options = req.getOptions();
+ModelInformation modelInformation;
+if (taskType == TaskType.FORECAST) {
+ String inputTypeListStr = options.get(INPUT_TYPE_LIST);
+ List inputTypeList =
+ Arrays.stream(inputTypeListStr.substring(1,
inputTypeListStr.length() - 1).split(","))
+ .sequential()
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] OneSizeFitsQuorum merged pull request #10996: [To rel/1.2][IOTDB-6119] Add ConfigNode leader service check
OneSizeFitsQuorum merged PR #10996: URL: https://github.com/apache/iotdb/pull/10996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1309938159
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##
@@ -341,6 +355,46 @@ public Analysis visitQuery(QueryStatement queryStatement,
MPPQueryContext contex
return analysis;
}
+ private void analyzeModelInference(Analysis analysis, QueryStatement
queryStatement) {
+FunctionExpression modelInferenceExpression =
+(FunctionExpression)
+
queryStatement.getSelectComponent().getResultColumns().get(0).getExpression();
+String modelId =
modelInferenceExpression.getFunctionAttributes().get(MODEL_ID);
+
+ModelInformation modelInformation =
partitionFetcher.getModelInformation(modelId);
+if (modelInformation == null || !modelInformation.available()) {
+ throw new SemanticException("Model " + modelId + " is not available");
+}
+
+ModelInferenceFunction functionType =
+
ModelInferenceFunction.valueOf(modelInferenceExpression.getFunctionName().toUpperCase());
+if (functionType == FORECAST) {
+ ForecastModelInferenceDescriptor modelInferenceDescriptor =
+ new ForecastModelInferenceDescriptor(
+ functionType, (ForecastModeInformation) modelInformation);
+ Map modelInferenceAttributes =
+ modelInferenceExpression.getFunctionAttributes();
+ if (modelInferenceAttributes.containsKey("predict_length")) {
+modelInferenceDescriptor.setExpectedPredictLength(
+Integer.parseInt(modelInferenceAttributes.get("predict_length")));
+ }
+ analysis.setModelInferenceDescriptor(modelInferenceDescriptor);
+
+ List newResultColumns = new ArrayList<>();
+ for (Expression inputExpression :
modelInferenceExpression.getExpressions()) {
+newResultColumns.add(new ResultColumn(inputExpression));
+ }
+ queryStatement.getSelectComponent().setResultColumns(newResultColumns);
+
+ OrderByComponent descTimeOrder = new OrderByComponent();
+ descTimeOrder.addSortItem(new SortItem("TIME", Ordering.DESC));
+ queryStatement.setOrderByComponent(descTimeOrder);
Review Comment:
Get the latest data for forecasting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] Caideyipi opened a new pull request, #10997: Pipe: Added queue size report to pipe heartbeat
Caideyipi opened a new pull request, #10997: URL: https://github.com/apache/iotdb/pull/10997 This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] JackieTien97 merged pull request #10939: IOTDB. auth refine.
JackieTien97 merged PR #10939: URL: https://github.com/apache/iotdb/pull/10939 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1309980113
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LogicalPlanVisitor.java:
##
@@ -214,6 +218,22 @@ public PlanNode visitQuery(QueryStatement queryStatement,
MPPQueryContext contex
.planOffset(queryStatement.getRowOffset())
.planLimit(queryStatement.getRowLimit());
+if (queryStatement.isModelInferenceQuery()) {
+ ModelInferenceDescriptor modelInferenceDescriptor =
analysis.getModelInferenceDescriptor();
+ if (Objects.requireNonNull(modelInferenceDescriptor.getFunctionType())
+ == ModelInferenceFunction.FORECAST) {
+ForecastModelInferenceDescriptor forecastModelInferenceDescriptor =
+(ForecastModelInferenceDescriptor) modelInferenceDescriptor;
+planBuilder
+.planLimit(forecastModelInferenceDescriptor.getModelInputLength())
Review Comment:
Forecast is a inference method. Model can only receive fixed lines of data
which is determined in training. Limit node is used to make sure the input data
match the model parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1310006197
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/ExpressionTypeAnalyzer.java:
##
@@ -269,6 +269,8 @@ public TSDataType
visitFunctionExpression(FunctionExpression functionExpression,
functionExpression,
TypeInferenceUtils.getBuiltInScalarFunctionDataType(
functionExpression,
expressionTypes.get(NodeRef.of(inputExpressions.get(0);
+ } else if (functionExpression.isModelInferenceFunction()) {
+return setExpressionType(functionExpression, TSDataType.DOUBLE);
Review Comment:
Yes, the result of forecast is from model in MLNode. The output of model is
always DOUBLE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] shuolinl opened a new pull request, #10998: [auth].fix auth resp error.
shuolinl opened a new pull request, #10998: URL: https://github.com/apache/iotdb/pull/10998 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] JackieTien97 merged pull request #10998: [auth].fix auth resp error.
JackieTien97 merged PR #10998: URL: https://github.com/apache/iotdb/pull/10998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] OneSizeFitsQuorum merged pull request #10988: [IOTDB-6129] ConfigNode restarts without relying on Seed-ConfigNode
OneSizeFitsQuorum merged PR #10988: URL: https://github.com/apache/iotdb/pull/10988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] shuwenwei opened a new pull request, #10999: Lazy load chunk in FastCompaction
shuwenwei opened a new pull request, #10999: URL: https://github.com/apache/iotdb/pull/10999 ## Description When an aligned chunk can be directly flushed into target file writer, read the data of chunk from disk only when it is necessary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1310360169
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##
@@ -341,6 +355,46 @@ public Analysis visitQuery(QueryStatement queryStatement,
MPPQueryContext contex
return analysis;
}
+ private void analyzeModelInference(Analysis analysis, QueryStatement
queryStatement) {
+FunctionExpression modelInferenceExpression =
+(FunctionExpression)
+
queryStatement.getSelectComponent().getResultColumns().get(0).getExpression();
+String modelId =
modelInferenceExpression.getFunctionAttributes().get(MODEL_ID);
+
+ModelInformation modelInformation =
partitionFetcher.getModelInformation(modelId);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1310360449
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/ClusterPartitionFetcher.java:
##
@@ -293,6 +296,28 @@ public void invalidAllCache() {
partitionCache.invalidAllCache();
}
+ @Override
+ public ModelInformation getModelInformation(String modelId) {
+try (ConfigNodeClient client =
+configNodeClientManager.borrowClient(ConfigNodeInfo.CONFIG_REGION_ID))
{
+ TGetModelInfoResp getModelInfoResp = client.getModelInfo(new
TGetModelInfoReq(modelId));
+ if (getModelInfoResp.getStatus().getCode() ==
TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+if (getModelInfoResp.modelInfo != null) {
+ return ModelInformation.deserialize(getModelInfoResp.modelInfo);
+} else {
+ return null;
+}
+ } else {
+throw new StatementAnalyzeException(
+"An error occurred when executing getModelInformation():"
++ getModelInfoResp.getStatus().getMessage());
+ }
+} catch (ClientManagerException | TException e) {
+ throw new StatementAnalyzeException(
+ "An error occurred when executing getModelInformation():" +
e.getMessage());
+}
+ }
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507: URL: https://github.com/apache/iotdb/pull/10507#discussion_r1310360841 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/IPartitionFetcher.java: ## @@ -88,4 +89,6 @@ SchemaNodeManagementPartition getSchemaNodeManagementPartitionWithLevel( /** Invalid all partition cache */ void invalidAllCache(); + + ModelInformation getModelInformation(String modelId); Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1310361110
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/config/metadata/model/ShowModelsTask.java:
##
@@ -79,16 +78,15 @@ public static void buildTsBlock(
builder.getColumnBuilder(2).writeBinary(Binary.valueOf(modelType));
builder.getColumnBuilder(3).writeBinary(Binary.valueOf(queryBody));
builder.getColumnBuilder(4).writeBinary(Binary.valueOf(trainingState));
- builder.getColumnBuilder(5).writeBinary(Binary.valueOf(modelPath));
-
builder.getColumnBuilder(6).writeBinary(Binary.valueOf(modelHyperparameter.get(0)));
+
builder.getColumnBuilder(5).writeBinary(Binary.valueOf(modelHyperparameter.get(0)));
builder.declarePosition();
for (int i = 1; i < listSize; i++) {
builder.getTimeColumnBuilder().writeLong(0L);
-for (int columnIndex = 0; columnIndex <= 5; columnIndex++) {
+for (int columnIndex = 0; columnIndex <= 4; columnIndex++) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] ycycse commented on a diff in pull request #10507: [IoTDB ML] Basic model training and inference features available
ycycse commented on code in PR #10507:
URL: https://github.com/apache/iotdb/pull/10507#discussion_r1310361476
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/ml/ForecastOperator.java:
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.ml;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.db.exception.ModelInferenceProcessException;
+import org.apache.iotdb.db.protocol.client.MLNodeClient;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import
org.apache.iotdb.db.queryengine.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.mlnode.rpc.thrift.TForecastResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.block.TsBlock;
+import org.apache.iotdb.tsfile.read.common.block.TsBlockBuilder;
+import org.apache.iotdb.tsfile.read.common.block.column.ColumnBuilder;
+import org.apache.iotdb.tsfile.read.common.block.column.TimeColumnBuilder;
+import org.apache.iotdb.tsfile.read.common.block.column.TsBlockSerde;
+
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+
+import static com.google.common.util.concurrent.Futures.successfulAsList;
+
+public class ForecastOperator implements ProcessOperator {
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(ForecastOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator child;
+
+ private final String modelPath;
+ private final List inputTypeList;
+ private final List inputColumnNameList;
+ private final int expectedPredictLength;
+
+ private final TsBlockBuilder inputTsBlockBuilder;
+
+ private MLNodeClient client;
+ private final ExecutorService modelInferenceExecutor;
+ private ListenableFuture forecastExecutionFuture;
+
+ private boolean finished = false;
+
+ private final long maxRetainedSize;
+ private final long maxReturnSize;
+
+ public ForecastOperator(
+ OperatorContext operatorContext,
+ Operator child,
+ String modelPath,
+ List inputTypeList,
+ List inputColumnNameList,
+ int expectedPredictLength,
+ ExecutorService modelInferenceExecutor,
+ long maxRetainedSize,
+ long maxReturnSize) {
+this.operatorContext = operatorContext;
+this.child = child;
+this.modelPath = modelPath;
+this.inputTypeList = inputTypeList;
+this.inputColumnNameList = inputColumnNameList;
+this.expectedPredictLength = expectedPredictLength;
+this.inputTsBlockBuilder = new TsBlockBuilder(inputTypeList);
+this.modelInferenceExecutor = modelInferenceExecutor;
+this.maxRetainedSize = maxRetainedSize;
+this.maxReturnSize = maxReturnSize;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+return operatorContext;
+ }
+
+ @Override
+ public ListenableFuture isBlocked() {
+ListenableFuture childBlocked = child.isBlocked();
+boolean executionDone = forecastExecutionDone();
+if (executionDone && childBlocked.isDone()) {
+ return NOT_BLOCKED;
+} else if (childBlocked.isDone()) {
+ return forecastExecutionFuture;
+} else if (executionDone) {
+ return childBlocked;
+} else {
+ return successfulAsList(Arrays.asList(forecastExecutionFuture,
childBlocked));
+}
+ }
+
+ private boolean forecastExecutionDone() {
+if (forecastExecutionFuture == null) {
+ return true;
+}
+return forecastExecutionFuture.isDone();
+ }
+
+ @Override
+ public boolean hasNext() throws Exception {
+return !finished;
+ }
+
+ @Override
+ public TsBlock next() throws Exception {
+if (forecastExecutionFuture == null) {
+
[GitHub] [iotdb] SteveYurongSu merged pull request #11001: Pipe: Fix start-time and end-time parameters not working when extracting history data
SteveYurongSu merged PR #11001: URL: https://github.com/apache/iotdb/pull/11001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] SteveYurongSu merged pull request #11002: [To rel/1.2] Pipe: Fix start-time and end-time parameters not working when extracting history data
SteveYurongSu merged PR #11002: URL: https://github.com/apache/iotdb/pull/11002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb-docs] xuanronaldo opened a new pull request, #76: add flink-sql-iotdb-connector docs
xuanronaldo opened a new pull request, #76: URL: https://github.com/apache/iotdb-docs/pull/76 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] Caideyipi opened a new pull request, #11004: Pipe: Improved the pipe performance
Caideyipi opened a new pull request, #11004: URL: https://github.com/apache/iotdb/pull/11004 This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] xingtanzjr merged pull request #10992: [To rel/1.2] Refactoring DeleteOutdatedFileTask in WalNode
xingtanzjr merged PR #10992: URL: https://github.com/apache/iotdb/pull/10992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] xingtanzjr merged pull request #10993: Fixed write rejection when wal size exceeds threshold
xingtanzjr merged PR #10993: URL: https://github.com/apache/iotdb/pull/10993 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] Caideyipi opened a new pull request, #11005: Release pr 1.2
Caideyipi opened a new pull request, #11005: URL: https://github.com/apache/iotdb/pull/11005 This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb-docs] Caideyipi opened a new pull request, #77: Added Flink-Sql-IoTDB-Connector
Caideyipi opened a new pull request, #77: URL: https://github.com/apache/iotdb-docs/pull/77 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] OneSizeFitsQuorum merged pull request #11003: [To rel/1.2][IOTDB-6129] ConfigNode restarts without relying on Seed-ConfigNode
OneSizeFitsQuorum merged PR #11003: URL: https://github.com/apache/iotdb/pull/11003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
