[jira] [Created] (IOTDB-4971) dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
刘珍 created IOTDB-4971:
-
Summary: dispatch write failed. status: TSStatus(code:506,
subStatus:[]), message: null
Key: IOTDB-4971
URL: https://issues.apache.org/jira/browse/IOTDB-4971
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jinrui Zhang
master_1117_d548214
1. start 3rep 3C 9D cluster
2. delete timeseries root.**和create metadata , write data 并发
datanode （IP18） 有ERROR ：
2022-11-17 14:52:38,172
[pool-24-IoTDB-DataNodeInternalRPC-Processor-17$20221117_065237_15126_11.1.0]
{color:red}*ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:173 - dispatch
write failed. status: TSStatus(code:506, subStatus:[]), message: null*{color}
复现流程
1. 启动3C9D集群
3C : 172.20.70.19/172.20.70.21/172.20.70.32
9D : 172.20.70.2/3/4/5/13/14/15/16/18
配置参数
ConfigNode：
MAX_HEAP_SIZE="8G"
MAX_DIRECT_MEMORY_SIZE="6G"
cn_connection_timeout_ms=360
Common ：
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
schema_replication_factor=3
data_replication_factor=3
connection_timeout_ms=360
max_connection_for_internal_service=200
query_timeout_threshold=360
schema_region_ratis_request_timeout_ms=180
Datanode：
MAX_HEAP_SIZE="20G"
MAX_DIRECT_MEMORY_SIZE="6G"
2. 启动测试脚本
down_delete_ts.conf放到${bm_dir}/conf下
del_ts.sh 、run_del_1.sh 、 run_del_2.sh 、run_iotdb_4563.sh 这4个脚本放到${bm_dir}下
启动脚本是：run_iotdb_4563.sh
运行完成，查看 ip18datanode 日志。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-4970) [sync] Enhanced sync safety management
yanze chen created IOTDB-4970: - Summary: [sync] Enhanced sync safety management Key: IOTDB-4970 URL: https://issues.apache.org/jira/browse/IOTDB-4970 Project: Apache IoTDB Issue Type: Improvement Reporter: yanze chen Assignee: yanze chen Before the sync sender can send data, a handshake with the receiver is required. The current handshake policy is to verify the sender's IP address based on the receiver's white list. However, the current sender's ip address is provided by the sender and may be falsified leading to security issues. # This ISSUE will get the ip address from the thrift connection. # The default whitelist is set to 0.0.0.0/32, which rejects all the sender's ip. # Some optimizations have been made to handle cases of handshake failure. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4969) Distribution plan is not correct for Aggregation query with AlignByDevice
Jinrui Zhang created IOTDB-4969: --- Summary: Distribution plan is not correct for Aggregation query with AlignByDevice Key: IOTDB-4969 URL: https://issues.apache.org/jira/browse/IOTDB-4969 Project: Apache IoTDB Issue Type: Bug Reporter: Jinrui Zhang Assignee: Jinrui Zhang Attachments: image-2022-11-17-14-41-37-947.png, image-2022-11-17-14-44-18-254.png If one device's data is distributed in more than one DataRegion. The aggregation query for this device is not correct due to wrong distribution Plan. See the wrong plan below. !image-2022-11-17-14-41-37-947.png|width=615,height=391! This plan will lead to wrong result as follow: !image-2022-11-17-14-44-18-254.png|width=380,height=88! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4855) [MultiLeader] Strength the memory control
[ https://issues.apache.org/jira/browse/IOTDB-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyu Tan reassigned IOTDB-4855: Assignee: Xinyu Tan (was: 张洪胤) > [MultiLeader] Strength the memory control > - > > Key: IOTDB-4855 > URL: https://issues.apache.org/jira/browse/IOTDB-4855 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: 张洪胤 >Assignee: Xinyu Tan >Priority: Major > Labels: pull-request-available > > We need to strength the control of multiLeader memory and taking the size of > syncStatus and pendingBatch that reading from WAL into consideration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4968) [0.14][sync] sender close partition and receiver open partition， will reported error and lost data
[ https://issues.apache.org/jira/browse/IOTDB-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangZhengming reassigned IOTDB-4968: - Assignee: yusicheng > [0.14][sync] sender close partition and receiver open partition， will > reported error and lost data > -- > > Key: IOTDB-4968 > URL: https://issues.apache.org/jira/browse/IOTDB-4968 > Project: Apache IoTDB > Issue Type: Bug >Reporter: ZhangZhengming >Assignee: yusicheng >Priority: Major > Attachments: image-2022-11-17-11-43-59-969.png, > image-2022-11-17-11-45-03-157.png, image-2022-11-17-11-45-29-894.png, > image-2022-11-17-11-49-35-009.png, log-receiver.zip, log-sender.zip > > > The sender without time partition will report an error and lose data when > sending data to the receiver with time partition enabled. > See attachment for log > > !image-2022-11-17-11-43-59-969.png|width=1133,height=298! > error: > !image-2022-11-17-11-45-03-157.png|width=813,height=506! > !image-2022-11-17-11-45-29-894.png|width=602,height=347! > > lost data > !image-2022-11-17-11-49-35-009.png|width=602,height=315! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4968) [0.14][sync] sender close partition and receiver open partition， will reported error and lost data
ZhangZhengming created IOTDB-4968: - Summary: [0.14][sync] sender close partition and receiver open partition， will reported error and lost data Key: IOTDB-4968 URL: https://issues.apache.org/jira/browse/IOTDB-4968 Project: Apache IoTDB Issue Type: Bug Reporter: ZhangZhengming The sender without time partition will report an error and lose data when sending data to the receiver with time partition enabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4967) [UDF] SlidingTimeWindow stops when encounter an empty window
liaolanyu created IOTDB-4967: Summary: [UDF] SlidingTimeWindow stops when encounter an empty window Key: IOTDB-4967 URL: https://issues.apache.org/jira/browse/IOTDB-4967 Project: Apache IoTDB Issue Type: Bug Reporter: liaolanyu Assignee: liaolanyu https://github.com/apache/iotdb/issues/7738 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-2069) cluster : Internal error processing fetchMultSeries
[
https://issues.apache.org/jira/browse/IOTDB-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haonan Hou reopened IOTDB-2069:
---
Assignee: (was: Xinyu Tan)
> cluster : Internal error processing fetchMultSeries
> ---
>
> Key: IOTDB-2069
> URL: https://issues.apache.org/jira/browse/IOTDB-2069
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Cluster
>Affects Versions: 0.12.3
>Reporter: 刘珍
>Priority: Major
> Attachments: config.properties, image-2021-11-27-09-41-21-879.png,
> image-2021-11-27-09-41-28-576.png, iotdb-engine.properties, iotdb-env.sh,
> q.sh, q_limit.sh, run_limit.sh, run_q.sh, screenshot-1.png
>
>
> 0.12.3 rc2 a8c5dc8a13a4b90303fec4b29e40a1b33e6b737d
> 1. iotdb conf
> 3 node 1 rep
> See the attachment
> 2. run benchmark (see the attach : config.properties)
> 3. 192.168.130.13
> nohup ./run_q.sh &
> nohup ./run_limit.sh &
> 4.192.168.130.10
> nohup ./run_q.sh &
> 5. 192.168.130.4
> nohup ./run_q.sh &
> 6. log error:
> !image-2021-11-27-09-41-21-879.png|thumbnail!
> !image-2021-11-27-09-41-28-576.png|thumbnail!
> 2021-11-27 00:11:22,139 [DataClientThread-77] ERROR o.a.t.ProcessFunction:47
> - {color:#DE350B}*Internal error processing fetchMultSeries*{color}
> org.apache.thrift.TException:
> org.apache.iotdb.cluster.exception.ReaderNotFoundException: The requested
> reader 42474 is not found
> at
> org.apache.iotdb.cluster.server.service.DataSyncService.fetchMultSeries(DataSyncService.java:247)
> at
> org.apache.iotdb.cluster.server.DataClusterServer.fetchMultSeries(DataClusterServer.java:797)
> at
> org.apache.iotdb.cluster.rpc.thrift.TSDataService$Processor$fetchMultSeries.getResult(TSDataService.java:2038)
> at
> org.apache.iotdb.cluster.rpc.thrift.TSDataService$Processor$fetchMultSeries.getResult(TSDataService.java:2018)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.iotdb.cluster.exception.ReaderNotFoundException: The
> requested reader 42474 is not found
> at
> org.apache.iotdb.cluster.query.LocalQueryExecutor.fetchMultSeries(LocalQueryExecutor.java:168)
> at
> org.apache.iotdb.cluster.server.service.DataSyncService.fetchMultSeries(DataSyncService.java:245)
> ... 9 common frames omitted
> 2021-11-27 08:41:27,329 [pool-1563-IoTDB-Query-3] ERROR
> o.a.i.c.q.r.m.RemoteMultSeriesReader:192 - Failed to fetch result sync,
> connect to DataSourceInfo{readerId=75827,
> curSource=Node(internalIp:192.168.130.4, metaPort:9003,
> nodeIdentifier:1622500152, dataPort:40010, clientPort:6667,
> clientIp:192.168.130.4), partitionGroup=[Node(internalIp:192.168.130.4,
> metaPort:9003, nodeIdentifier:1622500152, dataPort:40010, clientPort:6667,
> clientIp:192.168.130.4)],
> request=MultSeriesQueryRequest(path:[root.group_29.d_929.s_0,
> root.group_29.d_929.s_199, root.group_29.d_929.s_299,
> root.group_29.d_929.s_399, root.group_29.d_929.s_499,
> root.group_29.d_929.s_99], timeFilterBytes:03 01 00 80 00 00 00 00 00 00 00,
> queryId:152942, requester:Node(internalIp:192.168.130.13, metaPort:9007,
> nodeIdentifier:789322236, dataPort:40010, clientPort:6667,
> clientIp:192.168.130.13), header:Node(internalIp:192.168.130.4,
> metaPort:9003, nodeIdentifier:1622500152, dataPort:40010, clientPort:6667,
> clientIp:192.168.130.4), dataTypeOrdinal:[1, 1, 2, 2, 2, 1],
> deviceMeasurements:{root.group_29.d_929.s_99=[s_0, s_199, s_299, s_399,
> s_499, s_99], root.group_29.d_929.s_0=[s_0, s_199, s_299, s_399, s_499,
> s_99], root.group_29.d_929.s_499=[s_0, s_199, s_299, s_399, s_499, s_99],
> root.group_29.d_929.s_199=[s_0, s_199, s_299, s_399, s_499, s_99],
> root.group_29.d_929.s_399=[s_0, s_199, s_299, s_399, s_499, s_99],
> root.group_29.d_929.s_299=[s_0, s_199, s_299, s_399, s_499, s_99]},
> ascending:true, fetchSize:0, deduplicatedPathNum:0)}
> org.apache.thrift.TApplicationException:{color:#DE350B}* Internal error
> processing fetchMultSeries*{color}
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.iotdb.cluster.rpc.thrift.TSDataService$Client.recv_fetchMultSeries(TSDataService.java:378)
> at
> org.apache.iotdb.cluster.rpc.thrift.TSDataService$Client.fetchMultSeries(TSDataService.java:363)
> at
>
[jira] [Assigned] (IOTDB-4761) [ flush ] ERROR o.a.i.d.m.p.e.c.ConfigExecution:124 - Failures happened during running ConfigExecution
[ https://issues.apache.org/jira/browse/IOTDB-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaofei Cao reassigned IOTDB-4761: - Assignee: Yongzao Dan (was: 陈哲涵) > [ flush ] ERROR o.a.i.d.m.p.e.c.ConfigExecution:124 - Failures happened > during running ConfigExecution > -- > > Key: IOTDB-4761 > URL: https://issues.apache.org/jira/browse/IOTDB-4761 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Yongzao Dan >Priority: Major > Attachments: screenshot-1.png > > > /ssd_data/mpp_test/m_1025_cbc6225 > schema region : ratis > data region : multiLeader > 3rep , 3C3D > During benchmark writing, ip72 is set to read-only due to "no space left on > device". > Wait for the benchmark write to complete, ip72 release disk space. > ip72 : SET SYSTEM TO RUNNING > Wait for synchronization to complete > ip72 : flush , datanode error log: > 2022-10-26 14:48:29,578 [pool-21-IoTDB-MPPCoordinator-14] ERROR > o.a.i.d.m.p.e.c.ConfigExecution:124 - Failures happened during running > ConfigExecution. > org.apache.thrift.TException: Error in calling method flush > at > org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94) > at > org.apache.iotdb.db.client.ConfigNodeClient$$EnhancerByCGLIB$$4b2c71c8.flush() > at > org.apache.iotdb.db.mpp.plan.execution.config.executor.ClusterConfigTaskExecutor.flush(ClusterConfigTaskExecutor.java:516) > at > org.apache.iotdb.db.mpp.plan.execution.config.sys.FlushTask.execute(FlushTask.java:58) > at > org.apache.iotdb.db.mpp.plan.execution.config.ConfigExecution.start(ConfigExecution.java:99) > at > org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:146) > at > org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatement(ClientRPCServiceImpl.java:601) > at > org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3037) > at > org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3017) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: org.apache.thrift.TException: Fail to connect to any config node. > Please check server status > at > org.apache.iotdb.db.client.ConfigNodeClient.flush(ConfigNodeClient.java:722) > at > org.apache.iotdb.db.client.ConfigNodeClient$$EnhancerByCGLIB$$4b2c71c8.CGLIB$flush$70() > at > org.apache.iotdb.db.client.ConfigNodeClient$$EnhancerByCGLIB$$4b2c71c8$$FastClassByCGLIB$$46f69aba.invoke() > at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) > at > org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55) > ... 14 common frames omitted > !screenshot-1.png! > The test process is the same as > https://issues.apache.org/jira/browse/IOTDB-4752 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-2789) [Trigger] Status code: EXECUTE_STATEMENT_ERROR(400), operation: checkAuthority failed
[ https://issues.apache.org/jira/browse/IOTDB-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haonan Hou reassigned IOTDB-2789: - Assignee: (was: Steve Yurong Su) > [Trigger] Status code: EXECUTE_STATEMENT_ERROR(400), operation: > checkAuthority failed > - > > Key: IOTDB-2789 > URL: https://issues.apache.org/jira/browse/IOTDB-2789 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: 0.13.0 >Reporter: 刘珍 >Priority: Minor > Attachments: TriggerTest.jar, TriggerTest.java, > image-2022-03-23-14-40-45-507.png > > > 0.13.0 rc1 > 复现用例： > delete storage group root.**; > set storage group to root.sg1; > CREATE TIMESERIES root.sg1.dev1.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > CREATE TIMESERIES root.sg1.dev1.s_2 WITH DATATYPE=INT64, ENCODING=PLAIN; > CREATE TIMESERIES root.sg1.dev1.s_3 WITH DATATYPE=float, ENCODING=RLE; > CREATE TIMESERIES root.sg1.dev1.s_4 WITH DATATYPE=double, ENCODING=TS_2DIFF; > CREATE TIMESERIES root.sg1.dev1.s_5 WITH DATATYPE=text, ENCODING=DICTIONARY; > CREATE TIMESERIES root.sg1.dev1.s_6 WITH DATATYPE=boolean, ENCODING=PLAIN; > CREATE TRIGGER trig1 > AFTER INSERT > ON root.sg1.dev1.s_1 > AS 'TriggerTest' > WITH ( > 'ts_type' = 'int32', > 'remote_ip' = '127.0.0.1', > 'trig_name'='trig1' > ); > CREATE TRIGGER trig2 > AFTER INSERT > ON root.sg1.dev1.s_2 > AS 'TriggerTest' > WITH ( > 'ts_type' = 'int64', > 'remote_ip' = '127.0.0.1', > 'trig_name'='trig2' > ); > insert into root.sg1.dev1(time,s_1,s_2) values(1,100,200); > insert into root.sg1.dev1(time,s_1,s_2) values(2,300,400); > select * from root.sg1.dev1; > select * from root.target.alerting; > delete timeseries root.sg1.dev1.s_1; > insert into root.sg1.dev1(time,s_2) values(100,1600); > select * from root.sg1.dev1; > select * from root.target.alerting; > 最后一条查询，下图红框不应该是null，应该是1600 > !image-2022-03-23-14-40-45-507.png! > 错误日志： > 2022-03-23 14:37:21,435 [pool-3-thread-1] ERROR > o.a.i.d.u.ErrorHandlingUtils:59 - Status code: EXECUTE_STATEMENT_ERROR(400), > operation: checkAuthority failed > java.lang.NullPointerException: null > at org.apache.iotdb.db.concurrent.HashLock.readLock(HashLock.java:48) > at > org.apache.iotdb.db.auth.user.BasicUserManager.getUser(BasicUserManager.java:86) > at > org.apache.iotdb.db.auth.authorizer.BasicAuthorizer.checkUserPrivileges(BasicAuthorizer.java:285) > at > org.apache.iotdb.db.auth.AuthorityChecker.checkOnePath(AuthorityChecker.java:87) > at > org.apache.iotdb.db.auth.AuthorityChecker.check(AuthorityChecker.java:72) > at > org.apache.iotdb.db.service.basic.ServiceProvider.checkAuthorization(ServiceProvider.java:126) > at > org.apache.iotdb.db.service.basic.ServiceProvider.checkAuthority(ServiceProvider.java:132) > at > org.apache.iotdb.db.protocol.mqtt.PublishHandler.onPublish(PublishHandler.java:122) > at > io.moquette.interception.BrokerInterceptor.lambda$notifyTopicPublished$3(BrokerInterceptor.java:133) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4934) [UDF]attribute is null without quote, return 500
[
https://issues.apache.org/jira/browse/IOTDB-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635080#comment-17635080
]
李伟豪 commented on IOTDB-4934:
[https://github.com/apache/iotdb/pull/8022] the error message will be `303:
Check metadata error: Unsupported constant operand: null`
> [UDF]attribute is null without quote, return 500
> -
>
> Key: IOTDB-4934
> URL: https://issues.apache.org/jira/browse/IOTDB-4934
> Project: Apache IoTDB
> Issue Type: Bug
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: changxue
>Assignee: 李伟豪
>Priority: Minor
> Attachments: image-2022-11-15-10-38-05-254.png,
> normla-udf-0.14-SNAPSHOT.jar
>
>
> [UDF]attribute is null without quote, return 500
> 不应该报错500
> reproduction:
> {code}
> create function count_sec as
> "com.timecho.udf.normal.TestSessionTimeWindowAccessStrategy";
> CREATE DATABASE root.udf;
> CREATE TIMESERIES root.udf.d1.s1 WITH DATATYPE=text, ENCODING=PLAIN;
> CREATE TIMESERIES root.udf.d1.s2 WITH DATATYPE=text, ENCODING=PLAIN;
> insert into root.udf.d1 (time, s1, s2) values (1, 'udf', 'trigger'),(2,
> 'some' , 'old'),(3, null, 'trigger'),(9, 'udf', 'new'),(5, 'woman',
> 'udf'),(12, 'trigger', null),(14, 'udf', 14),(18, 'udf', 16),(21, 'trigger',
> 18),(24, 'udf', 20);
> select count_sec(s1, 'max_interval'=1, 'standard'=null) from root.udf.d1;
> select count_sec(s1, 'max_interval'=1, 'standard'=NULL) from root.udf.d1;
> {code}
> error message:
> {code}
> IoTDB> select count_sec(s1, 'max_interval'=4, 'standard'=null) from
> root.udf.d1;
> Msg: 500: [INTERNAL_SERVER_ERROR(500)] Exception occurred: "select
> count_sec(s1, 'max_interval'=4, 'standard'=null) from root.udf.d1".
> executeStatement failed.
> org.apache.iotdb.db.mpp.plan.expression.leaf.NullOperand cannot be cast to
> org.apache.iotdb.db.mpp.plan.expression.leaf.ConstantOperand{code}
> !image-2022-11-15-10-38-05-254.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-4966) Remove datanode to support graceful exit of processes
刘珍 created IOTDB-4966: - Summary: Remove datanode to support graceful exit of processes Key: IOTDB-4966 URL: https://issues.apache.org/jira/browse/IOTDB-4966 Project: Apache IoTDB Issue Type: Improvement Components: mpp-cluster Reporter: 刘珍 Assignee: Gaofei Cao 缩容datanode，进程退出不够优雅，有一些报错（可加入处理计划，优先级可低）： 2022-11-16 15:54:13,378 [pool-24-IoTDB-DataNodeInternalRPC-Processor-51] ERROR o.a.i.c.s.ThriftService:168 - IoTDB: close MPP Data exchange manager failed because: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.iotdb.commons.service.ThriftService.stopService(ThriftService.java:163) at org.apache.iotdb.commons.service.ThriftService.stop(ThriftService.java:74) at org.apache.iotdb.db.mpp.execution.exchange.MPPDataExchangeService.stop(MPPDataExchangeService.java:128) at org.apache.iotdb.commons.service.IService.waitAndStop(IService.java:36) at org.apache.iotdb.commons.service.RegisterManager.deregisterAll(RegisterManager.java:60) at org.apache.iotdb.db.service.DataNode.deactivate(DataNode.java:683) at org.apache.iotdb.db.service.DataNode.stop(DataNode.java:651) at org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.stopDataNode(DataNodeInternalRPCServiceImpl.java:1493) at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$stopDataNode.getResult(IDataNodeRPCService.java:4082) at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$stopDataNode.getResult(IDataNodeRPCService.java:4062) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-11-16 15:54:13,388 [pool-24-IoTDB-DataNodeInternalRPC-Processor-51] ERROR o.a.i.c.s.RegisterManager:63 - Failed to stop Storage Engine ServerService because: org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to stop because of Timed-Flush-Unseq-Memtable. at org.apache.iotdb.db.utils.ThreadUtils.stopThreadPool(ThreadUtils.java:47) at org.apache.iotdb.db.engine.StorageEngineV2.stop(StorageEngineV2.java:389) at org.apache.iotdb.commons.service.IService.waitAndStop(IService.java:36) at org.apache.iotdb.commons.service.RegisterManager.deregisterAll(RegisterManager.java:60) at org.apache.iotdb.db.service.DataNode.deactivate(DataNode.java:683) at org.apache.iotdb.db.service.DataNode.stop(DataNode.java:651) at org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.stopDataNode(DataNodeInternalRPCServiceImpl.java:1493) at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$stopDataNode.getResult(IDataNodeRPCService.java:4082) at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$stopDataNode.getResult(IDataNodeRPCService.java:4062) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2067) at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475) at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675) at org.apache.iotdb.commons.concurrent.threadpool.WrappedSingleThreadScheduledExecutor.awaitTermination(WrappedSingleThreadScheduledExecutor.java:99) at org.apache.iotdb.db.utils.ThreadUtils.stopThreadPool(ThreadUtils.java:40) ... 14 common frames omitted 2022-11-16 15:54:13,389 [pool-24-IoTDB-DataNodeInternalRPC-Processor-51] ERROR o.a.i.d.w.b.WALBuffer:521 - Fail to put CLOSE_SIGNAL to walEntries. java.lang.InterruptedException: null at
[jira] [Assigned] (IOTDB-4946) Return SDT information when show timeseries
[ https://issues.apache.org/jira/browse/IOTDB-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yukun Zhou reassigned IOTDB-4946: - Assignee: Yukun Zhou > Return SDT information when show timeseries > --- > > Key: IOTDB-4946 > URL: https://issues.apache.org/jira/browse/IOTDB-4946 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: Yukun Zhou >Priority: Major > Labels: pull-request-available > > We support SDT lossy compression but cannot show these information out. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4473) Support TSBS Timeseries Benchmark
[ https://issues.apache.org/jira/browse/IOTDB-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634921#comment-17634921 ] Xingyu Liu commented on IOTDB-4473: --- 目前有两项优化工作： 1. 数据插入方式优化为使用 InsertRecords方式插入。不使用tablet的原因为同一batch下相同节点的数据可能很少，需要插入大量只有几行的tablet，效率低。 2. tags处理方式改为使用一个不记录数据的节点 _tags 利用 attributes 记录tags信息，此处的tags指的是tsbs项目内的称呼。这里不使用iotdb中tags结构的原因是，attributes不会建立倒排索引更节省资源，并且tsbs测试中，tags的值始终不变，也与查询无关。 上述两项优化，应该都能提升插入测试性能，目前前者已完成，后者正在测试调整。 后续会实际测出优化的效果。 > Support TSBS Timeseries Benchmark > - > > Key: IOTDB-4473 > URL: https://issues.apache.org/jira/browse/IOTDB-4473 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Julian Feinauer >Assignee: Xingyu Liu >Priority: Major > Attachments: image-2022-11-06-19-37-51-033.png > > > The tsbs Benchmark was initially created by InfluxDB and is now maintained by > timescale db and compares many state of the art timeseries databases: > [https://github.com/timescale/tsbs] > > IMHO we should add support for IoTDB -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4616) Support serialization and deserialization for confignode onecopy mode
[ https://issues.apache.org/jira/browse/IOTDB-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634856#comment-17634856 ] 刘俊男 commented on IOTDB-4616: 在机器4C16G 12G堆内存配置下 10点的数据 simple的数据吞吐量：7310177.48 p/s左右 ratis的数据吞吐量：5357918.00 p/s左右 simple较ratis大约 36%左右提升 > Support serialization and deserialization for confignode onecopy mode > -- > > Key: IOTDB-4616 > URL: https://issues.apache.org/jira/browse/IOTDB-4616 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Gaofei Cao >Assignee: 刘俊男 >Priority: Major > Labels: pull-request-available > Attachments: image-2022-11-01-09-22-09-403.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-4850) Disable first election for now
[ https://issues.apache.org/jira/browse/IOTDB-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Song Ziyang reopened IOTDB-4850: > Disable first election for now > -- > > Key: IOTDB-4850 > URL: https://issues.apache.org/jira/browse/IOTDB-4850 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Song Ziyang >Assignee: Song Ziyang >Priority: Major > Labels: pull-request-available > > first election aims to reduce the unavailable time during consensus group > starts-up. However, in current implementation, it may cause new follower > receive SHUTDOWN errors. > I'll disable it for now. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4965) TTL not takes effect after datanode restart
[ https://issues.apache.org/jira/browse/IOTDB-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haonan Hou reassigned IOTDB-4965: - Assignee: Haonan Hou > TTL not takes effect after datanode restart > --- > > Key: IOTDB-4965 > URL: https://issues.apache.org/jira/browse/IOTDB-4965 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.14.0-SNAPSHOT >Reporter: Haonan Hou >Assignee: Haonan Hou >Priority: Major > > Set ttl on root.sg 1000 > insert into root.sg.d1(time,s1) values(1,2) [FAIL] > restart datanode > insert into root.sg.d1(time,s1) values(1,2) [SUCCESS] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4965) TTL not takes effect after datanode restart
Haonan Hou created IOTDB-4965: - Summary: TTL not takes effect after datanode restart Key: IOTDB-4965 URL: https://issues.apache.org/jira/browse/IOTDB-4965 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.14.0-SNAPSHOT Reporter: Haonan Hou Set ttl on root.sg 1000 restart datanode insert into root.sg.d1(time,s1) values(1,2) insert success -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4964) [ Region-Migrate ] Migrating regions can be implemented in parallel
刘珍 created IOTDB-4964: - Summary: [ Region-Migrate ] Migrating regions can be implemented in parallel Key: IOTDB-4964 URL: https://issues.apache.org/jira/browse/IOTDB-4964 Project: Apache IoTDB Issue Type: Improvement Components: mpp-cluster Reporter: 刘珍 Assignee: Gaofei Cao Attachments: image-2022-11-16-18-03-48-136.png Region迁移可优化为并行，提升迁移/缩容性能。 !image-2022-11-16-18-03-48-136.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4963) The verification msg of Continuous Interval Functions needs to be optimized
xiaozhihong created IOTDB-4963:
--
Summary: The verification msg of Continuous Interval Functions
needs to be optimized
Key: IOTDB-4963
URL: https://issues.apache.org/jira/browse/IOTDB-4963
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: xiaozhihong
Assignee: liaolanyu
Attachments: image-2022-11-16-17-11-55-305.png
Continuous Interval Functions for unsupported data types needs to be optimized,
for example, (index:0) need to indicate which timeseries datatype does not
match.
{code:java}
IoTDB> show timeseries root.sg.d1.*;
+-+-++++---++--+
| Timeseries|Alias|Database|DataType|Encoding|Compression|Tags|Attributes|
+-+-++++---++--+
|root.sg.d1.s3| null| root.sg| TEXT| PLAIN| SNAPPY|null| null|
|root.sg.d1.s1| null| root.sg| INT64| RLE| SNAPPY|null| null|
|root.sg.d1.s2| null| root.sg| FLOAT| RLE| SNAPPY|null| null|
+-+-++++---++--+
Total line number = 3
It costs 0.004s
IoTDB> select zero_duration(s3) from root.sg.d1;
Msg: 416: Error occurred during inferring UDF data type:
org.apache.iotdb.udf.api.exception.UDFInputSeriesDataTypeNotValidException: the
data type of the input series (index: 0) is not valid. expected: [INT32, INT64,
FLOAT, DOUBLE, BOOLEAN]. actual: TEXT.
{code}
!image-2022-11-16-17-11-55-305.png|width=803,height=141!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4875) When creating a timeseries path is illegal, the msg reports 500
[
https://issues.apache.org/jira/browse/IOTDB-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634736#comment-17634736
]
xiaozhihong commented on IOTDB-4875:
!image-2022-11-16-16-54-07-611.png!
> When creating a timeseries path is illegal, the msg reports 500
> ---
>
> Key: IOTDB-4875
> URL: https://issues.apache.org/jira/browse/IOTDB-4875
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: xiaozhihong
>Assignee: Yukun Zhou
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-11-16-16-54-07-611.png
>
>
> In 0.14.0 snapshot version， created a illegal path timeseries, then returned
> message reports 500.
> Like this:
> {code:java}
> IoTDB> create timeseries root with datatype=INT32,encoding=RLE;
> Msg: 500: [INTERNAL_SERVER_ERROR(500)] Exception occurred: "create timeseries
> root with datatype=INT32,encoding=RLE". executeStatement failed. Index 1 out
> of bounds for length 1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4960) [SchemaFile] Query large amounts of metadata throw ClosedChannelException
[
https://issues.apache.org/jira/browse/IOTDB-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634708#comment-17634708
]
yanze chen commented on IOTDB-4960:
---
原因可能是mpp框架查询超时，interrupt掉了正在read的channel，导致channel关闭
!image-2022-11-16-16-01-50-022.png|width=827,height=407!
> [SchemaFile] Query large amounts of metadata throw ClosedChannelException
> -
>
> Key: IOTDB-4960
> URL: https://issues.apache.org/jira/browse/IOTDB-4960
> Project: Apache IoTDB
> Issue Type: Bug
>Reporter: yanze chen
>Assignee: yanze chen
>Priority: Major
> Attachments: image-2022-11-16-15-29-18-436.png,
> image-2022-11-16-16-01-50-022.png
>
>
> reproduce:
> 1C1D Schema_File mode 128G for DataNode
>
> run benchmark with config:
> {*}DEVICE_NUMBER{*}=5000
> {*}SENSOR_NUMBER{*}=2
> {*}CLIENT_NUMBER{*}=5
> {*}GROUP_NUMBER{*}=10
> {*}BATCH_SIZE_PER_WRITE{*}=0
> {*}LOOP{*}=1
> {*}VECTOR{*}=true
>
> execute SQL:
> count timeseries root.**
>
> !image-2022-11-16-15-29-18-436.png|width=881,height=596!
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
