subject:"\?\?\?\?\?\?\?\?\?\?\?\?\?\?\-\?\?\?\?\?\?"

But having another look at the documentation at 
https://iotdb.apache.org/UserGuide/V1.0.x/Cluster/Cluster-Maintenance.html#show-datanode-information

The NodeID displayed in “show cluster” (datanodes have ids 3,4 and 5) seem to 
be different from the ones shown in show datanodes (1 and 2)
Also did show confignodes list “0, 1 and 2” … was every screen output captured 
on a different cluster configuration? If yes … we should probably update these 
(manually) to be more aligned.

Chris

Von: Christofer Dutz 
Datum: Donnerstag, 25. Juli 2024 um 09:44
An: dev@iotdb.apache.org 
Betreff: AW: How to handle the internal and external IPs of Data-Nodes?
Comparing the output of them,

I think deleting nodes only by ID is probably the best option.
Perhaps in a second PR we could update the output of show datanodes to include 
the InternalAddress and InternalPort?

I probably also should check if removing config nodes via ID works … because 
technically removing a datanode by Id didn’t work till my fix.

So, I think I should undo the parts I added for replacing the 0.0.0.0 with the 
internal address and double check the remove config-node code.

Chris

Von: Wang Critas 
Datum: Donnerstag, 25. Juli 2024 um 09:36
An: dev@iotdb.apache.org 
Betreff: 答复: How to handle the internal and external IPs of Data-Nodes?
Hi Chris

Perhaps more often than not, show clusters are used?

Best

Xuan Wang
发件人: Christofer Dutz 
日期: 星期四, 2024年7月25日 15:31
收件人: dev@iotdb.apache.org 
主题: AW: How to handle the internal and external IPs of Data-Nodes?
Hi all,

In that case we would need to show the internal ip in the show datanodes 
command, right?
Or would we simply expect the user to know the internal IP of the datanode he 
wants to remove?

Chris

Von: Wang Critas 
Datum: Donnerstag, 25. Juli 2024 um 08:16
An: dev@iotdb.apache.org 
Betreff: 答复: How to handle the internal and external IPs of Data-Nodes?
Hi

Why not use nodeId or internalAddress:internalPort for unified behavior

Remove ConfigNode through nodeId or internalAddress:internamPort

Best

Xuan Wang

发件人: Xinyu Tan 
日期: 星期四, 2024年7月25日 12:26
收件人: dev@iotdb.apache.org 
主题: Re: How to handle the internal and external IPs of Data-Nodes?
Hi, chris

+1 for identifying nodes solely by nodeid

Best

Xinyu Tan

On 2024/07/24 14:33:20 Christofer Dutz wrote:
> Hi all,
>
> I’m currently working on 
> https://github.com/apache/iotdb/pull/12914
>
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
>
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
>
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
>
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
>
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
>
> What do you think?
>
> Chris
>

AW: How to handle the internal and external IPs of Data-Nodes?

Comparing the output of them,

I think deleting nodes only by ID is probably the best option.
Perhaps in a second PR we could update the output of show datanodes to include 
the InternalAddress and InternalPort?

I probably also should check if removing config nodes via ID works … because 
technically removing a datanode by Id didn’t work till my fix.

So, I think I should undo the parts I added for replacing the 0.0.0.0 with the 
internal address and double check the remove config-node code.

Chris

Von: Wang Critas 
Datum: Donnerstag, 25. Juli 2024 um 09:36
An: dev@iotdb.apache.org 
Betreff: 答复: How to handle the internal and external IPs of Data-Nodes?
Hi Chris

Perhaps more often than not, show clusters are used?

Best

Xuan Wang
发件人: Christofer Dutz 
日期: 星期四, 2024年7月25日 15:31
收件人: dev@iotdb.apache.org 
主题: AW: How to handle the internal and external IPs of Data-Nodes?
Hi all,

In that case we would need to show the internal ip in the show datanodes 
command, right?
Or would we simply expect the user to know the internal IP of the datanode he 
wants to remove?

Chris

Von: Wang Critas 
Datum: Donnerstag, 25. Juli 2024 um 08:16
An: dev@iotdb.apache.org 
Betreff: 答复: How to handle the internal and external IPs of Data-Nodes?
Hi

Why not use nodeId or internalAddress:internalPort for unified behavior

Remove ConfigNode through nodeId or internalAddress:internamPort

Best

Xuan Wang

发件人: Xinyu Tan 
日期: 星期四, 2024年7月25日 12:26
收件人: dev@iotdb.apache.org 
主题: Re: How to handle the internal and external IPs of Data-Nodes?
Hi, chris

+1 for identifying nodes solely by nodeid

Best

Xinyu Tan

On 2024/07/24 14:33:20 Christofer Dutz wrote:
> Hi all,
>
> I’m currently working on 
> https://github.com/apache/iotdb/pull/12914
>
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
>
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
>
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
>
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
>
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
>
> What do you think?
>
> Chris
>

答复: How to handle the internal and external IPs of Data-Nodes?

2024-07-25 Thread Wang Critas

Hi Chris

Perhaps more often than not, show clusters are used?

Best

Xuan Wang
发件人: Christofer Dutz 
日期: 星期四, 2024年7月25日 15:31
收件人: dev@iotdb.apache.org 
主题: AW: How to handle the internal and external IPs of Data-Nodes?
Hi all,

In that case we would need to show the internal ip in the show datanodes 
command, right?
Or would we simply expect the user to know the internal IP of the datanode he 
wants to remove?

Chris

Von: Wang Critas 
Datum: Donnerstag, 25. Juli 2024 um 08:16
An: dev@iotdb.apache.org 
Betreff: 答复: How to handle the internal and external IPs of Data-Nodes?
Hi

Why not use nodeId or internalAddress:internalPort for unified behavior

Remove ConfigNode through nodeId or internalAddress:internamPort

Best

Xuan Wang

发件人: Xinyu Tan 
日期: 星期四, 2024年7月25日 12:26
收件人: dev@iotdb.apache.org 
主题: Re: How to handle the internal and external IPs of Data-Nodes?
Hi, chris

+1 for identifying nodes solely by nodeid

Best

Xinyu Tan

On 2024/07/24 14:33:20 Christofer Dutz wrote:
> Hi all,
>
> I’m currently working on 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fiotdb%2Fpull%2F12914=05%7C02%7C%7C56b5bbc3bf6a43ef86e808dcac7bd6ad%7C84df9e7fe9f640afb435%7C1%7C0%7C638574895089604721%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=yGMJdVfhHQTbUvl6s2gKPhs8Yha%2FF4I5J%2Frzm6jPW1c%3D=0
>
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
>
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
>
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
>
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
>
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
>
> What do you think?
>
> Chris
>

AW: How to handle the internal and external IPs of Data-Nodes?

Hi all,

In that case we would need to show the internal ip in the show datanodes 
command, right?
Or would we simply expect the user to know the internal IP of the datanode he 
wants to remove?

Chris

Von: Wang Critas 
Datum: Donnerstag, 25. Juli 2024 um 08:16
An: dev@iotdb.apache.org 
Betreff: 答复: How to handle the internal and external IPs of Data-Nodes?
Hi

Why not use nodeId or internalAddress:internalPort for unified behavior

Remove ConfigNode through nodeId or internalAddress:internamPort

Best

Xuan Wang

发件人: Xinyu Tan 
日期: 星期四, 2024年7月25日 12:26
收件人: dev@iotdb.apache.org 
主题: Re: How to handle the internal and external IPs of Data-Nodes?
Hi, chris

+1 for identifying nodes solely by nodeid

Best

Xinyu Tan

On 2024/07/24 14:33:20 Christofer Dutz wrote:
> Hi all,
>
> I’m currently working on 
> https://github.com/apache/iotdb/pull/12914
>
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
>
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
>
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
>
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
>
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
>
> What do you think?
>
> Chris
>

AW: How to handle the internal and external IPs of Data-Nodes?

That would be my favorite option … to directly show the internal and external 
ip/ports.

With my fix in the PR, removing by ID is now possible, so yes … then we should 
remove the option to use the Ip+port notation.

Chris

Von: 乔嘉林 
Datum: Donnerstag, 25. Juli 2024 um 03:05
An: dev@iotdb.apache.org 
Betreff: Re: How to handle the internal and external IPs of Data-Nodes?
Hi,

Could we list both two addresses? One for internal and another for client 
connection.

As for removing a datanode, maybe prividing the nodeid is enough.

Jialin
> From: "Christofer Dutz"
> Date:  Wed, Jul 24, 2024, 22:37
> Subject:  How to handle the internal and external IPs of Data-Nodes?
> To: "dev@iotdb.apache.org"
> Hi all,
>
> I’m currently working on https://github.com/apache/iotdb/pull/12914
>
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
>
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
>
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
>
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
>
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
>
> What do you think?
>
> Chris

答复: How to handle the internal and external IPs of Data-Nodes?

2024-07-25 Thread Wang Critas

Hi

Why not use nodeId or internalAddress:internalPort for unified behavior

Remove ConfigNode through nodeId or internalAddress:internamPort

Best

Xuan Wang

发件人: Xinyu Tan 
日期: 星期四, 2024年7月25日 12:26
收件人: dev@iotdb.apache.org 
主题: Re: How to handle the internal and external IPs of Data-Nodes?
Hi, chris

+1 for identifying nodes solely by nodeid

Best

Xinyu Tan

On 2024/07/24 14:33:20 Christofer Dutz wrote:
> Hi all,
>
> I’m currently working on 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fiotdb%2Fpull%2F12914=05%7C02%7C%7C0e82ba702c3f4f33d87108dcac61da79%7C84df9e7fe9f640afb435%7C1%7C0%7C638574783738380055%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=9TNkrTQlrQZHqhx%2B%2BxJslUQVALiZEzyAx%2BROPepFkmE%3D=0
>
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
>
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
>
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
>
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
>
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
>
> What do you think?
>
> Chris
>

Re: How to handle the internal and external IPs of Data-Nodes?

2024-07-24 Thread Xinyu Tan

Hi, chris

+1 for identifying nodes solely by nodeid

Best

Xinyu Tan

On 2024/07/24 14:33:20 Christofer Dutz wrote:
> Hi all,
> 
> I’m currently working on https://github.com/apache/iotdb/pull/12914
> 
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
> 
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
> 
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
> 
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
> 
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
> 
> What do you think?
> 
> Chris
>

Re: How to handle the internal and external IPs of Data-Nodes?

2024-07-24 Thread 乔嘉林

Hi,

Could we list both two addresses? One for internal and another for client 
connection.

As for removing a datanode, maybe prividing the nodeid is enough.

Jialin
> From: "Christofer Dutz"
> Date:  Wed, Jul 24, 2024, 22:37
> Subject:  How to handle the internal and external IPs of Data-Nodes?
> To: "dev@iotdb.apache.org"
> Hi all,
> 
> I’m currently working on https://github.com/apache/iotdb/pull/12914
> 
> Here one of the problems was, that in a cluster with 3 datanodes, where the 
> dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each 
> node with an address of 0.0.0.0. So, if someone wants to remove a data-node 
> by its IP, the cli will not find the corresponding node as it thinks it’s 
> 0.0.0.0. However, if you use the cli to delete 0.0.0.0, then it deletes all 
> nodes.
> 
> Now my initial fix for this, was, that if a data-node registers and says his 
> dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
> getting the request is being used (Obviously this one exists and belongs to 
> the data-node registering).
> The problem is that this usually will be the dn_internal_address instead of 
> the dn_rpc_address.
> 
> Now we could use that instead, but I think it would reduce the usefulness of 
> the “show datanodes” command, because it could be in a cluster-internal 
> network, that the client can’t connect to.
> So, if a client wants to know which other data-nodes there are in order to 
> connect to another one, this might not be helpful.
> However, 0.0.0.0 is also not helpful, as I see no chance to be able to 
> connect to a data-node using 0.0.0.0:6667 as that’s not really a real Ip 
> address.
> 
> So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
> this with the list of public IP addresses the data-node possesses. In this 
> case show data-nodes would no longer display only one IP-Address, but a list 
> of IP-Addresses.
> 
> When removing a data-node (or sending any other commands to it) we could now 
> identify a particular data-node as only one will have the IP+Port combination.
> 
> What do you think?
> 
> Chris

How to handle the internal and external IPs of Data-Nodes?

2024-07-24 Thread Christofer Dutz

Hi all,

I’m currently working on https://github.com/apache/iotdb/pull/12914

Here one of the problems was, that in a cluster with 3 datanodes, where the 
dn_rpc_address is set to 0.0.0.0, the command “show datanodes” lists each node 
with an address of 0.0.0.0. So, if someone wants to remove a data-node by its 
IP, the cli will not find the corresponding node as it thinks it’s 0.0.0.0. 
However, if you use the cli to delete 0.0.0.0, then it deletes all nodes.

Now my initial fix for this, was, that if a data-node registers and says his 
dn_rpc_address is 0.0.0.0, that instead of this, the IP from which we are 
getting the request is being used (Obviously this one exists and belongs to the 
data-node registering).
The problem is that this usually will be the dn_internal_address instead of the 
dn_rpc_address.

Now we could use that instead, but I think it would reduce the usefulness of 
the “show datanodes” command, because it could be in a cluster-internal 
network, that the client can’t connect to.
So, if a client wants to know which other data-nodes there are in order to 
connect to another one, this might not be helpful.
However, 0.0.0.0 is also not helpful, as I see no chance to be able to connect 
to a data-node using 0.0.0.0:6667 as that’s not really a real Ip address.

So, I think, that possibly instead of maintaining 0.0.0.0 we should replace 
this with the list of public IP addresses the data-node possesses. In this case 
show data-nodes would no longer display only one IP-Address, but a list of 
IP-Addresses.

When removing a data-node (or sending any other commands to it) we could now 
identify a particular data-node as only one will have the IP+Port combination.

What do you think?

Chris

Re: [PR] chore(deps): Bump braces from 3.0.2 to 3.0.3 in /connectors/grafana-plugin [iotdb-extras]

2024-07-24 Thread via GitHub



HTHou merged PR #12:
URL: https://github.com/apache/iotdb-extras/pull/12


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] chore(deps): Bump braces from 3.0.2 to 3.0.3 in /connectors/grafana-plugin [iotdb-extras]



dependabot[bot] opened a new pull request, #12:
URL: https://github.com/apache/iotdb-extras/pull/12

   Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3.
   
   Commits
   
   https://github.com/micromatch/braces/commit/74b2db2938fad48a2ea54a9c8bf27a37a62c350d;>74b2db2
 3.0.3
   https://github.com/micromatch/braces/commit/88f1429a0f47e1dd3813de35211fc97ffda27f9e;>88f1429
 update eslint. lint, fix unit tests.
   https://github.com/micromatch/braces/commit/415d660c3002d1ab7e63dbf490c9851da80596ff;>415d660
 Snyk js braces 6838727 (https://redirect.github.com/micromatch/braces/issues/40;>#40)
   https://github.com/micromatch/braces/commit/190510f79db1adf21d92798b0bb6fccc1f72c9d6;>190510f
 fix tests, skip 1 test in test/braces.expand
   https://github.com/micromatch/braces/commit/716eb9f12d820b145a831ad678618731927e8856;>716eb9f
 readme bump
   https://github.com/micromatch/braces/commit/a5851e57f45c3431a94d83fc565754bc10f5bbc3;>a5851e5
 Merge pull request https://redirect.github.com/micromatch/braces/issues/37;>#37 from 
coderaiser/fix/vulnerability
   https://github.com/micromatch/braces/commit/2092bd1fb108d2c59bd62e243b70ad98db961538;>2092bd1
 feature: braces: add maxSymbols (https://github.com/micromatch/braces/issues/;>https://github.com/micromatch/braces/issues/...
   https://github.com/micromatch/braces/commit/9f5b4cf47329351bcb64287223ffb6ecc9a5e6d3;>9f5b4cf
 fix: vulnerability (https://security.snyk.io/vuln/SNYK-JS-BRACES-6838727;>https://security.snyk.io/vuln/SNYK-JS-BRACES-6838727)
   https://github.com/micromatch/braces/commit/98414f9f1fabe021736e26836d8306d5de747e0d;>98414f9
 remove funding file
   https://github.com/micromatch/braces/commit/665ab5d561c017a38ba7aafd92cc6655b91d8c14;>665ab5d
 update keepEscaping doc (https://redirect.github.com/micromatch/braces/issues/27;>#27)
   Additional commits viewable in https://github.com/micromatch/braces/compare/3.0.2...3.0.3;>compare 
view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=braces=npm_and_yarn=3.0.2=3.0.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot show  ignore conditions` will show all of 
the ignore conditions of the specified dependency
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   You can disable automated security fix PRs for this repo from the [Security 
Alerts page](https://github.com/apache/iotdb-extras/network/alerts).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] chore(deps): Bump golang.org/x/net from 0.17.0 to 0.23.0 in /connectors/grafana-plugin [iotdb-extras]



HTHou merged PR #7:
URL: https://github.com/apache/iotdb-extras/pull/7


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] chore(deps): Bump ws from 7.5.9 to 7.5.10 in /connectors/grafana-plugin [iotdb-extras]



HTHou merged PR #9:
URL: https://github.com/apache/iotdb-extras/pull/9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] chore(deps): Bump fast-loops from 1.1.3 to 1.1.4 in /connectors/grafana-plugin [iotdb-extras]



HTHou merged PR #11:
URL: https://github.com/apache/iotdb-extras/pull/11


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] Next steps with predictive test selection?

Hi all,

I know the IoTDB build is taking longer and longer to run. Also is IoTDB the 
unfortunately one of the biggest consumer of GitHub Actions compute resources 
at Apache (At least we were pointed out as such at Community Over Code 
Bratislava). There I worked together with some folks from Develocity and we 
prepared a PR, that would enable predictive test selection and reordering in 
our build.

It would run tests that failed the last time first, if a build needs to be 
re-run. This reduces the waste of resources as the tests failing the last time 
are most probably going to be the ones failing again.

Another optimization is, that you can run only those test that are most 
probably related to code changes you did (Predictive test selection).

The way we currently test (with almost everything in the integration-test 
module) we can’t utilize the biggest benefit from this, but it did improve 
resource usage quite a bit when we tried it out.

I don’t think it’s fair, that we are using so much of the ASFs GitHub Compute 
resources and would really like to improve things here.

The PR has been waiting for quite some time now:
https://github.com/apache/iotdb/pull/12654

Chris

AW: AW: AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

Ok … guess my numbers were not 100% correct.
It was pointed out that it says: “that you are part of” at the top of the page.
So from all project that I’m a committer of (it shows 9) IoTDB consumed 95% of 
the resources.
Not quite as bad as I thought at first, but still not good.

Chris

Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 10:48
An: dev@iotdb.apache.org 
Betreff: AW: AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability
And just so you know how huge this problem is:
https://infra-reports.apache.org/#ghactions

For all non-members, that don’t have access to that resource:
It shows how much of the GitHub Actions resources each project is using.
Apache IoTDB is currently using 95% of all GitHub Actions resources, leaving 
less than 5% for the other 200 projects we have.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 10:41
An: dev@iotdb.apache.org 
Betreff: AW: AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability
I think for some tests, it might be worth migrating them back from an IT to an 
UT.
I’m currently trying to get resource-consumption stats from infra and wanted to 
start finding out which are our most “expensive” Its and possibly start a 
discussion on converting them to unit-tests. Right now, it feels as for every 
bit of functionality we add more Its, which all require spinning up of small 
instances of IoTDB.
Also is it really challenging to test the unhappy path in integration-test, 
which comes with other problems.

So yeah … I hope by removing the singletons from a component, we can make it 
Unit-Testable and then we can gradually reduce the load on the integration-test 
suite.

Chris


Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 10:35
An: dev 
Betreff: Re: AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability
Certainly, I am not actually expecting a really perfect solution, just hoping 
when someone else sees this discussion, he may provide a better trade-off.


And as I have said, I am happy with both solutions, as long as they can get rid 
of the singletons.


Mocking is helpful in tests. But the modifications (when actually removing 
singletons from the main source code) in constructors may still be painful.
Of course, it is enough just for testing purpose.


However, for the problem concerning resource consumption, are you going to 
remove some current ITs if your changes are applied?

If no, the resource consumption is still there; if yes, it may be challenging 
to guarantee the equivalence of the ITs and UTs.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 16:19 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability |
Hi Tian,

I think – as usual – there probably is no perfect solution.
I guess it comes down to deciding which things you are willing to do in order 
to solve some other problems on the other side.

I personally think our biggest problem right now (At least this is what’s 
making my Life a lot harder working on IoTDB) is the inability to run 
unit-tests and I see how much resources we are wasting on GitHub Actions. 
Constructor-Injection solves that problem.

Of course, could be build some sort of Factory, that helps initialize some 
methods, but admittedly I think a simple:

SomeService someService = Mockito.mock(SomeService.class);

Is not too challenging to do.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:55
An: dev 
Betreff: Re: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability


A template (or util) method can provide basic contexts, and the developers just 
need to override the parts they need (of course they should know which parts 
they need).
In the constructor-way, the developer should know what instances they will use, 
create them, and pass them to the constructor;
while in the context-way, the developer should know what fields they will use, 
create them, and override them in the context.
I do not think they are essentially different.


The problem is, for high level structures like DataRegion,  they will have many 
many more fields because their children use many different ones, and the 
constructor will be very long for them.
Certainly, you may further migrate the constructor-way to the builder-way, 
which may solve the problem. However, it requires much more extra work.


I am not particularly fond of either one and I just want to point out the 
problems, hoping someone may give a perfect solution.
If none is given, I am happy to accept either one.


Tian Jiang


 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:39 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW:

AW: AW: AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

And just so you know how huge this problem is:
https://infra-reports.apache.org/#ghactions

For all non-members, that don’t have access to that resource:
It shows how much of the GitHub Actions resources each project is using.
Apache IoTDB is currently using 95% of all GitHub Actions resources, leaving 
less than 5% for the other 200 projects we have.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 10:41
An: dev@iotdb.apache.org 
Betreff: AW: AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability
I think for some tests, it might be worth migrating them back from an IT to an 
UT.
I’m currently trying to get resource-consumption stats from infra and wanted to 
start finding out which are our most “expensive” Its and possibly start a 
discussion on converting them to unit-tests. Right now, it feels as for every 
bit of functionality we add more Its, which all require spinning up of small 
instances of IoTDB.
Also is it really challenging to test the unhappy path in integration-test, 
which comes with other problems.

So yeah … I hope by removing the singletons from a component, we can make it 
Unit-Testable and then we can gradually reduce the load on the integration-test 
suite.

Chris


Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 10:35
An: dev 
Betreff: Re: AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability
Certainly, I am not actually expecting a really perfect solution, just hoping 
when someone else sees this discussion, he may provide a better trade-off.


And as I have said, I am happy with both solutions, as long as they can get rid 
of the singletons.


Mocking is helpful in tests. But the modifications (when actually removing 
singletons from the main source code) in constructors may still be painful.
Of course, it is enough just for testing purpose.


However, for the problem concerning resource consumption, are you going to 
remove some current ITs if your changes are applied?

If no, the resource consumption is still there; if yes, it may be challenging 
to guarantee the equivalence of the ITs and UTs.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 16:19 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability |
Hi Tian,

I think – as usual – there probably is no perfect solution.
I guess it comes down to deciding which things you are willing to do in order 
to solve some other problems on the other side.

I personally think our biggest problem right now (At least this is what’s 
making my Life a lot harder working on IoTDB) is the inability to run 
unit-tests and I see how much resources we are wasting on GitHub Actions. 
Constructor-Injection solves that problem.

Of course, could be build some sort of Factory, that helps initialize some 
methods, but admittedly I think a simple:

SomeService someService = Mockito.mock(SomeService.class);

Is not too challenging to do.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:55
An: dev 
Betreff: Re: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability


A template (or util) method can provide basic contexts, and the developers just 
need to override the parts they need (of course they should know which parts 
they need).
In the constructor-way, the developer should know what instances they will use, 
create them, and pass them to the constructor;
while in the context-way, the developer should know what fields they will use, 
create them, and override them in the context.
I do not think they are essentially different.


The problem is, for high level structures like DataRegion,  they will have many 
many more fields because their children use many different ones, and the 
constructor will be very long for them.
Certainly, you may further migrate the constructor-way to the builder-way, 
which may solve the problem. However, it requires much more extra work.


I am not particularly fond of either one and I just want to point out the 
problems, hoping someone may give a perfect solution.
If none is given, I am happy to accept either one.


Tian Jiang


 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:39 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
However, then you need to construct these contexts with loads of parameters for 
unit tests … I personally don’t think passing in explicitly what you need into 
a constructor is bad.
So, if you have this service that you want to test: How can you tell the 
developer writing the test which services are needed and therefore need to be 
mocked in the context?
I would strongly opt for explicit dependency

AW: AW: AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

I think for some tests, it might be worth migrating them back from an IT to an 
UT.
I’m currently trying to get resource-consumption stats from infra and wanted to 
start finding out which are our most “expensive” Its and possibly start a 
discussion on converting them to unit-tests. Right now, it feels as for every 
bit of functionality we add more Its, which all require spinning up of small 
instances of IoTDB.
Also is it really challenging to test the unhappy path in integration-test, 
which comes with other problems.

So yeah … I hope by removing the singletons from a component, we can make it 
Unit-Testable and then we can gradually reduce the load on the integration-test 
suite.

Chris


Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 10:35
An: dev 
Betreff: Re: AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability
Certainly, I am not actually expecting a really perfect solution, just hoping 
when someone else sees this discussion, he may provide a better trade-off.


And as I have said, I am happy with both solutions, as long as they can get rid 
of the singletons.


Mocking is helpful in tests. But the modifications (when actually removing 
singletons from the main source code) in constructors may still be painful.
Of course, it is enough just for testing purpose.


However, for the problem concerning resource consumption, are you going to 
remove some current ITs if your changes are applied?

If no, the resource consumption is still there; if yes, it may be challenging 
to guarantee the equivalence of the ITs and UTs.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 16:19 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability |
Hi Tian,

I think – as usual – there probably is no perfect solution.
I guess it comes down to deciding which things you are willing to do in order 
to solve some other problems on the other side.

I personally think our biggest problem right now (At least this is what’s 
making my Life a lot harder working on IoTDB) is the inability to run 
unit-tests and I see how much resources we are wasting on GitHub Actions. 
Constructor-Injection solves that problem.

Of course, could be build some sort of Factory, that helps initialize some 
methods, but admittedly I think a simple:

SomeService someService = Mockito.mock(SomeService.class);

Is not too challenging to do.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:55
An: dev 
Betreff: Re: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability


A template (or util) method can provide basic contexts, and the developers just 
need to override the parts they need (of course they should know which parts 
they need).
In the constructor-way, the developer should know what instances they will use, 
create them, and pass them to the constructor;
while in the context-way, the developer should know what fields they will use, 
create them, and override them in the context.
I do not think they are essentially different.


The problem is, for high level structures like DataRegion,  they will have many 
many more fields because their children use many different ones, and the 
constructor will be very long for them.
Certainly, you may further migrate the constructor-way to the builder-way, 
which may solve the problem. However, it requires much more extra work.


I am not particularly fond of either one and I just want to point out the 
problems, hoping someone may give a perfect solution.
If none is given, I am happy to accept either one.


Tian Jiang


 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:39 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
However, then you need to construct these contexts with loads of parameters for 
unit tests … I personally don’t think passing in explicitly what you need into 
a constructor is bad.
So, if you have this service that you want to test: How can you tell the 
developer writing the test which services are needed and therefore need to be 
mocked in the context?
I would strongly opt for explicit dependency injection in the constructor and 
not using some context.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:27
An: dev 
Betreff: Re: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Yes, maybe we could have leveled context like DataNodeContext, 
DataRegionContext, TsFileProcessorContext... and put the former singletons in 
these contexts.


This may avoid constructors with dozens of parameters.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date

Re: AW: AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

2024-07-18 Thread Tian Jiang

Certainly, I am not actually expecting a really perfect solution, just hoping 
when someone else sees this discussion, he may provide a better trade-off.


And as I have said, I am happy with both solutions, as long as they can get rid 
of the singletons.


Mocking is helpful in tests. But the modifications (when actually removing 
singletons from the main source code) in constructors may still be painful.
Of course, it is enough just for testing purpose.


However, for the problem concerning resource consumption, are you going to 
remove some current ITs if your changes are applied?

If no, the resource consumption is still there; if yes, it may be challenging 
to guarantee the equivalence of the ITs and UTs.


Tian Jiang 


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 16:19 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability |
Hi Tian,

I think – as usual – there probably is no perfect solution.
I guess it comes down to deciding which things you are willing to do in order 
to solve some other problems on the other side.

I personally think our biggest problem right now (At least this is what’s 
making my Life a lot harder working on IoTDB) is the inability to run 
unit-tests and I see how much resources we are wasting on GitHub Actions. 
Constructor-Injection solves that problem.

Of course, could be build some sort of Factory, that helps initialize some 
methods, but admittedly I think a simple:

SomeService someService = Mockito.mock(SomeService.class);

Is not too challenging to do.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:55
An: dev 
Betreff: Re: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability


A template (or util) method can provide basic contexts, and the developers just 
need to override the parts they need (of course they should know which parts 
they need).
In the constructor-way, the developer should know what instances they will use, 
create them, and pass them to the constructor;
while in the context-way, the developer should know what fields they will use, 
create them, and override them in the context.
I do not think they are essentially different.


The problem is, for high level structures like DataRegion,  they will have many 
many more fields because their children use many different ones, and the 
constructor will be very long for them.
Certainly, you may further migrate the constructor-way to the builder-way, 
which may solve the problem. However, it requires much more extra work.


I am not particularly fond of either one and I just want to point out the 
problems, hoping someone may give a perfect solution.
If none is given, I am happy to accept either one.


Tian Jiang


 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:39 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
However, then you need to construct these contexts with loads of parameters for 
unit tests … I personally don’t think passing in explicitly what you need into 
a constructor is bad.
So, if you have this service that you want to test: How can you tell the 
developer writing the test which services are needed and therefore need to be 
mocked in the context?
I would strongly opt for explicit dependency injection in the constructor and 
not using some context.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:27
An: dev 
Betreff: Re: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Yes, maybe we could have leveled context like DataNodeContext, 
DataRegionContext, TsFileProcessorContext... and put the former singletons in 
these contexts.


This may avoid constructors with dozens of parameters.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:22 |
| To | dev@iotdb.apache.org |
| Subject | AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
Ideally,

we could inspect the usages of the constructors using the singletons and apply 
the same pattern there. Once all usages have been migrated that way, we could 
even eliminate the Singleton-using constructor. In the end getting a system for 
which we can finally write sensible unit-tests again and don’t need to rely on 
Integration tests.
Integration tests that have problems running in paralell, while unit-tests can 
run perfectly in paralell.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 09:03
An: dev@iotdb.apache.org 
Betreff: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Ok …

So, as there was no objection here, I’ll trat that as

How to use the new streaming queries?

Hi all,

I am currently working a bit on my little PLC4X + IoTDB Home automation project 
which I plan to use to make little videos about IoTDB features.
Right now, what I’m trying to figure out, is how can I define a streaming query?

I have multiple devices, each one providing numerous measuements.
The ones I’m interested now, is one where all my weather data is stored (Wind, 
Rain, Humidity, Temperature, …)
I have a little frontend, that displays the current values using a “LAST(*) 
query.
Now I would like to have a streaming query, which returns either all last 
values whenever one of the values changes.
Or a stream of changes for only the values that actually changes.

I’m a bit stuck at to how to do this using the Java API.

Help greatly appreciated.

Chris

AW: AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

Hi Tian,

I think – as usual – there probably is no perfect solution.
I guess it comes down to deciding which things you are willing to do in order 
to solve some other problems on the other side.

I personally think our biggest problem right now (At least this is what’s 
making my Life a lot harder working on IoTDB) is the inability to run 
unit-tests and I see how much resources we are wasting on GitHub Actions. 
Constructor-Injection solves that problem.

Of course, could be build some sort of Factory, that helps initialize some 
methods, but admittedly I think a simple:

SomeService someService = Mockito.mock(SomeService.class);

Is not too challenging to do.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:55
An: dev 
Betreff: Re: AW: AW: Possible pattern for reducing the negative effects of 
using singletons and improving testability


A template (or util) method can provide basic contexts, and the developers just 
need to override the parts they need (of course they should know which parts 
they need).
In the constructor-way, the developer should know what instances they will use, 
create them, and pass them to the constructor;
while in the context-way, the developer should know what fields they will use, 
create them, and override them in the context.
I do not think they are essentially different.


The problem is, for high level structures like DataRegion,  they will have many 
many more fields because their children use many different ones, and the 
constructor will be very long for them.
Certainly, you may further migrate the constructor-way to the builder-way, 
which may solve the problem. However, it requires much more extra work.


I am not particularly fond of either one and I just want to point out the 
problems, hoping someone may give a perfect solution.
If none is given, I am happy to accept either one.


Tian Jiang


 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:39 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
However, then you need to construct these contexts with loads of parameters for 
unit tests … I personally don’t think passing in explicitly what you need into 
a constructor is bad.
So, if you have this service that you want to test: How can you tell the 
developer writing the test which services are needed and therefore need to be 
mocked in the context?
I would strongly opt for explicit dependency injection in the constructor and 
not using some context.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:27
An: dev 
Betreff: Re: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Yes, maybe we could have leveled context like DataNodeContext, 
DataRegionContext, TsFileProcessorContext... and put the former singletons in 
these contexts.


This may avoid constructors with dozens of parameters.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:22 |
| To | dev@iotdb.apache.org |
| Subject | AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
Ideally,

we could inspect the usages of the constructors using the singletons and apply 
the same pattern there. Once all usages have been migrated that way, we could 
even eliminate the Singleton-using constructor. In the end getting a system for 
which we can finally write sensible unit-tests again and don’t need to rely on 
Integration tests.
Integration tests that have problems running in paralell, while unit-tests can 
run perfectly in paralell.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 09:03
An: dev@iotdb.apache.org 
Betreff: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Ok …

So, as there was no objection here, I’ll trat that as consensus and I’ll just 
do this for every component that I touch and hereby need to write tests for.

Chris

Von: Christofer Dutz 
Datum: Montag, 15. Juli 2024 um 11:49
An: dev@iotdb.apache.org 
Betreff: Possible pattern for reducing the negative effects of using singletons 
and improving testability
Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons

Re: AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

2024-07-18 Thread Tian Jiang



A template (or util) method can provide basic contexts, and the developers just 
need to override the parts they need (of course they should know which parts 
they need).
In the constructor-way, the developer should know what instances they will use, 
create them, and pass them to the constructor;
while in the context-way, the developer should know what fields they will use, 
create them, and override them in the context.
I do not think they are essentially different.


The problem is, for high level structures like DataRegion,  they will have many 
many more fields because their children use many different ones, and the 
constructor will be very long for them.
Certainly, you may further migrate the constructor-way to the builder-way, 
which may solve the problem. However, it requires much more extra work.


I am not particularly fond of either one and I just want to point out the 
problems, hoping someone may give a perfect solution.
If none is given, I am happy to accept either one.


Tian Jiang


 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:39 |
| To | dev@iotdb.apache.org |
| Subject | AW: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
However, then you need to construct these contexts with loads of parameters for 
unit tests … I personally don’t think passing in explicitly what you need into 
a constructor is bad.
So, if you have this service that you want to test: How can you tell the 
developer writing the test which services are needed and therefore need to be 
mocked in the context?
I would strongly opt for explicit dependency injection in the constructor and 
not using some context.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:27
An: dev 
Betreff: Re: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Yes, maybe we could have leveled context like DataNodeContext, 
DataRegionContext, TsFileProcessorContext... and put the former singletons in 
these contexts.


This may avoid constructors with dozens of parameters.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:22 |
| To | dev@iotdb.apache.org |
| Subject | AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
Ideally,

we could inspect the usages of the constructors using the singletons and apply 
the same pattern there. Once all usages have been migrated that way, we could 
even eliminate the Singleton-using constructor. In the end getting a system for 
which we can finally write sensible unit-tests again and don’t need to rely on 
Integration tests.
Integration tests that have problems running in paralell, while unit-tests can 
run perfectly in paralell.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 09:03
An: dev@iotdb.apache.org 
Betreff: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Ok …

So, as there was no objection here, I’ll trat that as consensus and I’ll just 
do this for every component that I touch and hereby need to write tests for.

Chris

Von: Christofer Dutz 
Datum: Montag, 15. Juli 2024 um 11:49
An: dev@iotdb.apache.org 
Betreff: Possible pattern for reducing the negative effects of using singletons 
and improving testability
Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons over 
time.

As I was just working on DataNodeServerCommandLine I did an experiment with 
that, and it worked nicely.

So, I searched for all singletons used in the component and defined private 
final member variables for each of them.
In the no-args constructor I initialize these via the singletons, but I also 
add a second constructor, that allows passing in explicit versions.


private final ConfigNodeInfo configNodeInfo;
private final IClientManager 
configNodeClientManager;
private final DataNode dataNode;

/** Default constructor using the singletons for initializing the relationship. 
*/
public DataNodeServerCommandLine() {
configNodeInfo = ConfigNodeInfo.getInstance();
configNodeClientManager = ConfigNodeClientManager.getInstance();
dataNode = DataNode.getInstance();
}

/**
* Additional constructor allowing injection of custom instances (mainly

AW: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

However, then you need to construct these contexts with loads of parameters for 
unit tests … I personally don’t think passing in explicitly what you need into 
a constructor is bad.
So, if you have this service that you want to test: How can you tell the 
developer writing the test which services are needed and therefore need to be 
mocked in the context?
I would strongly opt for explicit dependency injection in the constructor and 
not using some context.

Chris

Von: Tian Jiang 
Datum: Donnerstag, 18. Juli 2024 um 09:27
An: dev 
Betreff: Re: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Yes, maybe we could have leveled context like DataNodeContext, 
DataRegionContext, TsFileProcessorContext... and put the former singletons in 
these contexts.


This may avoid constructors with dozens of parameters.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:22 |
| To | dev@iotdb.apache.org |
| Subject | AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
Ideally,

we could inspect the usages of the constructors using the singletons and apply 
the same pattern there. Once all usages have been migrated that way, we could 
even eliminate the Singleton-using constructor. In the end getting a system for 
which we can finally write sensible unit-tests again and don’t need to rely on 
Integration tests.
Integration tests that have problems running in paralell, while unit-tests can 
run perfectly in paralell.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 09:03
An: dev@iotdb.apache.org 
Betreff: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Ok …

So, as there was no objection here, I’ll trat that as consensus and I’ll just 
do this for every component that I touch and hereby need to write tests for.

Chris

Von: Christofer Dutz 
Datum: Montag, 15. Juli 2024 um 11:49
An: dev@iotdb.apache.org 
Betreff: Possible pattern for reducing the negative effects of using singletons 
and improving testability
Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons over 
time.

As I was just working on DataNodeServerCommandLine I did an experiment with 
that, and it worked nicely.

So, I searched for all singletons used in the component and defined private 
final member variables for each of them.
In the no-args constructor I initialize these via the singletons, but I also 
add a second constructor, that allows passing in explicit versions.


private final ConfigNodeInfo configNodeInfo;
private final IClientManager 
configNodeClientManager;
private final DataNode dataNode;

/** Default constructor using the singletons for initializing the relationship. 
*/
public DataNodeServerCommandLine() {
configNodeInfo = ConfigNodeInfo.getInstance();
configNodeClientManager = ConfigNodeClientManager.getInstance();
dataNode = DataNode.getInstance();
}

/**
* Additional constructor allowing injection of custom instances (mainly for 
testing)
*
* @param configNodeInfo config node info
* @param configNodeClientManager config node client manager
* @param dataNode data node
*/
public DataNodeServerCommandLine(
ConfigNodeInfo configNodeInfo,
IClientManager configNodeClientManager,
DataNode dataNode) {
this.configNodeInfo = configNodeInfo;
this.configNodeClientManager = configNodeClientManager;
this.dataNode = dataNode;
}

With this was I now able to write real unit-tests for the component, without 
having to require writing an Integration-test which costs a lot more resources 
and which makes implementing unhappy-path tests really challenging.

With this approach, I think we can really improve testability in IoTDB … 
considering a class test coverage of only 24% and a method coverage of just 19% 
in the data node, we really need this.

What do you all think?


Chris

Re: AW: Possible pattern for reducing the negative effects of using singletons and improving testability

2024-07-18 Thread Tian Jiang

Yes, maybe we could have leveled context like DataNodeContext, 
DataRegionContext, TsFileProcessorContext... and put the former singletons in 
these contexts.


This may avoid constructors with dozens of parameters.


Tian Jiang


| |
Tian Jiang
|
|
jt2594...@163.com
|
 Replied Message 
| From | Christofer Dutz |
| Date | 7/18/2024 15:22 |
| To | dev@iotdb.apache.org |
| Subject | AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability |
Ideally,

we could inspect the usages of the constructors using the singletons and apply 
the same pattern there. Once all usages have been migrated that way, we could 
even eliminate the Singleton-using constructor. In the end getting a system for 
which we can finally write sensible unit-tests again and don’t need to rely on 
Integration tests.
Integration tests that have problems running in paralell, while unit-tests can 
run perfectly in paralell.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 09:03
An: dev@iotdb.apache.org 
Betreff: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Ok …

So, as there was no objection here, I’ll trat that as consensus and I’ll just 
do this for every component that I touch and hereby need to write tests for.

Chris

Von: Christofer Dutz 
Datum: Montag, 15. Juli 2024 um 11:49
An: dev@iotdb.apache.org 
Betreff: Possible pattern for reducing the negative effects of using singletons 
and improving testability
Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons over 
time.

As I was just working on DataNodeServerCommandLine I did an experiment with 
that, and it worked nicely.

So, I searched for all singletons used in the component and defined private 
final member variables for each of them.
In the no-args constructor I initialize these via the singletons, but I also 
add a second constructor, that allows passing in explicit versions.


private final ConfigNodeInfo configNodeInfo;
private final IClientManager 
configNodeClientManager;
private final DataNode dataNode;

/** Default constructor using the singletons for initializing the relationship. 
*/
public DataNodeServerCommandLine() {
configNodeInfo = ConfigNodeInfo.getInstance();
configNodeClientManager = ConfigNodeClientManager.getInstance();
dataNode = DataNode.getInstance();
}

/**
* Additional constructor allowing injection of custom instances (mainly for 
testing)
*
* @param configNodeInfo config node info
* @param configNodeClientManager config node client manager
* @param dataNode data node
*/
public DataNodeServerCommandLine(
ConfigNodeInfo configNodeInfo,
IClientManager configNodeClientManager,
DataNode dataNode) {
this.configNodeInfo = configNodeInfo;
this.configNodeClientManager = configNodeClientManager;
this.dataNode = dataNode;
}

With this was I now able to write real unit-tests for the component, without 
having to require writing an Integration-test which costs a lot more resources 
and which makes implementing unhappy-path tests really challenging.

With this approach, I think we can really improve testability in IoTDB … 
considering a class test coverage of only 24% and a method coverage of just 19% 
in the data node, we really need this.

What do you all think?


Chris

AW: Possible pattern for reducing the negative effects of using singletons and improving testability

Ideally,

we could inspect the usages of the constructors using the singletons and apply 
the same pattern there. Once all usages have been migrated that way, we could 
even eliminate the Singleton-using constructor. In the end getting a system for 
which we can finally write sensible unit-tests again and don’t need to rely on 
Integration tests.
Integration tests that have problems running in paralell, while unit-tests can 
run perfectly in paralell.

Chris


Von: Christofer Dutz 
Datum: Donnerstag, 18. Juli 2024 um 09:03
An: dev@iotdb.apache.org 
Betreff: AW: Possible pattern for reducing the negative effects of using 
singletons and improving testability
Ok …

So, as there was no objection here, I’ll trat that as consensus and I’ll just 
do this for every component that I touch and hereby need to write tests for.

Chris

Von: Christofer Dutz 
Datum: Montag, 15. Juli 2024 um 11:49
An: dev@iotdb.apache.org 
Betreff: Possible pattern for reducing the negative effects of using singletons 
and improving testability
Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons over 
time.

As I was just working on DataNodeServerCommandLine I did an experiment with 
that, and it worked nicely.

So, I searched for all singletons used in the component and defined private 
final member variables for each of them.
In the no-args constructor I initialize these via the singletons, but I also 
add a second constructor, that allows passing in explicit versions.


private final ConfigNodeInfo configNodeInfo;
private final IClientManager 
configNodeClientManager;
private final DataNode dataNode;

/** Default constructor using the singletons for initializing the relationship. 
*/
public DataNodeServerCommandLine() {
  configNodeInfo = ConfigNodeInfo.getInstance();
  configNodeClientManager = ConfigNodeClientManager.getInstance();
  dataNode = DataNode.getInstance();
}

/**
* Additional constructor allowing injection of custom instances (mainly for 
testing)
*
* @param configNodeInfo config node info
* @param configNodeClientManager config node client manager
* @param dataNode data node
*/
public DataNodeServerCommandLine(
ConfigNodeInfo configNodeInfo,
IClientManager configNodeClientManager,
DataNode dataNode) {
  this.configNodeInfo = configNodeInfo;
  this.configNodeClientManager = configNodeClientManager;
  this.dataNode = dataNode;
}

With this was I now able to write real unit-tests for the component, without 
having to require writing an Integration-test which costs a lot more resources 
and which makes implementing unhappy-path tests really challenging.

With this approach, I think we can really improve testability in IoTDB … 
considering a class test coverage of only 24% and a method coverage of just 19% 
in the data node, we really need this.

What do you all think?


Chris

AW: Possible pattern for reducing the negative effects of using singletons and improving testability

Ok …

So, as there was no objection here, I’ll trat that as consensus and I’ll just 
do this for every component that I touch and hereby need to write tests for.

Chris

Von: Christofer Dutz 
Datum: Montag, 15. Juli 2024 um 11:49
An: dev@iotdb.apache.org 
Betreff: Possible pattern for reducing the negative effects of using singletons 
and improving testability
Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons over 
time.

As I was just working on DataNodeServerCommandLine I did an experiment with 
that, and it worked nicely.

So, I searched for all singletons used in the component and defined private 
final member variables for each of them.
In the no-args constructor I initialize these via the singletons, but I also 
add a second constructor, that allows passing in explicit versions.


private final ConfigNodeInfo configNodeInfo;
private final IClientManager 
configNodeClientManager;
private final DataNode dataNode;

/** Default constructor using the singletons for initializing the relationship. 
*/
public DataNodeServerCommandLine() {
  configNodeInfo = ConfigNodeInfo.getInstance();
  configNodeClientManager = ConfigNodeClientManager.getInstance();
  dataNode = DataNode.getInstance();
}

/**
* Additional constructor allowing injection of custom instances (mainly for 
testing)
*
* @param configNodeInfo config node info
* @param configNodeClientManager config node client manager
* @param dataNode data node
*/
public DataNodeServerCommandLine(
ConfigNodeInfo configNodeInfo,
IClientManager configNodeClientManager,
DataNode dataNode) {
  this.configNodeInfo = configNodeInfo;
  this.configNodeClientManager = configNodeClientManager;
  this.dataNode = dataNode;
}

With this was I now able to write real unit-tests for the component, without 
having to require writing an Integration-test which costs a lot more resources 
and which makes implementing unhappy-path tests really challenging.

With this approach, I think we can really improve testability in IoTDB … 
considering a class test coverage of only 24% and a method coverage of just 19% 
in the data node, we really need this.

What do you all think?


Chris

RE: [DISCUSS] Clean handling of issues in CLIs?

2024-07-15 Thread Stephen Lawrence



> -Original Message-
> From: Christofer Dutz 
> Sent: Monday, July 15, 2024 1:03 PM
> To: dev@iotdb.apache.org
> Subject: [DISCUSS] Clean handling of issues in CLIs?
> 
> Hi all,
> 
> also while working on the DataNodeServerCli I noticed that we don't
> actually handle how CLIs deal with issues correctly.
> While in most case in case of a successful operation we return a code of 0,
> we don't consequently do this in case of issues.
> Sometimes we return -1 or other cases, but in most cases we simply fire
> exceptions that terminate the CLI.
> 
> I think we should define a set of globally adopted return codes and
> consequently catch and handle all exceptions that could occur in CLIs and
> map these to these standard return codes and log readable messages instead
> of dumping stack-traces.
> 
> While for a human running these scripts, the stacktrace usually should
> provide enough information, they might be overwhelmed by chains of
> stacktraces ... also does a process calling these scripts don't know how to
> interpret them.
> 
> What do you think?

+1 for this idea.

Particularly if the IoTDB client CLI is affected as it will be a common
method for newbies trying out IoTDB, as well as being used in scripting.

I refer to its use in the COVESA Central Data Service Playground documentation
as it fits interactive trials, and this will increase as we add more examples.

Best Wishes,

Steve

[DISCUSS] Clean handling of issues in CLIs?

2024-07-15 Thread Christofer Dutz

Hi all,

also while working on the DataNodeServerCli I noticed that we don’t actually 
handle how CLIs deal with issues correctly.
While in most case in case of a successful operation we return a code of 0, we 
don’t consequently do this in case of issues.
Sometimes we return -1 or other cases, but in most cases we simply fire 
exceptions that terminate the CLI.

I think we should define a set of globally adopted return codes and 
consequently catch and handle all exceptions that could occur in CLIs and map 
these to these standard return codes and log readable messages instead of 
dumping stack-traces.

While for a human running these scripts, the stacktrace usually should provide 
enough information, they might be overwhelmed by chains of stacktraces … also 
does a process calling these scripts don’t know how to interpret them.

What do you think?

Possible pattern for reducing the negative effects of using singletons and improving testability

2024-07-15 Thread Christofer Dutz

Hi all,

I am currently working on some functions and when writing tests for these, I 
had problems sensibly testing things, as the singletons were preventing that.
As I mentioned before … I consider the usage of singletons the probably worst 
design decision done in IoTDBs core. Even if it simplifies “composition”, it 
really makes everything else more complicated.

Knowing how deeply rooted in IoTDB the use of singletons is, I always thought 
this would be a never-ending story.
But I just recently had an idea of how we could probably find a way to get the 
testability that we need without having to give up on singletons.
Possibly this path could also provide a path to getting rid of singletons over 
time.

As I was just working on DataNodeServerCommandLine I did an experiment with 
that, and it worked nicely.

So, I searched for all singletons used in the component and defined private 
final member variables for each of them.
In the no-args constructor I initialize these via the singletons, but I also 
add a second constructor, that allows passing in explicit versions.


private final ConfigNodeInfo configNodeInfo;
private final IClientManager 
configNodeClientManager;
private final DataNode dataNode;

/** Default constructor using the singletons for initializing the relationship. 
*/
public DataNodeServerCommandLine() {
  configNodeInfo = ConfigNodeInfo.getInstance();
  configNodeClientManager = ConfigNodeClientManager.getInstance();
  dataNode = DataNode.getInstance();
}

/**
* Additional constructor allowing injection of custom instances (mainly for 
testing)
*
* @param configNodeInfo config node info
* @param configNodeClientManager config node client manager
* @param dataNode data node
*/
public DataNodeServerCommandLine(
ConfigNodeInfo configNodeInfo,
IClientManager configNodeClientManager,
DataNode dataNode) {
  this.configNodeInfo = configNodeInfo;
  this.configNodeClientManager = configNodeClientManager;
  this.dataNode = dataNode;
}

With this was I now able to write real unit-tests for the component, without 
having to require writing an Integration-test which costs a lot more resources 
and which makes implementing unhappy-path tests really challenging.

With this approach, I think we can really improve testability in IoTDB … 
considering a class test coverage of only 24% and a method coverage of just 19% 
in the data node, we really need this.

What do you all think?


Chris

AW: Re:[DISCUSS] Design of Table Model Permission Feature

2024-07-11 Thread Christofer Dutz

Apache email lists (except security lists) don’t accept attachments.

Chris

Von: Colin_Lee 
Datum: Donnerstag, 11. Juli 2024 um 06:40
An: dev@iotdb.apache.org 
Betreff: Re:[DISCUSS] Design of Table Model Permission Feature
Soory that It appears that there has been an unexpected issue; the attachment 
was not accepted and forwarded by the mail server.

You can get the design document from the URL bellow.




Design_zh: 
https://apache-iotdb.feishu.cn/docx/BYwjdvm73ohwgmxQldbcQ9w8nTy?from=from_copylink

Design_eng: 
https://apache-iotdb.feishu.cn/docx/IRmYdOIhhoAAECxugxScxHYAnfg?from=from_copylink




Thanks,

Colin.




At 2024-07-11 12:24:40, "Colin_Lee"  wrote:

Hello everyone,


I'm recently working on the functionality related to table model permissions. 
At present, there is already a basic table model permission design document 
that is open for discussion. This document has referenced the design ideas of 
permissions in the mainstream relational databases. In the design document, I 
have provided the categories of permissions, their scope, and the corresponding 
operation-permission relationship tables. This feature design is public and 
open to everyone's suggestions.


The design documents are attached; if you are unable to view the design 
documents, please visit mailList[1] to download them.


[1] https://lists.apache.org/list.html?dev@iotdb.apache.org


Regards,
Colin.

Re:[DISCUSS] Design of Table Model Permission Feature

2024-07-10 Thread Colin_Lee

Soory that It appears that there has been an unexpected issue; the attachment 
was not accepted and forwarded by the mail server.

You can get the design document from the URL bellow.




Design_zh: 
https://apache-iotdb.feishu.cn/docx/BYwjdvm73ohwgmxQldbcQ9w8nTy?from=from_copylink

Design_eng: 
https://apache-iotdb.feishu.cn/docx/IRmYdOIhhoAAECxugxScxHYAnfg?from=from_copylink




Thanks,

Colin.




At 2024-07-11 12:24:40, "Colin_Lee"  wrote:

Hello everyone,


I'm recently working on the functionality related to table model permissions. 
At present, there is already a basic table model permission design document 
that is open for discussion. This document has referenced the design ideas of 
permissions in the mainstream relational databases. In the design document, I 
have provided the categories of permissions, their scope, and the corresponding 
operation-permission relationship tables. This feature design is public and 
open to everyone's suggestions.


The design documents are attached; if you are unable to view the design 
documents, please visit mailList[1] to download them.


[1] https://lists.apache.org/list.html?dev@iotdb.apache.org


Regards,
Colin.

Re:[DISCUSS] Design of Table Model Permission Feature

2024-07-10 Thread Colin_Lee




I have re-attached the email attachment.










At 2024-07-11 12:24:40, "Colin_Lee"  wrote:

Hello everyone,


I'm recently working on the functionality related to table model permissions. 
At present, there is already a basic table model permission design document 
that is open for discussion. This document has referenced the design ideas of 
permissions in the mainstream relational databases. In the design document, I 
have provided the categories of permissions, their scope, and the corresponding 
operation-permission relationship tables. This feature design is public and 
open to everyone's suggestions.


The design documents are attached; if you are unable to view the design 
documents, please visit mailList[1] to download them.


[1] https://lists.apache.org/list.html?dev@iotdb.apache.org


Regards,
Colin.

[DISCUSS] Design of Table Model Permission Feature

2024-07-10 Thread Colin_Lee

Hello everyone,


I'm recently working on the functionality related to table model permissions. 
At present, there is already a basic table model permission design document 
that is open for discussion. This document has referenced the design ideas of 
permissions in the mainstream relational databases. In the design document, I 
have provided the categories of permissions, their scope, and the corresponding 
operation-permission relationship tables. This feature design is public and 
open to everyone's suggestions.


The design documents are attached; if you are unable to view the design 
documents, please visit mailList[1] to download them.


[1] https://lists.apache.org/list.html?dev@iotdb.apache.org


Regards,
Colin.

[PR] chore(deps): Bump fast-loops from 1.1.3 to 1.1.4 in /connectors/grafana-plugin [iotdb-extras]

2024-07-10 Thread via GitHub



dependabot[bot] opened a new pull request, #11:
URL: https://github.com/apache/iotdb-extras/pull/11

   Bumps [fast-loops](https://github.com/robinweser/fast-loops) from 1.1.3 to 
1.1.4.
   
   Commits
   
   See full diff in https://github.com/robinweser/fast-loops/commits;>compare view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=fast-loops=npm_and_yarn=1.1.3=1.1.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot show  ignore conditions` will show all of 
the ignore conditions of the specified dependency
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   You can disable automated security fix PRs for this repo from the [Security 
Alerts page](https://github.com/apache/iotdb-extras/network/alerts).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Enable C# Client to Intialize and Reconnect with Multiple Node Urls

2024-07-07 Thread ZHAN LU

Hi,

I have implemented an interface within the C# client that supports
initialization using multiple Node URLs. Additionally, the reconnection
method has been enhanced to select one of the various Nodes for
reestablishing a connection. For further details, please refer to PR[1].

Would someone be available to assist with a review? Thank you!

Zhan Lu

[1] https://github.com/apache/iotdb-client-csharp/pull/8

RE: Re:Integrating UDF Library and Grafana (conf update) in Docker

2024-07-01 Thread Stephen Lawrence

Miaohsh wrote:
> The rest service currently does not support hot loading. After modifying
> the configuration, the iotdb service needs to be restarted before it can
> be used
> 

Thank you for the confirmation. I just wanted to check I was understanding
the IoTDB configuration mechanisms correctly.

As I needed to integrate the UDF Data Quality Library as well I took the
decision to maintain an IoTDB Dockerfile in the downstream COVESA Data 
Service Playground project that builds on the upstream IoTDB image. 
I have that working locally, along with the REST configuration change.

I have a short list of issues I found in the IoTDB documentation.
I will look at sending a Github PR to fix them. 
I will also look at updating the Playground to use IoTDB 1.3.2.

Best Wishes,

Steve

[ANNOUNCE] Apache IoTDB 1.3.2 released

2024-07-01 Thread Haonan Hou

The Apache IoTDB team is pleased to announce the release of Apache IoTDB
1.3.2.

Apache IoTDB (Database for Internet of Things) is an IoT native database
with high performance for data management and analysis, deployable on the
edge and the cloud.

Apache IoTDB 1.3.2 is the latest version of the v1 series. Feel free to upgrade
from earlier version.

## Features & Improvements
- Storage Module: Performance improvement in the insertRecords interface for 
writing
- Query Module: New Explain Analyze statement added (monitoring the time spent 
on each stage of a single SQL execution)
- Query Module: New UDAF (User-Defined Aggregate Function) framework added
- Query Module: New MaxBy/MinBy functions added, supporting the retrieval of 
maximum/minimum values along with the corresponding timestamps
- Query Module: Performance improvement in value filtering queries
- Data Synchronization: Path matching supports path pattern
- Data Synchronization: Supports metadata synchronization (including time 
series and related attributes, permissions, etc.)
- Stream Processing: Added Alter Pipe statement, supporting hot updates of 
plugins for Pipe tasks
- System Module: System data point count statistics now include statistics for 
data imported by loading TsFile
- Scripts and Tools: New local upgrade backup tool added (backing up original 
data through hard links)
- Scripts and Tools: New export-data/import-data scripts added, supporting data 
export in CSV, TsFile formats or SQL statements
- Scripts and Tools: Windows environment now supports distinguishing 
ConfigNode, DataNode, and Cli by window name
...
## Bugs
- Optimize the error message when a NullPointerException (NPE) occurs due to a 
timeout when dropping a database.
- Add logs for notifyLeaderReady, notifyLeaderChanged, and procedure worker.
- Add compatibility handling for existing erroneous data during file merging.
- Fix the deadlock issue caused by flushing empty files during querying.
- Fix the issue where Ratis becomes unresponsive during read, write, and delete 
operations.
- Fix the concurrent bug in load and merge operations.
- Fix the issue where the system's compression ratio is recorded as a negative 
number in the file for certain scenarios.
- Fix the ConcurrentModificationException issue during memory estimation for 
merge tasks.
- Fix potential deadlocks that may occur when writing, automatically creating, 
and deleting databases concurrently.
...

The full release note is available at:
https://raw.githubusercontent.com/apache/iotdb/rc/1.3.2/RELEASE_NOTES.md

The release is available for download at:
http://iotdb.apache.org/Download

Maven artifacts for JDBC driver, session SDK, TsFile SDK can be found at:
https://search.maven.org/search?q=3Dg:org.apache.iotdb


Regards,
The Apache IoTDB team

[RESULT][VOTE] Release Apache IoTDB 1.3.2

2024-06-30 Thread Haonan Hou

Hi all,

The release is approved by accepting 3 +1 votes:

3 from PMC members,
Qingxin Feng,
Christofer Dutz,
Jialin Qiao.

Vote threads:

https://lists.apache.org/thread/svdkp2gqy960t9svpmygp4hdjcdnoswo


Thanks,
Haonan Hou

Re: [VOTE] Apache IoTDB 1.3.2 RC1 release

2024-06-29 Thread Jialin Qiao

Hi,

+1 (binding)

The source release:
apache headers [ok]
signatures and hashes [ok]
LICENSE and NOTICE [ok]
no jar files [ok]

The binary distribution:
version number in CLI [ok]
signatures and hashes [ok]
start in mac, jdk8 [ok]

Thanks,
Jialin Qiao

Christofer Dutz  于2024年6月28日周五 19:15写道：
>
> +1 (binding)
>
> Chris
>
> [OK] Download all staged artifacts under the url specified in the release 
> vote email.
> [OK] Verify the signature is correct.
> [OK] Check if the signature references an Apache email address.
> [OK] Verify the SHA512 hashes.
> [OK] Unzip the archive.
> [OK] Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in 
> the extracted source bundle.
> [MINOR] Verify the content of LICENSE, NOTICE, README, RELEASE_NOTES files in 
> the extracted source bundle.
>
>   *   RELEASE_NOTES hasn’t been updated for quite some time.
>   *   NOTICE: Could possibly need an update on the Apache Commons Collections 
> section, as we’re most probably using versions released after 2019.
> [MINOR] Run RAT externally to ensure there are no surprises.
>
>   *   Integration-test module contains 3 archives, 1 of them I couldn’t find 
> the sources to (TriggerFireTimesCounter)
> [MINOR] Search for SNAPSHOT references
>
>   *   It seems the following projects were not part of the release and 
> therefore their version was not updated:
>  *   example/client-cpp-example
>  *   iotdb-client/client-cpp
> [OK] Search for Copyright references, and if they are in headers, make sure 
> these files containing them are mentioned in the LICENSE file.
>
>
> Von: 冯 庆新 
> Datum: Donnerstag, 27. Juni 2024 um 15:05
> An: dev@iotdb.apache.org 
> Betreff: 回复: [VOTE] Apache IoTDB 1.3.2 RC1 release
> Hello  all :
> +1
>
> The binary distribution:
> version number in CLI [ok]
> start in CentOS7, jdk11 [ok]
> performance verification passed:  [ok]
> ### Server Configurations ###//
> CPU=16
> Memory=32G
> Disk=1.8T HDD
> Ethernet=1000Mbit
> ///
> ### IoTDB Configurations ###///
> MAX_HEAP_SIZE="20G" in datanode-env.sh
> ///
> ### Client Mode ###
> Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET
> CLIENT_NUMBER=10
> GROUP_NUMBER=10
> DEVICE_NUMBER=50
> SENSOR_NUMBER=500
> BATCH_SIZE_PER_WRITE=10
> LOOP=86400
> ///
>
> ### Test Result ###
> 1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points；
> Non-Aligned Timseries:
> Throughput(points/s): 17,009,956.18  Cost Time(s): 1,269.84
> Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48
> Aligned Timseries:
> Throughput(points/s): 18,141,900.23  Cost Time(s): 1,190.61
> Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48
> ///
>
>
> Avg: Average time cost of all ingestion operations. [ms]
> MiddleAvg: Average time cost of ingestion operations without 5% head and 
> tail. [ms]
> Min: Min single operation time cost of all ingestion operations. [ms]
> Max: Max single operation  time cost of all ingestion operations. [ms]
> These results are tested with 
> iot-benchmark(https://github.com/thulab/iot-benchmark)
> That's all.
> Please feel free to contact me if you have any questions
> Thank you.
> Best Regards!
> Qingxin Feng
>
> 从 Windows 版邮件发送
>
> 
> 发件人: Haonan Hou 
> 发送时间: Thursday, June 27, 2024 5:55:36 PM
> 收件人: dev@iotdb.apache.org 
> 主题: [VOTE] Apache IoTDB 1.3.2 RC1 release
>
> Hi all,
>
> Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
> on accepting it for release. All Maven artifacts are available under [1].
> Voting will be open for 72hr.
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
>
> Release tag: v1.3.2
> Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885
>
> Before voting +1, PMC members are required to download
> the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also
> verifying that the package meets the requirements of the ASF policy
> on releases. [3]
>
> You can achieve the above by following [4].
>
> [ ] +1 accept (indicate what you validated - e.g. performed the
> non-RM items in [4])
> [ ] -1 reject (explanation required)
>
>
> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
> [2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4] 
>

AW: [VOTE] Apache IoTDB 1.3.2 RC1 release

2024-06-28 Thread Christofer Dutz

+1 (binding)

Chris

[OK] Download all staged artifacts under the url specified in the release vote 
email.
[OK] Verify the signature is correct.
[OK] Check if the signature references an Apache email address.
[OK] Verify the SHA512 hashes.
[OK] Unzip the archive.
[OK] Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in 
the extracted source bundle.
[MINOR] Verify the content of LICENSE, NOTICE, README, RELEASE_NOTES files in 
the extracted source bundle.

  *   RELEASE_NOTES hasn’t been updated for quite some time.
  *   NOTICE: Could possibly need an update on the Apache Commons Collections 
section, as we’re most probably using versions released after 2019.
[MINOR] Run RAT externally to ensure there are no surprises.

  *   Integration-test module contains 3 archives, 1 of them I couldn’t find 
the sources to (TriggerFireTimesCounter)
[MINOR] Search for SNAPSHOT references

  *   It seems the following projects were not part of the release and 
therefore their version was not updated:
 *   example/client-cpp-example
 *   iotdb-client/client-cpp
[OK] Search for Copyright references, and if they are in headers, make sure 
these files containing them are mentioned in the LICENSE file.


Von: 冯 庆新 
Datum: Donnerstag, 27. Juni 2024 um 15:05
An: dev@iotdb.apache.org 
Betreff: 回复: [VOTE] Apache IoTDB 1.3.2 RC1 release
Hello  all :
+1

The binary distribution:
version number in CLI [ok]
start in CentOS7, jdk11 [ok]
performance verification passed:  [ok]
### Server Configurations ###//
CPU=16
Memory=32G
Disk=1.8T HDD
Ethernet=1000Mbit
///
### IoTDB Configurations ###///
MAX_HEAP_SIZE="20G" in datanode-env.sh
///
### Client Mode ###
Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET
CLIENT_NUMBER=10
GROUP_NUMBER=10
DEVICE_NUMBER=50
SENSOR_NUMBER=500
BATCH_SIZE_PER_WRITE=10
LOOP=86400
///

### Test Result ###
1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points；
Non-Aligned Timseries:
Throughput(points/s): 17,009,956.18  Cost Time(s): 1,269.84
Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48
Aligned Timseries:
Throughput(points/s): 18,141,900.23  Cost Time(s): 1,190.61
Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48
///


Avg: Average time cost of all ingestion operations. [ms]
MiddleAvg: Average time cost of ingestion operations without 5% head and tail. 
[ms]
Min: Min single operation time cost of all ingestion operations. [ms]
Max: Max single operation  time cost of all ingestion operations. [ms]
These results are tested with 
iot-benchmark(https://github.com/thulab/iot-benchmark)
That's all.
Please feel free to contact me if you have any questions
Thank you.
Best Regards!
Qingxin Feng

从 Windows 版邮件发送


发件人: Haonan Hou 
发送时间: Thursday, June 27, 2024 5:55:36 PM
收件人: dev@iotdb.apache.org 
主题: [VOTE] Apache IoTDB 1.3.2 RC1 release

Hi all,

Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
on accepting it for release. All Maven artifacts are available under [1].
Voting will be open for 72hr.
A minimum of 3 binding +1 votes and more binding +1 than binding -1
are required to pass.

Release tag: v1.3.2
Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885

Before voting +1, PMC members are required to download
the signed source code package, compile it as provided, and test
the resulting executable on their own platform, along with also
verifying that the package meets the requirements of the ASF policy
on releases. [3]

You can achieve the above by following [4].

[ ] +1 accept (indicate what you validated - e.g. performed the
non-RM items in [4])
[ ] -1 reject (explanation required)


[1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
[2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
[3] https://www.apache.org/dev/release.html#approving-a-release
[4] 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release

Best,

Haonan Hou

回复: [VOTE] Apache IoTDB 1.3.2 RC1 release

2024-06-27 Thread 冯庆新

Hello  all :
+1

The binary distribution:
version number in CLI [ok]
start in CentOS7, jdk11 [ok]
performance verification passed:  [ok]
### Server Configurations ###//
CPU=16
Memory=32G
Disk=1.8T HDD
Ethernet=1000Mbit
///
### IoTDB Configurations ###///
MAX_HEAP_SIZE="20G" in datanode-env.sh
///
### Client Mode ###
Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET
CLIENT_NUMBER=10
GROUP_NUMBER=10
DEVICE_NUMBER=50
SENSOR_NUMBER=500
BATCH_SIZE_PER_WRITE=10
LOOP=86400
///

### Test Result ###
1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points；
Non-Aligned Timseries:
Throughput(points/s): 17,009,956.18  Cost Time(s): 1,269.84
Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48
Aligned Timseries:
Throughput(points/s): 18,141,900.23  Cost Time(s): 1,190.61
Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48
///


Avg: Average time cost of all ingestion operations. [ms]
MiddleAvg: Average time cost of ingestion operations without 5% head and tail. 
[ms]
Min: Min single operation time cost of all ingestion operations. [ms]
Max: Max single operation  time cost of all ingestion operations. [ms]
These results are tested with 
iot-benchmark(https://github.com/thulab/iot-benchmark)
That's all.
Please feel free to contact me if you have any questions
Thank you.
Best Regards!
Qingxin Feng

从 Windows 版邮件发送


发件人: Haonan Hou 
发送时间: Thursday, June 27, 2024 5:55:36 PM
收件人: dev@iotdb.apache.org 
主题: [VOTE] Apache IoTDB 1.3.2 RC1 release

Hi all,

Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
on accepting it for release. All Maven artifacts are available under [1].
Voting will be open for 72hr.
A minimum of 3 binding +1 votes and more binding +1 than binding -1
are required to pass.

Release tag: v1.3.2
Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885

Before voting +1, PMC members are required to download
the signed source code package, compile it as provided, and test
the resulting executable on their own platform, along with also
verifying that the package meets the requirements of the ASF policy
on releases. [3]

You can achieve the above by following [4].

[ ] +1 accept (indicate what you validated - e.g. performed the
non-RM items in [4])
[ ] -1 reject (explanation required)


[1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
[2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
[3] https://www.apache.org/dev/release.html#approving-a-release
[4] 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release

Best,

Haonan Hou

Re: Allow the client with python 3.6 to use IoTDB

2024-06-27 Thread Haonan Hou

Ok, this PR has been merged.

Best,
Haonan

On 2024/06/24 14:19:36 Haonan Hou wrote:
> Hi,
> 
> The current python client of IoTDB requires the python version >= 3.7.
> However, I met some applications running on some old platforms which only 
> support python 3.6, and I found it is easy to make the adaption. Here is the 
> PR[1].
> 
> In this PR, I also added a CI with python 3.6 and it can work correctly. It 
> would be good if I can get some approval.
> 
> Haonan Hou
> 
> [1] https://github.com/apache/iotdb/pull/12792
>

[VOTE] Apache IoTDB 1.3.2 RC1 release

2024-06-27 Thread Haonan Hou

Hi all,

Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
on accepting it for release. All Maven artifacts are available under [1].
Voting will be open for 72hr.
A minimum of 3 binding +1 votes and more binding +1 than binding -1
are required to pass.

Release tag: v1.3.2
Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885

Before voting +1, PMC members are required to download
the signed source code package, compile it as provided, and test
the resulting executable on their own platform, along with also
verifying that the package meets the requirements of the ASF policy
on releases. [3]

You can achieve the above by following [4].

[ ] +1 accept (indicate what you validated - e.g. performed the
non-RM items in [4])
[ ] -1 reject (explanation required)


[1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
[2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
[3] https://www.apache.org/dev/release.html#approving-a-release
[4] 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release

Best,

Haonan Hou

Integrating UDF Library and Grafana (conf update) in Docker

2024-06-26 Thread Stephen Lawrence

Hi,

After some successful experiments I would like to enable the optional 
data quality UDF Library and support Grafana in the COVESA Central 
Data Service Playground (CDSP). Functions like the LTTB Triangle 
sampling are very useful in automotive use-cases.

I have some questions related to UDF integration and IoTDB conf 
changes in a Docker deployment. I am currently using IoTDB v1.2.2

UDF LIBRARY
I was hoping to keep things simple for a while and not have a
CDSP specific IoTDB Dockerfile, but I can create one to add the
Library JAR to the image. Registration is a little more tricky to 
automate reliably as it is a one time process. I was thinking to 
include the library JAR by default and ask users to run the 
registration script themselves if they wish to use the library 
functions. That separates any registration issues from the build 
process, making debugging for non-experts easier.

However, if anyone has a simple method to do the registration in
the container that they have found reliable I would be happy to 
hear about it?
 
Q1) I noticed if the library functions are registered but the JAR
is not available on IoTDB restart, the exception handling causes
IoTDB to not start. Is that expected behaviour?

CONF CHANGES (enabling REST for Grafana)
Q2) From reading the IoTDB documentation it appears REST
can not be enabled through runtime environment variables,
e.g. docker env, and instead must be done by modifying 
the key in conf/iotdb-datanode.properties. Is that correct?

Q3) The IoTDB docker documentation suggests sharing a conf
directory, like as with log/data, into the container if you 
wish to change conf files. I found that on startup IoTDB 
does not create a default conf environment if it is given 
an empty conf directory. As a result IoTDB does not start.
Is that expected behaviour?

As I will probably need to create an IoTDB dockerfile to
integrate the UDF Library JAR I may modify the conf file
there instead.

Best Wishes,

Steve

Allow the client with python 3.6 to use IoTDB

2024-06-24 Thread Haonan Hou

Hi,

The current python client of IoTDB requires the python version >= 3.7.
However, I met some applications running on some old platforms which only 
support python 3.6, and I found it is easy to make the adaption. Here is the 
PR[1].

In this PR, I also added a CI with python 3.6 and it can work correctly. It 
would be good if I can get some approval.

Haonan Hou

[1] https://github.com/apache/iotdb/pull/12792

RE: Introducing Apache IoTDB use in COVESA Central Data Service Playground

2024-06-21 Thread Stephen Lawrence

Hi,

Thank you both, for the encouragement and the offers of support.

> -Original Message-
> From: Jialin Qiao 
> Sent: Tuesday, June 18, 2024 9:50 AM
> To: dev@iotdb.apache.org
> Subject: Re: Introducing Apache IoTDB use in COVESA Central Data Service
> Playground
> 
> Hi,
> 
> Glad to see the integration!
> Welcome and feel free to ask :D
> 
> Jialin Qiao
> 
> Pengcheng Zheng  于2024年6月15日周六 04:38写道：
> >
> > Hi Stephen,
> >
> > Very exciting project!
> >
> > Thanks for sharing the details of COVESA/CDSP. I remember that you asked
> > some questions in Slack several months ago, and seems that you have made
> > great progress ;-)

Yes, originally I had some questions about running IoTDB on embedded h/w and 
the answers helped me commit to using IoTDB in the Playground.

BTW as part of the project I have done some work integrating IoTDB to other
technology:

1) COVESA and the W3C co-developed a REST-like get/set/sub API for accessing 
vehicle data in-vehicle called VISS. VISS has a reference implementation 
maintained in the VISSR project that supported SQLite and Redis as data
store backends. I upstreamed support for VISSR to also support IoTDB.

2) The Playground includes a bridge for streaming real car data from the 
RemotiveLabs virtual signal platform in the cloud into IoTDB. It has a
very simple implementation currently for the IoTDB connection but it has
no requirements on any other part of the Playground and could be a useful
example to anyone wanting to using RemotiveLabs with IoTDB. 

> >
> > IoTDB has already proved its excellence in the scenario of connected
> > vehicles before, and the key questions you've listed in the
> > presentation happen to be exactly what IoTDB & TsFile are addressing,
> such
> > as data syncing between all touchpoints/ bidirectional sync/ permissions
> &
> > privacy/ model updating/ subscription, etc.
> >
> > Also your ideas of the deployment in Docker and edge-cloud sync (if my
> > memory serves me right..) shall benefit a lot from IoTDB.

Yes, IoTDB has many features that made it attractive for me to use it in
the Playground. I am sure I will have questions as the project progresses
Especially, I would like to leverage the greater IoTDB experience here to
check we are using it effectively. So please excuse the odd basic question.

Over the last year I have looked through the presentations on the IoTDB website
and occasionally checked back for updates. Including the Chinese language 
ones where I have been able to translate. I remember seeing some mentions 
of IoTDB implementation in automotive but not many details. If I remember
correctly there was a major instance for a telematics use-case. I also saw 
that Table 4 of the IoTDB paper from SIGMOD 2023 listed three automotive 
deployments.

If you have details of IoTDB automotive deployments I would be very interested.

> >
> > Hopefully IoTDB could support the CDSP project well and help transform
> the
> > "Playground" to POC - and finally from theory to practice.

I think it will. Put simply the Playground is a toolset of Lego pieces.
These pieces are intended to be connected in a range of ways to create
PoCs or examples of logical concepts large and small and prove them out
in implementation. Hence "Playground". This is not a project to make a 
specific middleware 'product' or address a specific singular use-case.
At least in the current project setup.

The open source community aspect of COVESA means for effective collaboration
we need to also support diversity in architecture. In-vehicle architecture is
very complex and OEM A will make different choices to OEM B.

So, production will be a second step for individual companies to take
with their partners suited to their architectural and purchasing decisions.
That said, of course the advantage of using proven technology like IoTDB 
is that the path to production is there.

> >
> > Looking forward to your updates!

Thanks.

Best Wishes,

Stephen Lawrence

> >
> >
> > Best Wishes,
> > Pengcheng Zheng
> > Timecho Europe

[snip]

Re: Apply for a development platform account

2024-06-21 Thread Yuan Tian

Hi yuyong,

Welcome to IoTDB Community, I've already approved your jira account
application. It didn't work?


Best regards,
--
Yuan Tian



On Fri, Jun 21, 2024 at 4:50 PM 喻勇 <1471335...@qq.com.invalid> wrote:

> Dear IoTDB PMC:
>
> Greetings!
>
> I am yuyong,I am currently a graduate student at Chongqing University of
> Posts and Telecommunications, and I am currently very interested in IoTDB
> and would like to contribute my part in it. I am passionate about
> participating in the open source community.
>
> After studying the documentation and code base of IoTDB in detail, I found
> that the project is highly compatible with my technical background and
> interests. I believe that through my participation, I can bring some new
> ideas and solutions to the project and help it develop further.
>
> To prove my identity and willingness to contribute, I am providing the
> following information:
>
> My JIRA platform ID is: FearfulTomcat27
> My email address is: 1471335...@qq.com
>
> I plan to actively contribute to IoTDB by submitting bug fixes, optimizing
> code, writing documentation, or participating in discussions. I am aware
> that as a developer, I need to adhere to the project's development
> specifications and code quality standards, and respect and uphold the
> values of the open source community.
>
> If you need to know more about my technical background or experience,
> please feel free to check out my personal website or GitHub page:
> https://github.com/FearfulTomcat27
>
> Very much looking forward to being a part of IoTDB and working with other
> developers to make the project a success. Please feel free to contact me
> for any additional information or further discussion.
>
> Thank you again for taking the time to read my application. Looking
> forward to your reply!
>
> Best wishes!
>
> Name:yuyong
> Phone:15892323376
> Email:1471335...@qq.com

Apply for a development platform account

2024-06-21 Thread 喻勇

Dear IoTDB PMC:
 
Greetings!
 
I am yuyong,I am currently a graduate student at Chongqing University of Posts 
and Telecommunications, and I am currently very interested in IoTDB and would 
like to contribute my part in it. I am passionate about participating in the 
open source community.

After studying the documentation and code base of IoTDB in detail, I found that 
the project is highly compatible with my technical background and interests. I 
believe that through my participation, I can bring some new ideas and solutions 
to the project and help it develop further.
 
To prove my identity and willingness to contribute, I am providing the 
following information:
 
My JIRA platform ID is: FearfulTomcat27
My email address is: 1471335...@qq.com

I plan to actively contribute to IoTDB by submitting bug fixes, optimizing 
code, writing documentation, or participating in discussions. I am aware that 
as a developer, I need to adhere to the project's development specifications 
and code quality standards, and respect and uphold the values of the open 
source community.
 
If you need to know more about my technical background or experience, please 
feel free to check out my personal website or GitHub 
page:https://github.com/FearfulTomcat27
 
Very much looking forward to being a part of IoTDB and working with other 
developers to make the project a success. Please feel free to contact me for 
any additional information or further discussion.
 
Thank you again for taking the time to read my application. Looking forward to 
your reply!
 
Best wishes!
 
Name:yuyong
Phone:15892323376
Email:1471335...@qq.com

答复: Recover/Reset Root Password?

2024-06-18 Thread Wang Critas

In addition, in versions after 1.3.1, if the file and the files in snapshot are 
deleted, restarting will automatically generate a file containing the default 
password, without the need to find the file and put it in it.

And you can also take a look at the relevant source code

发件人: Wang Critas 
日期: 星期二, 2024年6月18日 17:21
收件人: dev@iotdb.apache.org 
主题: 答复: Recover/Reset Root Password?
Hi Trevor
The method you mentioned is feasible, but it is important to note that the 
corresponding IoTDB version of the file needs to be matched. Additionally, the 
cluster needs to be restarted.
This operation is quite dangerous. It is recommended to make a backup in 
advance, and the best practice is to keep the password properly

发件人: Trevor Hart 
日期: 星期二, 2024年6月18日 16:33
收件人: Dev 
主题: Recover/Reset Root Password?
Hello



Is there a process to recover or reset the root user password? For example, if 
a customer forgets their root user password.



Is it possible to replace the "root.profile" file under 
data/confignode/system/users with the default? Would this work?


Thanks

Trevor Hart

答复: Recover/Reset Root Password?

2024-06-18 Thread Wang Critas

Hi Trevor
The method you mentioned is feasible, but it is important to note that the 
corresponding IoTDB version of the file needs to be matched. Additionally, the 
cluster needs to be restarted.
This operation is quite dangerous. It is recommended to make a backup in 
advance, and the best practice is to keep the password properly

发件人: Trevor Hart 
日期: 星期二, 2024年6月18日 16:33
收件人: Dev 
主题: Recover/Reset Root Password?
Hello



Is there a process to recover or reset the root user password? For example, if 
a customer forgets their root user password.



Is it possible to replace the "root.profile" file under 
data/confignode/system/users with the default? Would this work?


Thanks

Trevor Hart

Re: Introducing Apache IoTDB use in COVESA Central Data Service Playground

2024-06-18 Thread Jialin Qiao

Hi,

Glad to see the integration!
Welcome and feel free to ask :D

Jialin Qiao

Pengcheng Zheng  于2024年6月15日周六 04:38写道：
>
> Hi Stephen,
>
> Very exciting project!
>
> Thanks for sharing the details of COVESA/CDSP. I remember that you asked
> some questions in Slack several months ago, and seems that you have made
> great progress ;-)
>
> IoTDB has already proved its excellence in the scenario of connected
> vehicles before, and the key questions you've listed in the
> presentation happen to be exactly what IoTDB & TsFile are addressing, such
> as data syncing between all touchpoints/ bidirectional sync/ permissions &
> privacy/ model updating/ subscription, etc.
>
> Also your ideas of the deployment in Docker and edge-cloud sync (if my
> memory serves me right..) shall benefit a lot from IoTDB.
>
> Hopefully IoTDB could support the CDSP project well and help transform the
> "Playground" to POC - and finally from theory to practice.
>
> Looking forward to your updates!
>
>
> Best Wishes,
> Pengcheng Zheng
> Timecho Europe
>
> Am Fr., 14. Juni 2024 um 21:12 Uhr schrieb Stephen Lawrence <
> stephen.lawre...@renesas.com>:
>
> > Hi,
> >
> > As suggested in the wiki I am introducing myself and a new OSS project
> > using Apache IoTDB.
> >
> > My name is Stephen Lawrence from Renesas Electronics and I work in the
> > COVESA[1] open automotive alliance. COVESA is focused on developing common
> > approaches and technologies for connected vehicles, such as the VSS data
> > model [2]. I lead the COVESA Data Architecture and Infrastructure group and
> > am a Board member.
> >
> > We recently released the first version of the OSS COVESA Central Data
> > Service Playground (CDSP)[3] which uses Apache IoTDB as a data
> > store/server. The Playground is a neutral, open playground for
> > investigating data services in-vehicle and off-board in the context of
> > data-centric architectures. Importantly in community terms it provides a
> > means to publish and collaborate on such work in the open.
> >
> > Constructed as a "project of projects" this first release was focused on
> > creating the first version of the "toolkit". Now we pivot to improving it,
> > creating examples for its use and investigating industry issues. Currently
> > we are focused on Docker deployment for ease of development and connections
> > to other technology. Deployment on Automotive h/w is part of the future
> > roadmap. The contributors include various large vehicle OEMs.
> >
> > At a recent COVESA Conference I demonstrated this first release. The
> > online documentation is a work in progress but contains a summary of the
> > What/Why/How of the project. If you are interested in more details I led a
> > track of sessions about the Playground at the conference, the presentation
> > for which you can find here [4].
> >
> > I know I am going to have an increasing number of questions in the coming
> > months, so I wanted to first introduce myself and the project. I hope to be
> > able to contribute back to the IoTDB project as well.
> >
> > Apologies for the corporately expanded reference URLs in plain text:
> > [1] http://www.covesa.global/
> > [2] https://covesa.global/project/vehicle-signal-specification/
> > [3] https://github.com/COVESA/cdsp
> > [4]
> > https://wiki.covesa.global/download/attachments/98271360/Central_Data_Service_Playground_track_v2.pptx?version=1=1714486822982=v2
> >
> > Best Wishes,
> >
> > Stephen Lawrence
> > COVESA Data Architecture Lead and Board Member
> > (Renesas Electronics)
> >

Recover/Reset Root Password?

2024-06-18 Thread Trevor Hart

Hello



Is there a process to recover or reset the root user password? For example, if 
a customer forgets their root user password.



Is it possible to replace the "root.profile" file under 
data/confignode/system/users with the default? Would this work?


Thanks 

Trevor Hart

Introducing Apache IoTDB use in COVESA Central Data Service Playground

2024-06-14 Thread Stephen Lawrence

Hi,

As suggested in the wiki I am introducing myself and a new OSS project using
Apache IoTDB.

My name is Stephen Lawrence from Renesas Electronics and I work in the
COVESA[1] open automotive alliance. COVESA is focused on developing common
approaches and technologies for connected vehicles, such as the VSS data model
[2]. I lead the COVESA Data Architecture and Infrastructure group and am a
Board member.

We recently released the first version of the OSS COVESA Central Data Service
Playground (CDSP)[3] which uses Apache IoTDB as a data store/server. The
Playground is a neutral, open playground for investigating data services
in-vehicle and off-board in the context of data-centric architectures.
Importantly in community terms it provides a means to publish and collaborate
on such work in the open.

Constructed as a "project of projects" this first release was focused on
creating the first version of the "toolkit". Now we pivot to improving it,
creating examples for its use and investigating industry issues. Currently we
are focused on Docker deployment for ease of development and connections to
other technology. Deployment on Automotive h/w is part of the future roadmap.
The contributors include various large vehicle OEMs.

At a recent COVESA Conference I demonstrated this first release. The online
documentation is a work in progress but contains a summary of the What/Why/How
of the project. If you are interested in more details I led a track of sessions
about the Playground at the conference, the presentation for which you can find
here [4].

I know I am going to have an increasing number of questions in the coming
months, so I wanted to first introduce myself and the project. I hope to be
able to contribute back to the IoTDB project as well.

Apologies for the corporately expanded reference URLs in plain text:
[1] http://www.covesa.global/
[2] https://covesa.global/project/vehicle-signal-specification/
[3] https://github.com/COVESA/cdsp
[4]
https://wiki.covesa.global/download/attachments/98271360/Central_Data_Service_Playground_track_v2.pptx?version=1=1714486822982=v2

Best Wishes,

Stephen Lawrence
COVESA Data Architecture Lead and Board Member
(Renesas Electronics)

Re: IoTDB quarter report (2024 Q2)

2024-06-12 Thread Xiangdong Huang

> And I have something more to add:

Cool. I will add these activities.

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Pengcheng Zheng  于2024年6月12日周三 16:16写道：
>
> Hi Xiangdong,
>
> Great ;)
>
> And I have something more to add:
> - on 2024-04-09, IoTDB was presented at the JavaLand Summit in Germany. The
> talk was given by Christofer Dutz. [1]
> - on 2024-06-03, IoTDB was presented in 2 talks at CommunityOverCode EU.
> The talks were given by Christofer Dutz and Julian Feinauer. [2]
>
>
> [1] https://meine.doag.org/events/javaland/2024/agenda/#agendaId.3893
> [2] https://eu.communityovercode.org/program/
>
>
> Best regards,
> Pengcheng
>
>
> Am Mi., 12. Juni 2024 um 03:11 Uhr schrieb Xiangdong Huang <
> saint...@gmail.com>:
>
> > Hi all,
> >
> > This is the draft of IoTDB quarter report
> >
> > ## Description:
> > The mission of Apache IoTDB is the creation and maintenance of software
> > related
> > to an IoT native database with high performance for data management and
> > analysis
> >
> > ## Project Status:
> > Current project status: Ongoing with high activity.
> > Issues for the board:
> > - there are some trademark issues but the community has resolved them.
> >
> > ## Membership Data:
> > Apache IoTDB was founded 2020-09-16 (4 years ago)
> > There are currently 63 committers and 29 PMC members in this project.
> > The Committer-to-PMC ratio is roughly 2:1.
> >
> > Community changes, past quarter:
> > - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
> > - William Song was added as committer on 2024-03-22
> > - Caiyin Yang was added as committer on 2024-06-12
> >
> > ## Project Activity:
> > - Recent releases:
> > IOTDB-1.3.2 is on the way.
> > IOTDB-1.3.1 was released on 2024-04-22.
> > IOTDB-1.3.0 was released on 2024-01-01.
> > IOTDB-1.2.2 was released on 2023-10-15.
> >
> > - Main work of project:
> > * We are working on the table model providing data to users in a table view
> > in the same way as relational databases which will greatly reduce the
> > learning curve of IoTDB.
> > * Collaborate with the Apache StreamPipes community to improve the IoTDB
> > plugin
> >  and explore other means on integration.
> > * New query optimization rules are added to greatly improve the query
> > performance of IoTDB, like PredicatePushDown and AggregatePushDown.
> > * Explain Analyze tool which can be used to analyze specific query
> > performance better and is a good tool for troubleshooting performance
> > problems
> >
> >
> > ## Community Health:
> > Overall community health is good.
> > - Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
> > - the mailing list traffic increases (226 emails compared to 76)
> >
> > Best,
> > ---
> > Xiangdong Huang
> >

Re: IoTDB quarter report (2024 Q2)

2024-06-12 Thread Pengcheng Zheng

Hi Xiangdong,

Great ;)

And I have something more to add:
- on 2024-04-09, IoTDB was presented at the JavaLand Summit in Germany. The
talk was given by Christofer Dutz. [1]
- on 2024-06-03, IoTDB was presented in 2 talks at CommunityOverCode EU.
The talks were given by Christofer Dutz and Julian Feinauer. [2]


[1] https://meine.doag.org/events/javaland/2024/agenda/#agendaId.3893
[2] https://eu.communityovercode.org/program/


Best regards,
Pengcheng


Am Mi., 12. Juni 2024 um 03:11 Uhr schrieb Xiangdong Huang <
saint...@gmail.com>:

> Hi all,
>
> This is the draft of IoTDB quarter report
>
> ## Description:
> The mission of Apache IoTDB is the creation and maintenance of software
> related
> to an IoT native database with high performance for data management and
> analysis
>
> ## Project Status:
> Current project status: Ongoing with high activity.
> Issues for the board:
> - there are some trademark issues but the community has resolved them.
>
> ## Membership Data:
> Apache IoTDB was founded 2020-09-16 (4 years ago)
> There are currently 63 committers and 29 PMC members in this project.
> The Committer-to-PMC ratio is roughly 2:1.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
> - William Song was added as committer on 2024-03-22
> - Caiyin Yang was added as committer on 2024-06-12
>
> ## Project Activity:
> - Recent releases:
> IOTDB-1.3.2 is on the way.
> IOTDB-1.3.1 was released on 2024-04-22.
> IOTDB-1.3.0 was released on 2024-01-01.
> IOTDB-1.2.2 was released on 2023-10-15.
>
> - Main work of project:
> * We are working on the table model providing data to users in a table view
> in the same way as relational databases which will greatly reduce the
> learning curve of IoTDB.
> * Collaborate with the Apache StreamPipes community to improve the IoTDB
> plugin
>  and explore other means on integration.
> * New query optimization rules are added to greatly improve the query
> performance of IoTDB, like PredicatePushDown and AggregatePushDown.
> * Explain Analyze tool which can be used to analyze specific query
> performance better and is a good tool for troubleshooting performance
> problems
>
>
> ## Community Health:
> Overall community health is good.
> - Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
> - the mailing list traffic increases (226 emails compared to 76)
>
> Best,
> ---
> Xiangdong Huang
>

Introduction of Enhanced Load Functionality with Multi-Drive Support

2024-06-11 Thread ITAMI SHO

Hello everyone,

My name is Itami Sho, a contributor to IoTDB, and I am excited to share with 
you a recent enhancement I have implemented in our Load functionality. This 
optimization introduces support for multiple drive letters during the second 
phase of Load functionality to store TsFile PieceNode.

Previously, the Load feature only supported PieceNode storing on a single drive 
letter. However, in environments with multi-drive clusters, this limitation 
could lead to performance bottlenecks. To address this issue, I have optimized 
the Load functionality to support multiple drive letters, significantly 
improving overall performance in such scenarios.

You can review the details of my implementation in the following PR:
https://github.com/apache/iotdb/pull/12675

Best regards,
Itami Sho

Re: ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.

2024-06-11 Thread Trevor Hart

Ah that is great to hear! Thank you!




Thanks 

Trevor Hart




 On Wed, 12 Jun 2024 13:50:43 +1200 Jialin Qiao  
wrote ---



Hi, 
 
We just changed the default infer type of integer and floating to 
DOUBLE [1], it will be released in 1.3.2. 
 
For <=1.3.1, you could explicitly set this to DOUBLE. 
 
[1] https://github.com/apache/iotdb/pull/12223 
 
Jialin Qiao 
 
Trevor Hart  于2024年6月12日周三 06:41写道： 
> 
> Hello Team 
> 
> 
> 
> I have a customer using my application with IoTDB 1.3.0 in the backend. 
> 
> 
> 
> Auto create schema is enabled as there is no fixed template. Multiple devices 
> are pushing data to IotDB. 
> 
> 
> 
> I have seen some ALIGN BY DEVICE queries fail with the error in the subject 
> ie 
> 
> 
> 
> ALIGN BY DEVICE: the data types of the same measurement column should be the 
> same across devices. 
> 
> 
> 
> The "show timeseries" shows that from one device the value has been inferred 
> as FLOAT and another as DOUBLE. 
> 
> 
> Per the documentation it is suggested that the default 
> "floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why 
> am I getting FLOAT? Is the default "floating_string_infer_type" not being 
> honoured? 
> 
> 
> 
> Currently in the config file "floating_string_infer_type" is commented out. 
> Should I explicitly set this to DOUBLE? 
> 
> 
> Thanks 
> 
> Trevor Hart

Re: ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.

2024-06-11 Thread Jialin Qiao

Hi,

We just changed the default infer type of integer and floating to
DOUBLE [1], it will be released in 1.3.2.

For <=1.3.1, you could explicitly set this to DOUBLE.

[1] https://github.com/apache/iotdb/pull/12223

Jialin Qiao

Trevor Hart  于2024年6月12日周三 06:41写道：
>
> Hello Team
>
>
>
> I have a customer using my application with IoTDB 1.3.0 in the backend.
>
>
>
> Auto create schema is enabled as there is no fixed template. Multiple devices 
> are pushing data to IotDB.
>
>
>
> I have seen some ALIGN BY DEVICE queries fail with the error in the subject ie
>
>
>
> ALIGN BY DEVICE: the data types of the same measurement column should be the 
> same across devices.
>
>
>
> The "show timeseries" shows that from one device the value has been inferred 
> as FLOAT and another as DOUBLE.
>
>
> Per the documentation it is suggested that the default 
> "floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why 
> am I getting FLOAT? Is the default "floating_string_infer_type" not being 
> honoured?
>
>
>
> Currently in the config file "floating_string_infer_type" is commented out. 
> Should I explicitly set this to DOUBLE?
>
>
> Thanks
>
> Trevor Hart

Re: IoTDB quarter report (2024 Q2)

2024-06-11 Thread Jialin Qiao

Looks good!

Jialin Qiao

Xiangdong Huang  于2024年6月12日周三 09:11写道：
>
> Hi all,
>
> This is the draft of IoTDB quarter report
>
> ## Description:
> The mission of Apache IoTDB is the creation and maintenance of software 
> related
> to an IoT native database with high performance for data management and 
> analysis
>
> ## Project Status:
> Current project status: Ongoing with high activity.
> Issues for the board:
> - there are some trademark issues but the community has resolved them.
>
> ## Membership Data:
> Apache IoTDB was founded 2020-09-16 (4 years ago)
> There are currently 63 committers and 29 PMC members in this project.
> The Committer-to-PMC ratio is roughly 2:1.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
> - William Song was added as committer on 2024-03-22
> - Caiyin Yang was added as committer on 2024-06-12
>
> ## Project Activity:
> - Recent releases:
> IOTDB-1.3.2 is on the way.
> IOTDB-1.3.1 was released on 2024-04-22.
> IOTDB-1.3.0 was released on 2024-01-01.
> IOTDB-1.2.2 was released on 2023-10-15.
>
> - Main work of project:
> * We are working on the table model providing data to users in a table view
> in the same way as relational databases which will greatly reduce the
> learning curve of IoTDB.
> * Collaborate with the Apache StreamPipes community to improve the IoTDB 
> plugin
>  and explore other means on integration.
> * New query optimization rules are added to greatly improve the query
> performance of IoTDB, like PredicatePushDown and AggregatePushDown.
> * Explain Analyze tool which can be used to analyze specific query
> performance better and is a good tool for troubleshooting performance problems
>
>
> ## Community Health:
> Overall community health is good.
> - Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
> - the mailing list traffic increases (226 emails compared to 76)
>
> Best,
> ---
> Xiangdong Huang

IoTDB quarter report (2024 Q2)

2024-06-11 Thread Xiangdong Huang

Hi all,

This is the draft of IoTDB quarter report

## Description:
The mission of Apache IoTDB is the creation and maintenance of software related
to an IoT native database with high performance for data management and analysis

## Project Status:
Current project status: Ongoing with high activity.
Issues for the board:
- there are some trademark issues but the community has resolved them.

## Membership Data:
Apache IoTDB was founded 2020-09-16 (4 years ago)
There are currently 63 committers and 29 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
- William Song was added as committer on 2024-03-22
- Caiyin Yang was added as committer on 2024-06-12

## Project Activity:
- Recent releases:
IOTDB-1.3.2 is on the way.
IOTDB-1.3.1 was released on 2024-04-22.
IOTDB-1.3.0 was released on 2024-01-01.
IOTDB-1.2.2 was released on 2023-10-15.

- Main work of project:
* We are working on the table model providing data to users in a table view
in the same way as relational databases which will greatly reduce the
learning curve of IoTDB.
* Collaborate with the Apache StreamPipes community to improve the IoTDB plugin
 and explore other means on integration.
* New query optimization rules are added to greatly improve the query
performance of IoTDB, like PredicatePushDown and AggregatePushDown.
* Explain Analyze tool which can be used to analyze specific query
performance better and is a good tool for troubleshooting performance problems


## Community Health:
Overall community health is good.
- Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
- the mailing list traffic increases (226 emails compared to 76)

Best,
---
Xiangdong Huang

ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.

2024-06-11 Thread Trevor Hart

Hello Team



I have a customer using my application with IoTDB 1.3.0 in the backend.



Auto create schema is enabled as there is no fixed template. Multiple devices 
are pushing data to IotDB.



I have seen some ALIGN BY DEVICE queries fail with the error in the subject ie



ALIGN BY DEVICE: the data types of the same measurement column should be the 
same across devices.



The "show timeseries" shows that from one device the value has been inferred as 
FLOAT and another as DOUBLE.


Per the documentation it is suggested that the default 
"floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why am 
I getting FLOAT? Is the default "floating_string_infer_type" not being honoured?



Currently in the config file "floating_string_infer_type" is commented out. 
Should I explicitly set this to DOUBLE?


Thanks 

Trevor Hart

Re: AW: version about odbc

2024-06-10 Thread Haonan Hou

There is another example of the rest-client-c-example without the pom.xml that 
demonstrate how to use rest api with c language. Should we consider it as 
document example, or a "compilable" example? 

I prefer it is a compilable module and I think it is possible to add a pom.xml 
and make it as a maven module.

Haonan

On 2024/06/07 10:20:21 Christofer Dutz wrote:
> The ODBC is not really a “compilable” example.
> It contains a description how you can use a commercial product to wrap our 
> JDBC driver in order to produce an ODBC driver.
> Also does it contain a compilable C# project file, that demonstates the usage 
> of the ODBC API.
> 
> However even if the example would be compiled as part of the build, you would 
> still need to manually prepare the ODBC wrapper.
> 
> I would consider this module more documentation than “compilable code”.
> 
> Chris
> 
> 
> Von: Yuan Tian 
> Datum: Freitag, 7. Juni 2024 um 12:14
> An: dev 
> Betreff: version about odbc
> Hi all,
> 
> I found that in master branch, we've already upgraded the version to 
> 1.3.3-SNAPSHOT, but the examples about odbc module is missed?
> 
> [cid:ii_lx4j3xy50]
> 
> Best regards,
> -
> Yuan Tian
>

AW: Session API issues

2024-06-10 Thread Christofer Dutz

I guess the simplest solution would simply be to return an error if not all 
paths in the query are valid. Right now I added a check on the client side that 
throws an exception if the size of the paths is different from that of the 
fields.

Chris



Von: Christofer Dutz 
Datum: Sonntag, 9. Juni 2024 um 16:40
An: dev@iotdb.apache.org 
Betreff: Session API issues
Hi all,

I have been playing around with the session API and have been stumbling into 
some pits, that I think would be worth addressing.

The biggest issue I have had recently is using the executeAggregationQuery 
method, passing in pairs of queries and aggregations.
However, I noticed that if for example I pass in 3 queries with the middle one 
being invalid, that I simply get back a 2-field response and I have no way of 
seeing which field was invalid.

In general, I think the passing in multiple lists with each the same size sort 
of feels a bit odd. Instead of passing in one list of objects which again have 
multiple options would be a preferrable approach.
Also, would this allow us to add these as references to the responses. So 
having a Field returned by RowRecord.getFields() have a reference to the query 
object, would make the API better to handle.
We wouldn’t have to pass the fields when returning the result, this could be 
something the client does completely in the client to avoid unnecessary 
payload. I do think however for this to work, that we would need some sort of 
error handling, passing back something like an ErrorField, if a query was wrong 
or anything similar.

Chris

Session API issues

2024-06-09 Thread Christofer Dutz

Hi all,

I have been playing around with the session API and have been stumbling into 
some pits, that I think would be worth addressing.

The biggest issue I have had recently is using the executeAggregationQuery 
method, passing in pairs of queries and aggregations.
However, I noticed that if for example I pass in 3 queries with the middle one 
being invalid, that I simply get back a 2-field response and I have no way of 
seeing which field was invalid.

In general, I think the passing in multiple lists with each the same size sort 
of feels a bit odd. Instead of passing in one list of objects which again have 
multiple options would be a preferrable approach.
Also, would this allow us to add these as references to the responses. So 
having a Field returned by RowRecord.getFields() have a reference to the query 
object, would make the API better to handle.
We wouldn’t have to pass the fields when returning the result, this could be 
something the client does completely in the client to avoid unnecessary 
payload. I do think however for this to work, that we would need some sort of 
error handling, passing back something like an ErrorField, if a query was wrong 
or anything similar.

Chris

AW: version about odbc

2024-06-07 Thread Christofer Dutz

The ODBC is not really a “compilable” example.
It contains a description how you can use a commercial product to wrap our JDBC 
driver in order to produce an ODBC driver.
Also does it contain a compilable C# project file, that demonstates the usage 
of the ODBC API.

However even if the example would be compiled as part of the build, you would 
still need to manually prepare the ODBC wrapper.

I would consider this module more documentation than “compilable code”.

Chris


Von: Yuan Tian 
Datum: Freitag, 7. Juni 2024 um 12:14
An: dev 
Betreff: version about odbc
Hi all,

I found that in master branch, we've already upgraded the version to 
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[cid:ii_lx4j3xy50]

Best regards,
-
Yuan Tian

version about odbc

2024-06-07 Thread Yuan Tian

Hi all,

I found that in master branch, we've already upgraded the version to
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[image: image.png]

Best regards,
-
Yuan Tian

Re: Prepare releasing 1.3.2?

2024-06-05 Thread Haonan Hou

Ok, the version of master branch has updated to 1.3.3-SNAPSHOT.
Once Ratis 3.1.0 released, I'll prepare the rc/1.3.2 branch. 

Best,
Haonan


On 2024/06/03 02:07:55 Xinyu Tan wrote:
> Hi haonan
> 
> +1 for bumping the version of master branch to 1.3.3-SNAPSHOT.
> 
> As for the 1.3.2 release, currently ratis is releasing there new version 
> 3.1.0[1], we can rely on the official Ratis 3.1.0 release in the open source 
> version IoTDB 1.3.2 instead of the snapshot release to conform to the 
> guidelines of the Apache community.
> 
> Best
> -
> Xinyu Tan
> 
> [1] https://lists.apache.org/thread/q8gdfz1j1l4qv1v24z4dxcg01j9qs28n
> 
> On 2024/06/03 01:44:17 Haonan Hou wrote:
> > Hi all,
> > 
> > IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over 
> > bug fixes and improvements. I think we can start the process of releasing 
> > new version.
> > 
> > If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the 
> > version of master branch to 1.3.3-SNAPSHOT.
> > 
> > BR,
> > Haonan
> > 
>

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

2024-06-05 Thread ITAMI SHO

Dear Junzhi Peng and Sicheng Yu,

Thank you for sharing the exciting news about the new consensus protocols, 
FastIoTConsensus and IoTV2Consensus. Your efforts and contributions to 
improving the IoTDB project are greatly appreciated. We look forward to 
exploring these new features and providing feedback.

Best regards,

Itami Sho

2024年6月3日 18:28，Junzhi Peng  写道：

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Device Level TTL for IoTDB

Hi Peichen

I'm very excited to see the implementation of device-level TTL. Before your 
work, if users wanted to set different TTLs for different devices, they had to 
model them into many databases, which was very inconvenient. After your work, 
users can set different TTLs within a single database as they wish, greatly 
enhancing the user experience for those users with TTL needs.

I am thrilled to see your work being merged and look forward to it creating 
greater value for users.

Best

Xinyu Tan

On 2024/06/03 14:10:05 BensonChou wrote:
> Hello, everyone.
> 
> I am BensonChou(Peichen Zhou), a committer of IoTDB. Over the past few 
> months, I have implemented a new feature, which is device-level TTL, and I am 
> so excited to share this new feature with you.
> 
> Previously, IoTDB only supported database-level TTL. For devices with 
> different collection frequencies and data validity periods under a single 
> database, we want to use a more granular device-level TTL to manage the data 
> lifecycle and optimize disk space as much as possible.
> 
> The new version of IoTDB will support device-level TTL management, allowing 
> users to set, modify, and unset TTLs in bulk. When setting TTL, the system 
> will apply it to all devices that match the specified path. If multiple TTLs 
> exist on a device path, the TTL closest to the device node will take effect. 
> Once device data expires, it will no longer be queryable. While the data in 
> disk files is not guaranteed to be immediately deleted, it will eventually be 
> deleted.
> 
> The related SQL syntax isas following:
> 
> 
> 
> SET TTL TO path value. Eg: SET TTL TO ROOT.DB 3600;
> 
> UNSET TTL TO path. Eg: UNSET TTL TO ROOT.DB;
> 
> SHOW ALL TTL
> Additionally, we have introduced the keyword `INF` to represent an infinite 
> TTL value.You can achieve more diversified configurations to meet 
> different scenarios and requirements through the configuration parameters 
> `default_ttl_in_ms`, `ttl_check_interval`, `ttl_rule_capacity
> `, `max_expired_time`, and `expired_data_ratio` in the configuration file. 
> 
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> BensonChou
> 
> Reference:
> 
> https://github.com/apache/iotdb/pull/12122

Re: [Proposal] Device Level TTL for IoTDB

Hi Peichen

I'm very excited to see the implementation of device-level TTL. Before your 
work, if users wanted to set different TTLs for different devices, they had to 
model them into many databases, which was very inconvenient. After your work, 
users can set different TTLs within a single database as they wish, greatly 
enhancing the user experience for those users with TTL needs.

I am thrilled to see your work being merged and look forward to it creating 
greater value for users.

Best

Xinyu Tan

On 2024/06/03 14:10:05 BensonChou wrote:
> Hello, everyone.
> 
> I am BensonChou(Peichen Zhou), a committer of IoTDB. Over the past few 
> months, I have implemented a new feature, which is device-level TTL, and I am 
> so excited to share this new feature with you.
> 
> Previously, IoTDB only supported database-level TTL. For devices with 
> different collection frequencies and data validity periods under a single 
> database, we want to use a more granular device-level TTL to manage the data 
> lifecycle and optimize disk space as much as possible.
> 
> The new version of IoTDB will support device-level TTL management, allowing 
> users to set, modify, and unset TTLs in bulk. When setting TTL, the system 
> will apply it to all devices that match the specified path. If multiple TTLs 
> exist on a device path, the TTL closest to the device node will take effect. 
> Once device data expires, it will no longer be queryable. While the data in 
> disk files is not guaranteed to be immediately deleted, it will eventually be 
> deleted.
> 
> The related SQL syntax isas following:
> 
> 
> 
> SET TTL TO path value. Eg: SET TTL TO ROOT.DB 3600;
> 
> UNSET TTL TO path. Eg: UNSET TTL TO ROOT.DB;
> 
> SHOW ALL TTL
> Additionally, we have introduced the keyword `INF` to represent an infinite 
> TTL value.You can achieve more diversified configurations to meet 
> different scenarios and requirements through the configuration parameters 
> `default_ttl_in_ms`, `ttl_check_interval`, `ttl_rule_capacity
> `, `max_expired_time`, and `expired_data_ratio` in the configuration file. 
> 
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> BensonChou
> 
> Reference:
> 
> https://github.com/apache/iotdb/pull/12122

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
>

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
>

Re: [DISCUSS] GH Actions optimizations

2024-06-05 Thread Yuan Tian

ok..., if so, I think it's really nice.

On Wed, Jun 5, 2024 at 8:28 PM Christofer Dutz 
wrote:

> Also is the settings currently done that it only skips any tests in
> non-master branches … so if there’s a PR on a branch, that will be
> optimized, but every commit going back to master should have all tests run.
>
> Chris
>
>
> Von: Christofer Dutz 
> Datum: Mittwoch, 5. Juni 2024 um 14:22
> An: dev@iotdb.apache.org 
> Betreff: AW: [DISCUSS] GH Actions optimizations
> Well even with the re-run option still failures get counted and a
> dashboard like this:
>
> https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
> Is a lot better than having to continuously go back to your PRs and click
> on “re-run failed tests”
>
> Chris
>
> Von: Yuan Tian 
> Datum: Mittwoch, 5. Juni 2024 um 03:06
> An: dev@iotdb.apache.org 
> Betreff: Re: [DISCUSS] GH Actions optimizations
> Hi Chris,
>
> That's really good news. About the `surefire` part, I think we need to
> discuss more about it because some of the previously failed tests have
> indeed helped us discover concurrency issues in IoTDB. I am concerned that
> doing this might reduce the likelihood of exposing such bugs.
>
>
> Best regards,
> 
> Yuan Tian
>

AW: [DISCUSS] GH Actions optimizations

2024-06-05 Thread Christofer Dutz

Also is the settings currently done that it only skips any tests in non-master 
branches … so if there’s a PR on a branch, that will be optimized, but every 
commit going back to master should have all tests run.

Chris


Von: Christofer Dutz 
Datum: Mittwoch, 5. Juni 2024 um 14:22
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] GH Actions optimizations
Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

AW: [DISCUSS] GH Actions optimizations

2024-06-05 Thread Christofer Dutz

Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

Re: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

2024-06-05 Thread Jialin Qiao

Thank Chris!

Jialin Qiao

Christofer Dutz  于2024年6月4日周二 17:28写道：
>
> Ok … artifacts should be available …
>
> Chris
>
> Von: Christofer Dutz 
> Datum: Dienstag, 4. Juni 2024 um 11:26
> An: dev@iotdb.apache.org 
> Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
> Thrift 0.20.0)
> So, the vote passes with 7 +1 votes at least 5 I could verify as binding 
> votes.
>
> In the future, it would be great if people would add their name/apacheId etc. 
> to the votes to make it easier to verify and count for the RM.
>
> Releasing the maven artifacts right away.
>
> Thanks for all those who voted.
>
> Chris
>
>
> Von: Christofer Dutz 
> Datum: Dienstag, 4. Juni 2024 um 09:37
> An: dev@iotdb.apache.org 
> Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
> 0.20.0)
> Yeah.. Currently at community over code... Was planning on doing this in a 
> break today.
>
> Gesendet von Outlook für Android
> 
> From: Haonan Hou 
> Sent: Monday, June 3, 2024 4:11:04 AM
> To: dev@iotdb.apache.org 
> Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
> 0.20.0)
>
> Hi Chris,
>
> I think 72 hours has passed. These artifacts can be released now?
>
> Thanks,
> Haonan
>
> On 2024/05/28 14:40:39 Christofer Dutz wrote:
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the 
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible 
> > again), I also updated the Thrift version and staged a release of Thrift 
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so 
> > it should be quick to review.
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be 
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then finish 
> > changing the main repo to use these artifacts.
> >
> >
> > Chris
> >

[DISCUSS] GH Actions optimizations

At today’s sessions at community over code EU in a session about “Apache build 
observability” we got a little spotlight on IoTDB.
The develocity folks pointed out Apache IoTDB as being the one Apache project 
that provides the biggest room for improvement by enabling some of their 
build-optimization tools.

I have created a PR : https://github.com/apache/iotdb/pull/12654

This enables predictive test-selection which should prioritize tests that are 
most probable to be affected by changes and to skip ones that are more 
definitely not going to be affected.

I was actively approached by the develocity folks here at the conference and 
together we created a PR that should enable this for us.
Also did I take the liberty to also enable surefire to automatically re-run 
failed tests for a maximum of 3 times before actually failing a test.

These changes should allow reducing the load we have on GH Actions by 40% 
(according to the dashboards develocity has)

Also did they recommend utilizing build cache extensions, which should even 
more greatly reduce the build time and the load on the GH runners.

If anyone wants to play around with the dashboards for IoTDB …. Please have a 
look here.

https://ge.apache.org/scans?search.rootProjectNames=*IoTDB*

There are very interesting dashboards that show us stuff like flaky tests:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY

Chris

AW: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

Ok … artifacts should be available …

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 11:26
An: dev@iotdb.apache.org 
Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
Thrift 0.20.0)
So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

[RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)