Enable C# Client to Intialize and Reconnect with Multiple Node Urls

[ZHAN LU]

Hi,

I have implemented an interface within the C# client that supports
initialization using multiple Node URLs. Additionally, the reconnection
method has been enhanced to select one of the various Nodes for
reestablishing a connection. For further details, please refer to PR[1].

Would someone be available to assist with a review? Thank you!

Zhan Lu

[1] https://github.com/apache/iotdb-client-csharp/pull/8

RE: Re:Integrating UDF Library and Grafana (conf update) in Docker

[Stephen Lawrence]

Miaohsh wrote:
> The rest service currently does not support hot loading. After modifying
> the configuration, the iotdb service needs to be restarted before it can
> be used
> 

Thank you for the confirmation. I just wanted to check I was understanding
the IoTDB configuration mechanisms correctly.

As I needed to integrate the UDF Data Quality Library as well I took the
decision to maintain an IoTDB Dockerfile in the downstream COVESA Data 
Service Playground project that builds on the upstream IoTDB image. 
I have that working locally, along with the REST configuration change.

I have a short list of issues I found in the IoTDB documentation.
I will look at sending a Github PR to fix them. 
I will also look at updating the Playground to use IoTDB 1.3.2.

Best Wishes,

Steve

[ANNOUNCE] Apache IoTDB 1.3.2 released

[Haonan Hou]

The Apache IoTDB team is pleased to announce the release of Apache IoTDB
1.3.2.

Apache IoTDB (Database for Internet of Things) is an IoT native database
with high performance for data management and analysis, deployable on the
edge and the cloud.

Apache IoTDB 1.3.2 is the latest version of the v1 series. Feel free to upgrade
from earlier version.

## Features & Improvements
- Storage Module: Performance improvement in the insertRecords interface for 
writing
- Query Module: New Explain Analyze statement added (monitoring the time spent 
on each stage of a single SQL execution)
- Query Module: New UDAF (User-Defined Aggregate Function) framework added
- Query Module: New MaxBy/MinBy functions added, supporting the retrieval of 
maximum/minimum values along with the corresponding timestamps
- Query Module: Performance improvement in value filtering queries
- Data Synchronization: Path matching supports path pattern
- Data Synchronization: Supports metadata synchronization (including time 
series and related attributes, permissions, etc.)
- Stream Processing: Added Alter Pipe statement, supporting hot updates of 
plugins for Pipe tasks
- System Module: System data point count statistics now include statistics for 
data imported by loading TsFile
- Scripts and Tools: New local upgrade backup tool added (backing up original 
data through hard links)
- Scripts and Tools: New export-data/import-data scripts added, supporting data 
export in CSV, TsFile formats or SQL statements
- Scripts and Tools: Windows environment now supports distinguishing 
ConfigNode, DataNode, and Cli by window name
...
## Bugs
- Optimize the error message when a NullPointerException (NPE) occurs due to a 
timeout when dropping a database.
- Add logs for notifyLeaderReady, notifyLeaderChanged, and procedure worker.
- Add compatibility handling for existing erroneous data during file merging.
- Fix the deadlock issue caused by flushing empty files during querying.
- Fix the issue where Ratis becomes unresponsive during read, write, and delete 
operations.
- Fix the concurrent bug in load and merge operations.
- Fix the issue where the system's compression ratio is recorded as a negative 
number in the file for certain scenarios.
- Fix the ConcurrentModificationException issue during memory estimation for 
merge tasks.
- Fix potential deadlocks that may occur when writing, automatically creating, 
and deleting databases concurrently.
...

The full release note is available at:
https://raw.githubusercontent.com/apache/iotdb/rc/1.3.2/RELEASE_NOTES.md

The release is available for download at:
http://iotdb.apache.org/Download

Maven artifacts for JDBC driver, session SDK, TsFile SDK can be found at:
https://search.maven.org/search?q=3Dg:org.apache.iotdb


Regards,
The Apache IoTDB team

[RESULT][VOTE] Release Apache IoTDB 1.3.2

[Haonan Hou]

Hi all,

The release is approved by accepting 3 +1 votes:

3 from PMC members,
Qingxin Feng,
Christofer Dutz,
Jialin Qiao.

Vote threads:

https://lists.apache.org/thread/svdkp2gqy960t9svpmygp4hdjcdnoswo


Thanks,
Haonan Hou

Re: [VOTE] Apache IoTDB 1.3.2 RC1 release

[Jialin Qiao]

Hi,

+1 (binding)

The source release:
apache headers [ok]
signatures and hashes [ok]
LICENSE and NOTICE [ok]
no jar files [ok]

The binary distribution:
version number in CLI [ok]
signatures and hashes [ok]
start in mac, jdk8 [ok]

Thanks,
Jialin Qiao

Christofer Dutz  于2024年6月28日周五 19:15写道：
>
> +1 (binding)
>
> Chris
>
> [OK] Download all staged artifacts under the url specified in the release 
> vote email.
> [OK] Verify the signature is correct.
> [OK] Check if the signature references an Apache email address.
> [OK] Verify the SHA512 hashes.
> [OK] Unzip the archive.
> [OK] Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in 
> the extracted source bundle.
> [MINOR] Verify the content of LICENSE, NOTICE, README, RELEASE_NOTES files in 
> the extracted source bundle.
>
>   *   RELEASE_NOTES hasn’t been updated for quite some time.
>   *   NOTICE: Could possibly need an update on the Apache Commons Collections 
> section, as we’re most probably using versions released after 2019.
> [MINOR] Run RAT externally to ensure there are no surprises.
>
>   *   Integration-test module contains 3 archives, 1 of them I couldn’t find 
> the sources to (TriggerFireTimesCounter)
> [MINOR] Search for SNAPSHOT references
>
>   *   It seems the following projects were not part of the release and 
> therefore their version was not updated:
>  *   example/client-cpp-example
>  *   iotdb-client/client-cpp
> [OK] Search for Copyright references, and if they are in headers, make sure 
> these files containing them are mentioned in the LICENSE file.
>
>
> Von: 冯 庆新 
> Datum: Donnerstag, 27. Juni 2024 um 15:05
> An: dev@iotdb.apache.org 
> Betreff: 回复: [VOTE] Apache IoTDB 1.3.2 RC1 release
> Hello  all :
> +1
>
> The binary distribution:
> version number in CLI [ok]
> start in CentOS7, jdk11 [ok]
> performance verification passed:  [ok]
> ### Server Configurations ###//
> CPU=16
> Memory=32G
> Disk=1.8T HDD
> Ethernet=1000Mbit
> ///
> ### IoTDB Configurations ###///
> MAX_HEAP_SIZE="20G" in datanode-env.sh
> ///
> ### Client Mode ###
> Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET
> CLIENT_NUMBER=10
> GROUP_NUMBER=10
> DEVICE_NUMBER=50
> SENSOR_NUMBER=500
> BATCH_SIZE_PER_WRITE=10
> LOOP=86400
> ///
>
> ### Test Result ###
> 1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points；
> Non-Aligned Timseries:
> Throughput(points/s): 17,009,956.18  Cost Time(s): 1,269.84
> Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48
> Aligned Timseries:
> Throughput(points/s): 18,141,900.23  Cost Time(s): 1,190.61
> Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48
> ///
>
>
> Avg: Average time cost of all ingestion operations. [ms]
> MiddleAvg: Average time cost of ingestion operations without 5% head and 
> tail. [ms]
> Min: Min single operation time cost of all ingestion operations. [ms]
> Max: Max single operation  time cost of all ingestion operations. [ms]
> These results are tested with 
> iot-benchmark(https://github.com/thulab/iot-benchmark)
> That's all.
> Please feel free to contact me if you have any questions
> Thank you.
> Best Regards!
> Qingxin Feng
>
> 从 Windows 版邮件发送
>
> 
> 发件人: Haonan Hou 
> 发送时间: Thursday, June 27, 2024 5:55:36 PM
> 收件人: dev@iotdb.apache.org 
> 主题: [VOTE] Apache IoTDB 1.3.2 RC1 release
>
> Hi all,
>
> Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
> on accepting it for release. All Maven artifacts are available under [1].
> Voting will be open for 72hr.
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
>
> Release tag: v1.3.2
> Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885
>
> Before voting +1, PMC members are required to download
> the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also
> verifying that the package meets the requirements of the ASF policy
> on releases. [3]
>
> You can achieve the above by following [4].
>
> [ ] +1 accept (indicate what you validated - e.g. performed the
> non-RM items in [4])
> [ ] -1 reject (explanation required)
>
>
> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
> [2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4] 
> 

AW: [VOTE] Apache IoTDB 1.3.2 RC1 release

[Christofer Dutz]

+1 (binding)

Chris

[OK] Download all staged artifacts under the url specified in the release vote 
email.
[OK] Verify the signature is correct.
[OK] Check if the signature references an Apache email address.
[OK] Verify the SHA512 hashes.
[OK] Unzip the archive.
[OK] Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in 
the extracted source bundle.
[MINOR] Verify the content of LICENSE, NOTICE, README, RELEASE_NOTES files in 
the extracted source bundle.

  *   RELEASE_NOTES hasn’t been updated for quite some time.
  *   NOTICE: Could possibly need an update on the Apache Commons Collections 
section, as we’re most probably using versions released after 2019.
[MINOR] Run RAT externally to ensure there are no surprises.

  *   Integration-test module contains 3 archives, 1 of them I couldn’t find 
the sources to (TriggerFireTimesCounter)
[MINOR] Search for SNAPSHOT references

  *   It seems the following projects were not part of the release and 
therefore their version was not updated:
 *   example/client-cpp-example
 *   iotdb-client/client-cpp
[OK] Search for Copyright references, and if they are in headers, make sure 
these files containing them are mentioned in the LICENSE file.


Von: 冯 庆新 
Datum: Donnerstag, 27. Juni 2024 um 15:05
An: dev@iotdb.apache.org 
Betreff: 回复: [VOTE] Apache IoTDB 1.3.2 RC1 release
Hello  all :
+1

The binary distribution:
version number in CLI [ok]
start in CentOS7, jdk11 [ok]
performance verification passed:  [ok]
### Server Configurations ###//
CPU=16
Memory=32G
Disk=1.8T HDD
Ethernet=1000Mbit
///
### IoTDB Configurations ###///
MAX_HEAP_SIZE="20G" in datanode-env.sh
///
### Client Mode ###
Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET
CLIENT_NUMBER=10
GROUP_NUMBER=10
DEVICE_NUMBER=50
SENSOR_NUMBER=500
BATCH_SIZE_PER_WRITE=10
LOOP=86400
///

### Test Result ###
1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points；
Non-Aligned Timseries:
Throughput(points/s): 17,009,956.18  Cost Time(s): 1,269.84
Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48
Aligned Timseries:
Throughput(points/s): 18,141,900.23  Cost Time(s): 1,190.61
Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48
///


Avg: Average time cost of all ingestion operations. [ms]
MiddleAvg: Average time cost of ingestion operations without 5% head and tail. 
[ms]
Min: Min single operation time cost of all ingestion operations. [ms]
Max: Max single operation  time cost of all ingestion operations. [ms]
These results are tested with 
iot-benchmark(https://github.com/thulab/iot-benchmark)
That's all.
Please feel free to contact me if you have any questions
Thank you.
Best Regards!
Qingxin Feng

从 Windows 版邮件发送


发件人: Haonan Hou 
发送时间: Thursday, June 27, 2024 5:55:36 PM
收件人: dev@iotdb.apache.org 
主题: [VOTE] Apache IoTDB 1.3.2 RC1 release

Hi all,

Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
on accepting it for release. All Maven artifacts are available under [1].
Voting will be open for 72hr.
A minimum of 3 binding +1 votes and more binding +1 than binding -1
are required to pass.

Release tag: v1.3.2
Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885

Before voting +1, PMC members are required to download
the signed source code package, compile it as provided, and test
the resulting executable on their own platform, along with also
verifying that the package meets the requirements of the ASF policy
on releases. [3]

You can achieve the above by following [4].

[ ] +1 accept (indicate what you validated - e.g. performed the
non-RM items in [4])
[ ] -1 reject (explanation required)


[1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
[2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
[3] https://www.apache.org/dev/release.html#approving-a-release
[4] 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release

Best,

Haonan Hou

回复: [VOTE] Apache IoTDB 1.3.2 RC1 release

[冯 庆新]

Hello  all :
+1

The binary distribution:
version number in CLI [ok]
start in CentOS7, jdk11 [ok]
performance verification passed:  [ok]
### Server Configurations ###//
CPU=16
Memory=32G
Disk=1.8T HDD
Ethernet=1000Mbit
///
### IoTDB Configurations ###///
MAX_HEAP_SIZE="20G" in datanode-env.sh
///
### Client Mode ###
Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET
CLIENT_NUMBER=10
GROUP_NUMBER=10
DEVICE_NUMBER=50
SENSOR_NUMBER=500
BATCH_SIZE_PER_WRITE=10
LOOP=86400
///

### Test Result ###
1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points；
Non-Aligned Timseries:
Throughput(points/s): 17,009,956.18  Cost Time(s): 1,269.84
Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48
Aligned Timseries:
Throughput(points/s): 18,141,900.23  Cost Time(s): 1,190.61
Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48
///


Avg: Average time cost of all ingestion operations. [ms]
MiddleAvg: Average time cost of ingestion operations without 5% head and tail. 
[ms]
Min: Min single operation time cost of all ingestion operations. [ms]
Max: Max single operation  time cost of all ingestion operations. [ms]
These results are tested with 
iot-benchmark(https://github.com/thulab/iot-benchmark)
That's all.
Please feel free to contact me if you have any questions
Thank you.
Best Regards!
Qingxin Feng

从 Windows 版邮件发送


发件人: Haonan Hou 
发送时间: Thursday, June 27, 2024 5:55:36 PM
收件人: dev@iotdb.apache.org 
主题: [VOTE] Apache IoTDB 1.3.2 RC1 release

Hi all,

Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
on accepting it for release. All Maven artifacts are available under [1].
Voting will be open for 72hr.
A minimum of 3 binding +1 votes and more binding +1 than binding -1
are required to pass.

Release tag: v1.3.2
Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885

Before voting +1, PMC members are required to download
the signed source code package, compile it as provided, and test
the resulting executable on their own platform, along with also
verifying that the package meets the requirements of the ASF policy
on releases. [3]

You can achieve the above by following [4].

[ ] +1 accept (indicate what you validated - e.g. performed the
non-RM items in [4])
[ ] -1 reject (explanation required)


[1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
[2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
[3] https://www.apache.org/dev/release.html#approving-a-release
[4] 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release

Best,

Haonan Hou

Re: Allow the client with python 3.6 to use IoTDB

[Haonan Hou]

Ok, this PR has been merged.

Best,
Haonan

On 2024/06/24 14:19:36 Haonan Hou wrote:
> Hi,
> 
> The current python client of IoTDB requires the python version >= 3.7.
> However, I met some applications running on some old platforms which only 
> support python 3.6, and I found it is easy to make the adaption. Here is the 
> PR[1].
> 
> In this PR, I also added a CI with python 3.6 and it can work correctly. It 
> would be good if I can get some approval.
> 
> Haonan Hou
> 
> [1] https://github.com/apache/iotdb/pull/12792
> 

[VOTE] Apache IoTDB 1.3.2 RC1 release

[Haonan Hou]

Hi all,

Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote
on accepting it for release. All Maven artifacts are available under [1].
Voting will be open for 72hr.
A minimum of 3 binding +1 votes and more binding +1 than binding -1
are required to pass.

Release tag: v1.3.2
Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885

Before voting +1, PMC members are required to download
the signed source code package, compile it as provided, and test
the resulting executable on their own platform, along with also
verifying that the package meets the requirements of the ASF policy
on releases. [3]

You can achieve the above by following [4].

[ ] +1 accept (indicate what you validated - e.g. performed the
non-RM items in [4])
[ ] -1 reject (explanation required)


[1] https://repository.apache.org/content/repositories/orgapacheiotdb-1159
[2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.2/rc1
[3] https://www.apache.org/dev/release.html#approving-a-release
[4] 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release

Best,

Haonan Hou

Integrating UDF Library and Grafana (conf update) in Docker

[Stephen Lawrence]

Hi,

After some successful experiments I would like to enable the optional 
data quality UDF Library and support Grafana in the COVESA Central 
Data Service Playground (CDSP). Functions like the LTTB Triangle 
sampling are very useful in automotive use-cases.

I have some questions related to UDF integration and IoTDB conf 
changes in a Docker deployment. I am currently using IoTDB v1.2.2

UDF LIBRARY
I was hoping to keep things simple for a while and not have a
CDSP specific IoTDB Dockerfile, but I can create one to add the
Library JAR to the image. Registration is a little more tricky to 
automate reliably as it is a one time process. I was thinking to 
include the library JAR by default and ask users to run the 
registration script themselves if they wish to use the library 
functions. That separates any registration issues from the build 
process, making debugging for non-experts easier.

However, if anyone has a simple method to do the registration in
the container that they have found reliable I would be happy to 
hear about it?
 
Q1) I noticed if the library functions are registered but the JAR
is not available on IoTDB restart, the exception handling causes
IoTDB to not start. Is that expected behaviour?

CONF CHANGES (enabling REST for Grafana)
Q2) From reading the IoTDB documentation it appears REST
can not be enabled through runtime environment variables,
e.g. docker env, and instead must be done by modifying 
the key in conf/iotdb-datanode.properties. Is that correct?

Q3) The IoTDB docker documentation suggests sharing a conf
directory, like as with log/data, into the container if you 
wish to change conf files. I found that on startup IoTDB 
does not create a default conf environment if it is given 
an empty conf directory. As a result IoTDB does not start.
Is that expected behaviour?

As I will probably need to create an IoTDB dockerfile to
integrate the UDF Library JAR I may modify the conf file
there instead.

Best Wishes,

Steve

Allow the client with python 3.6 to use IoTDB

[Haonan Hou]

Hi,

The current python client of IoTDB requires the python version >= 3.7.
However, I met some applications running on some old platforms which only 
support python 3.6, and I found it is easy to make the adaption. Here is the 
PR[1].

In this PR, I also added a CI with python 3.6 and it can work correctly. It 
would be good if I can get some approval.

Haonan Hou

[1] https://github.com/apache/iotdb/pull/12792

RE: Introducing Apache IoTDB use in COVESA Central Data Service Playground

[Stephen Lawrence]

Hi,

Thank you both, for the encouragement and the offers of support.

> -Original Message-
> From: Jialin Qiao 
> Sent: Tuesday, June 18, 2024 9:50 AM
> To: dev@iotdb.apache.org
> Subject: Re: Introducing Apache IoTDB use in COVESA Central Data Service
> Playground
> 
> Hi,
> 
> Glad to see the integration!
> Welcome and feel free to ask :D
> 
> Jialin Qiao
> 
> Pengcheng Zheng  于2024年6月15日周六 04:38写道：
> >
> > Hi Stephen,
> >
> > Very exciting project!
> >
> > Thanks for sharing the details of COVESA/CDSP. I remember that you asked
> > some questions in Slack several months ago, and seems that you have made
> > great progress ;-)

Yes, originally I had some questions about running IoTDB on embedded h/w and 
the answers helped me commit to using IoTDB in the Playground.

BTW as part of the project I have done some work integrating IoTDB to other
technology:

1) COVESA and the W3C co-developed a REST-like get/set/sub API for accessing 
vehicle data in-vehicle called VISS. VISS has a reference implementation 
maintained in the VISSR project that supported SQLite and Redis as data
store backends. I upstreamed support for VISSR to also support IoTDB.

2) The Playground includes a bridge for streaming real car data from the 
RemotiveLabs virtual signal platform in the cloud into IoTDB. It has a
very simple implementation currently for the IoTDB connection but it has
no requirements on any other part of the Playground and could be a useful
example to anyone wanting to using RemotiveLabs with IoTDB. 

> >
> > IoTDB has already proved its excellence in the scenario of connected
> > vehicles before, and the key questions you've listed in the
> > presentation happen to be exactly what IoTDB & TsFile are addressing,
> such
> > as data syncing between all touchpoints/ bidirectional sync/ permissions
> &
> > privacy/ model updating/ subscription, etc.
> >
> > Also your ideas of the deployment in Docker and edge-cloud sync (if my
> > memory serves me right..) shall benefit a lot from IoTDB.

Yes, IoTDB has many features that made it attractive for me to use it in
the Playground. I am sure I will have questions as the project progresses
Especially, I would like to leverage the greater IoTDB experience here to
check we are using it effectively. So please excuse the odd basic question.

Over the last year I have looked through the presentations on the IoTDB website
and occasionally checked back for updates. Including the Chinese language 
ones where I have been able to translate. I remember seeing some mentions 
of IoTDB implementation in automotive but not many details. If I remember
correctly there was a major instance for a telematics use-case. I also saw 
that Table 4 of the IoTDB paper from SIGMOD 2023 listed three automotive 
deployments.

If you have details of IoTDB automotive deployments I would be very interested.

> >
> > Hopefully IoTDB could support the CDSP project well and help transform
> the
> > "Playground" to POC - and finally from theory to practice.

I think it will. Put simply the Playground is a toolset of Lego pieces.
These pieces are intended to be connected in a range of ways to create
PoCs or examples of logical concepts large and small and prove them out
in implementation. Hence "Playground". This is not a project to make a 
specific middleware 'product' or address a specific singular use-case.
At least in the current project setup.

The open source community aspect of COVESA means for effective collaboration
we need to also support diversity in architecture. In-vehicle architecture is
very complex and OEM A will make different choices to OEM B.

So, production will be a second step for individual companies to take
with their partners suited to their architectural and purchasing decisions.
That said, of course the advantage of using proven technology like IoTDB 
is that the path to production is there.

> >
> > Looking forward to your updates!

Thanks.

Best Wishes,

Stephen Lawrence

> >
> >
> > Best Wishes,
> > Pengcheng Zheng
> > Timecho Europe

[snip]

Re: Apply for a development platform account

[Yuan Tian]

Hi yuyong,

Welcome to IoTDB Community, I've already approved your jira account
application. It didn't work?


Best regards,
--
Yuan Tian



On Fri, Jun 21, 2024 at 4:50 PM 喻勇 <1471335...@qq.com.invalid> wrote:

> Dear IoTDB PMC:
>
> Greetings!
>
> I am yuyong,I am currently a graduate student at Chongqing University of
> Posts and Telecommunications, and I am currently very interested in IoTDB
> and would like to contribute my part in it. I am passionate about
> participating in the open source community.
>
> After studying the documentation and code base of IoTDB in detail, I found
> that the project is highly compatible with my technical background and
> interests. I believe that through my participation, I can bring some new
> ideas and solutions to the project and help it develop further.
>
> To prove my identity and willingness to contribute, I am providing the
> following information:
>
> My JIRA platform ID is: FearfulTomcat27
> My email address is: 1471335...@qq.com
>
> I plan to actively contribute to IoTDB by submitting bug fixes, optimizing
> code, writing documentation, or participating in discussions. I am aware
> that as a developer, I need to adhere to the project's development
> specifications and code quality standards, and respect and uphold the
> values of the open source community.
>
> If you need to know more about my technical background or experience,
> please feel free to check out my personal website or GitHub page:
> https://github.com/FearfulTomcat27
>
> Very much looking forward to being a part of IoTDB and working with other
> developers to make the project a success. Please feel free to contact me
> for any additional information or further discussion.
>
> Thank you again for taking the time to read my application. Looking
> forward to your reply!
>
> Best wishes!
>
> Name:yuyong
> Phone:15892323376
> Email:1471335...@qq.com

Apply for a development platform account

[喻勇]

Dear IoTDB PMC:
 
Greetings!
 
I am yuyong,I am currently a graduate student at Chongqing University of Posts 
and Telecommunications, and I am currently very interested in IoTDB and would 
like to contribute my part in it. I am passionate about participating in the 
open source community.

After studying the documentation and code base of IoTDB in detail, I found that 
the project is highly compatible with my technical background and interests. I 
believe that through my participation, I can bring some new ideas and solutions 
to the project and help it develop further.
 
To prove my identity and willingness to contribute, I am providing the 
following information:
 
My JIRA platform ID is: FearfulTomcat27
My email address is: 1471335...@qq.com

I plan to actively contribute to IoTDB by submitting bug fixes, optimizing 
code, writing documentation, or participating in discussions. I am aware that 
as a developer, I need to adhere to the project's development specifications 
and code quality standards, and respect and uphold the values of the open 
source community.
 
If you need to know more about my technical background or experience, please 
feel free to check out my personal website or GitHub 
page:https://github.com/FearfulTomcat27
 
Very much looking forward to being a part of IoTDB and working with other 
developers to make the project a success. Please feel free to contact me for 
any additional information or further discussion.
 
Thank you again for taking the time to read my application. Looking forward to 
your reply!
 
Best wishes!
 
Name:yuyong
Phone:15892323376
Email:1471335...@qq.com

答复: Recover/Reset Root Password?

[Wang Critas]

In addition, in versions after 1.3.1, if the file and the files in snapshot are 
deleted, restarting will automatically generate a file containing the default 
password, without the need to find the file and put it in it.

And you can also take a look at the relevant source code

发件人: Wang Critas 
日期: 星期二, 2024年6月18日 17:21
收件人: dev@iotdb.apache.org 
主题: 答复: Recover/Reset Root Password?
Hi Trevor
The method you mentioned is feasible, but it is important to note that the 
corresponding IoTDB version of the file needs to be matched. Additionally, the 
cluster needs to be restarted.
This operation is quite dangerous. It is recommended to make a backup in 
advance, and the best practice is to keep the password properly

发件人: Trevor Hart 
日期: 星期二, 2024年6月18日 16:33
收件人: Dev 
主题: Recover/Reset Root Password?
Hello



Is there a process to recover or reset the root user password? For example, if 
a customer forgets their root user password.



Is it possible to replace the "root.profile" file under 
data/confignode/system/users with the default? Would this work?


Thanks

Trevor Hart

答复: Recover/Reset Root Password?

[Wang Critas]

Hi Trevor
The method you mentioned is feasible, but it is important to note that the 
corresponding IoTDB version of the file needs to be matched. Additionally, the 
cluster needs to be restarted.
This operation is quite dangerous. It is recommended to make a backup in 
advance, and the best practice is to keep the password properly

发件人: Trevor Hart 
日期: 星期二, 2024年6月18日 16:33
收件人: Dev 
主题: Recover/Reset Root Password?
Hello



Is there a process to recover or reset the root user password? For example, if 
a customer forgets their root user password.



Is it possible to replace the "root.profile" file under 
data/confignode/system/users with the default? Would this work?


Thanks

Trevor Hart

Re: Introducing Apache IoTDB use in COVESA Central Data Service Playground

[Jialin Qiao]

Hi,

Glad to see the integration!
Welcome and feel free to ask :D

Jialin Qiao

Pengcheng Zheng  于2024年6月15日周六 04:38写道：
>
> Hi Stephen,
>
> Very exciting project!
>
> Thanks for sharing the details of COVESA/CDSP. I remember that you asked
> some questions in Slack several months ago, and seems that you have made
> great progress ;-)
>
> IoTDB has already proved its excellence in the scenario of connected
> vehicles before, and the key questions you've listed in the
> presentation happen to be exactly what IoTDB & TsFile are addressing, such
> as data syncing between all touchpoints/ bidirectional sync/ permissions &
> privacy/ model updating/ subscription, etc.
>
> Also your ideas of the deployment in Docker and edge-cloud sync (if my
> memory serves me right..) shall benefit a lot from IoTDB.
>
> Hopefully IoTDB could support the CDSP project well and help transform the
> "Playground" to POC - and finally from theory to practice.
>
> Looking forward to your updates!
>
>
> Best Wishes,
> Pengcheng Zheng
> Timecho Europe
>
> Am Fr., 14. Juni 2024 um 21:12 Uhr schrieb Stephen Lawrence <
> stephen.lawre...@renesas.com>:
>
> > Hi,
> >
> > As suggested in the wiki I am introducing myself and a new OSS project
> > using Apache IoTDB.
> >
> > My name is Stephen Lawrence from Renesas Electronics and I work in the
> > COVESA[1] open automotive alliance. COVESA is focused on developing common
> > approaches and technologies for connected vehicles, such as the VSS data
> > model [2]. I lead the COVESA Data Architecture and Infrastructure group and
> > am a Board member.
> >
> > We recently released the first version of the OSS COVESA Central Data
> > Service Playground (CDSP)[3] which uses Apache IoTDB as a data
> > store/server. The Playground is a neutral, open playground for
> > investigating data services in-vehicle and off-board in the context of
> > data-centric architectures. Importantly in community terms it provides a
> > means to publish and collaborate on such work in the open.
> >
> > Constructed as a "project of projects" this first release was focused on
> > creating the first version of the "toolkit". Now we pivot to improving it,
> > creating examples for its use and investigating industry issues. Currently
> > we are focused on Docker deployment for ease of development and connections
> > to other technology. Deployment on Automotive h/w is part of the future
> > roadmap. The contributors include various large vehicle OEMs.
> >
> > At a recent COVESA Conference I demonstrated this first release. The
> > online documentation is a work in progress but contains a summary of the
> > What/Why/How of the project. If you are interested in more details I led a
> > track of sessions about the Playground at the conference, the presentation
> > for which you can find here [4].
> >
> > I know I am going to have an increasing number of questions in the coming
> > months, so I wanted to first introduce myself and the project. I hope to be
> > able to contribute back to the IoTDB project as well.
> >
> > Apologies for the corporately expanded reference URLs in plain text:
> > [1] http://www.covesa.global/
> > [2] https://covesa.global/project/vehicle-signal-specification/
> > [3] https://github.com/COVESA/cdsp
> > [4]
> > https://wiki.covesa.global/download/attachments/98271360/Central_Data_Service_Playground_track_v2.pptx?version=1=1714486822982=v2
> >
> > Best Wishes,
> >
> > Stephen Lawrence
> > COVESA Data Architecture Lead and Board Member
> > (Renesas Electronics)
> >

Recover/Reset Root Password?

[Trevor Hart]

Hello



Is there a process to recover or reset the root user password? For example, if 
a customer forgets their root user password.



Is it possible to replace the "root.profile" file under 
data/confignode/system/users with the default? Would this work?


Thanks 

Trevor Hart

Introducing Apache IoTDB use in COVESA Central Data Service Playground

[Stephen Lawrence]

Hi,

As suggested in the wiki I am introducing myself and a new OSS project using 
Apache IoTDB.

My name is Stephen Lawrence from Renesas Electronics and I work in the 
COVESA[1] open automotive alliance. COVESA is focused on developing common 
approaches and technologies for connected vehicles, such as the VSS data model 
[2]. I lead the COVESA Data Architecture and Infrastructure group and am a 
Board member.

We recently released the first version of the OSS COVESA Central Data Service 
Playground (CDSP)[3] which uses Apache IoTDB as a data store/server. The 
Playground is a neutral, open playground for investigating data services 
in-vehicle and off-board in the context of data-centric architectures. 
Importantly in community terms it provides a means to publish and collaborate 
on such work in the open.

Constructed as a "project of projects" this first release was focused on 
creating the first version of the "toolkit". Now we pivot to improving it, 
creating examples for its use and investigating industry issues. Currently we 
are focused on Docker deployment for ease of development and connections to 
other technology. Deployment on Automotive h/w is part of the future roadmap. 
The contributors include various large vehicle OEMs.

At a recent COVESA Conference I demonstrated this first release. The online 
documentation is a work in progress but contains a summary of the What/Why/How 
of the project. If you are interested in more details I led a track of sessions 
about the Playground at the conference, the presentation for which you can find 
here [4].

I know I am going to have an increasing number of questions in the coming 
months, so I wanted to first introduce myself and the project. I hope to be 
able to contribute back to the IoTDB project as well.

Apologies for the corporately expanded reference URLs in plain text:
[1] http://www.covesa.global/
[2] https://covesa.global/project/vehicle-signal-specification/
[3] https://github.com/COVESA/cdsp
[4] 
https://wiki.covesa.global/download/attachments/98271360/Central_Data_Service_Playground_track_v2.pptx?version=1=1714486822982=v2

Best Wishes,

Stephen Lawrence
COVESA Data Architecture Lead and Board Member
(Renesas Electronics)

Re: IoTDB quarter report (2024 Q2)

[Xiangdong Huang]

> And I have something more to add:

Cool. I will add these activities.

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Pengcheng Zheng  于2024年6月12日周三 16:16写道：
>
> Hi Xiangdong,
>
> Great ;)
>
> And I have something more to add:
> - on 2024-04-09, IoTDB was presented at the JavaLand Summit in Germany. The
> talk was given by Christofer Dutz. [1]
> - on 2024-06-03, IoTDB was presented in 2 talks at CommunityOverCode EU.
> The talks were given by Christofer Dutz and Julian Feinauer. [2]
>
>
> [1] https://meine.doag.org/events/javaland/2024/agenda/#agendaId.3893
> [2] https://eu.communityovercode.org/program/
>
>
> Best regards,
> Pengcheng
>
>
> Am Mi., 12. Juni 2024 um 03:11 Uhr schrieb Xiangdong Huang <
> saint...@gmail.com>:
>
> > Hi all,
> >
> > This is the draft of IoTDB quarter report
> >
> > ## Description:
> > The mission of Apache IoTDB is the creation and maintenance of software
> > related
> > to an IoT native database with high performance for data management and
> > analysis
> >
> > ## Project Status:
> > Current project status: Ongoing with high activity.
> > Issues for the board:
> > - there are some trademark issues but the community has resolved them.
> >
> > ## Membership Data:
> > Apache IoTDB was founded 2020-09-16 (4 years ago)
> > There are currently 63 committers and 29 PMC members in this project.
> > The Committer-to-PMC ratio is roughly 2:1.
> >
> > Community changes, past quarter:
> > - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
> > - William Song was added as committer on 2024-03-22
> > - Caiyin Yang was added as committer on 2024-06-12
> >
> > ## Project Activity:
> > - Recent releases:
> > IOTDB-1.3.2 is on the way.
> > IOTDB-1.3.1 was released on 2024-04-22.
> > IOTDB-1.3.0 was released on 2024-01-01.
> > IOTDB-1.2.2 was released on 2023-10-15.
> >
> > - Main work of project:
> > * We are working on the table model providing data to users in a table view
> > in the same way as relational databases which will greatly reduce the
> > learning curve of IoTDB.
> > * Collaborate with the Apache StreamPipes community to improve the IoTDB
> > plugin
> >  and explore other means on integration.
> > * New query optimization rules are added to greatly improve the query
> > performance of IoTDB, like PredicatePushDown and AggregatePushDown.
> > * Explain Analyze tool which can be used to analyze specific query
> > performance better and is a good tool for troubleshooting performance
> > problems
> >
> >
> > ## Community Health:
> > Overall community health is good.
> > - Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
> > - the mailing list traffic increases (226 emails compared to 76)
> >
> > Best,
> > ---
> > Xiangdong Huang
> >

Re: IoTDB quarter report (2024 Q2)

[Pengcheng Zheng]

Hi Xiangdong,

Great ;)

And I have something more to add:
- on 2024-04-09, IoTDB was presented at the JavaLand Summit in Germany. The
talk was given by Christofer Dutz. [1]
- on 2024-06-03, IoTDB was presented in 2 talks at CommunityOverCode EU.
The talks were given by Christofer Dutz and Julian Feinauer. [2]


[1] https://meine.doag.org/events/javaland/2024/agenda/#agendaId.3893
[2] https://eu.communityovercode.org/program/


Best regards,
Pengcheng


Am Mi., 12. Juni 2024 um 03:11 Uhr schrieb Xiangdong Huang <
saint...@gmail.com>:

> Hi all,
>
> This is the draft of IoTDB quarter report
>
> ## Description:
> The mission of Apache IoTDB is the creation and maintenance of software
> related
> to an IoT native database with high performance for data management and
> analysis
>
> ## Project Status:
> Current project status: Ongoing with high activity.
> Issues for the board:
> - there are some trademark issues but the community has resolved them.
>
> ## Membership Data:
> Apache IoTDB was founded 2020-09-16 (4 years ago)
> There are currently 63 committers and 29 PMC members in this project.
> The Committer-to-PMC ratio is roughly 2:1.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
> - William Song was added as committer on 2024-03-22
> - Caiyin Yang was added as committer on 2024-06-12
>
> ## Project Activity:
> - Recent releases:
> IOTDB-1.3.2 is on the way.
> IOTDB-1.3.1 was released on 2024-04-22.
> IOTDB-1.3.0 was released on 2024-01-01.
> IOTDB-1.2.2 was released on 2023-10-15.
>
> - Main work of project:
> * We are working on the table model providing data to users in a table view
> in the same way as relational databases which will greatly reduce the
> learning curve of IoTDB.
> * Collaborate with the Apache StreamPipes community to improve the IoTDB
> plugin
>  and explore other means on integration.
> * New query optimization rules are added to greatly improve the query
> performance of IoTDB, like PredicatePushDown and AggregatePushDown.
> * Explain Analyze tool which can be used to analyze specific query
> performance better and is a good tool for troubleshooting performance
> problems
>
>
> ## Community Health:
> Overall community health is good.
> - Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
> - the mailing list traffic increases (226 emails compared to 76)
>
> Best,
> ---
> Xiangdong Huang
>

Introduction of Enhanced Load Functionality with Multi-Drive Support

[ITAMI SHO]

Hello everyone,

My name is Itami Sho, a contributor to IoTDB, and I am excited to share with 
you a recent enhancement I have implemented in our Load functionality. This 
optimization introduces support for multiple drive letters during the second 
phase of Load functionality to store TsFile PieceNode.

Previously, the Load feature only supported PieceNode storing on a single drive 
letter. However, in environments with multi-drive clusters, this limitation 
could lead to performance bottlenecks. To address this issue, I have optimized 
the Load functionality to support multiple drive letters, significantly 
improving overall performance in such scenarios.

You can review the details of my implementation in the following PR:
https://github.com/apache/iotdb/pull/12675

Best regards,
Itami Sho

Re: ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.

[Trevor Hart]

Ah that is great to hear! Thank you!




Thanks 

Trevor Hart




 On Wed, 12 Jun 2024 13:50:43 +1200 Jialin Qiao  
wrote ---



Hi, 
 
We just changed the default infer type of integer and floating to 
DOUBLE [1], it will be released in 1.3.2. 
 
For <=1.3.1, you could explicitly set this to DOUBLE. 
 
[1] https://github.com/apache/iotdb/pull/12223 
 
Jialin Qiao 
 
Trevor Hart  于2024年6月12日周三 06:41写道： 
> 
> Hello Team 
> 
> 
> 
> I have a customer using my application with IoTDB 1.3.0 in the backend. 
> 
> 
> 
> Auto create schema is enabled as there is no fixed template. Multiple devices 
> are pushing data to IotDB. 
> 
> 
> 
> I have seen some ALIGN BY DEVICE queries fail with the error in the subject 
> ie 
> 
> 
> 
> ALIGN BY DEVICE: the data types of the same measurement column should be the 
> same across devices. 
> 
> 
> 
> The "show timeseries" shows that from one device the value has been inferred 
> as FLOAT and another as DOUBLE. 
> 
> 
> Per the documentation it is suggested that the default 
> "floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why 
> am I getting FLOAT? Is the default "floating_string_infer_type" not being 
> honoured? 
> 
> 
> 
> Currently in the config file "floating_string_infer_type" is commented out. 
> Should I explicitly set this to DOUBLE? 
> 
> 
> Thanks 
> 
> Trevor Hart

Re: ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.

[Jialin Qiao]

Hi,

We just changed the default infer type of integer and floating to
DOUBLE [1], it will be released in 1.3.2.

For <=1.3.1, you could explicitly set this to DOUBLE.

[1] https://github.com/apache/iotdb/pull/12223

Jialin Qiao

Trevor Hart  于2024年6月12日周三 06:41写道：
>
> Hello Team
>
>
>
> I have a customer using my application with IoTDB 1.3.0 in the backend.
>
>
>
> Auto create schema is enabled as there is no fixed template. Multiple devices 
> are pushing data to IotDB.
>
>
>
> I have seen some ALIGN BY DEVICE queries fail with the error in the subject ie
>
>
>
> ALIGN BY DEVICE: the data types of the same measurement column should be the 
> same across devices.
>
>
>
> The "show timeseries" shows that from one device the value has been inferred 
> as FLOAT and another as DOUBLE.
>
>
> Per the documentation it is suggested that the default 
> "floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why 
> am I getting FLOAT? Is the default "floating_string_infer_type" not being 
> honoured?
>
>
>
> Currently in the config file "floating_string_infer_type" is commented out. 
> Should I explicitly set this to DOUBLE?
>
>
> Thanks
>
> Trevor Hart

Re: IoTDB quarter report (2024 Q2)

[Jialin Qiao]

Looks good!

Jialin Qiao

Xiangdong Huang  于2024年6月12日周三 09:11写道：
>
> Hi all,
>
> This is the draft of IoTDB quarter report
>
> ## Description:
> The mission of Apache IoTDB is the creation and maintenance of software 
> related
> to an IoT native database with high performance for data management and 
> analysis
>
> ## Project Status:
> Current project status: Ongoing with high activity.
> Issues for the board:
> - there are some trademark issues but the community has resolved them.
>
> ## Membership Data:
> Apache IoTDB was founded 2020-09-16 (4 years ago)
> There are currently 63 committers and 29 PMC members in this project.
> The Committer-to-PMC ratio is roughly 2:1.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
> - William Song was added as committer on 2024-03-22
> - Caiyin Yang was added as committer on 2024-06-12
>
> ## Project Activity:
> - Recent releases:
> IOTDB-1.3.2 is on the way.
> IOTDB-1.3.1 was released on 2024-04-22.
> IOTDB-1.3.0 was released on 2024-01-01.
> IOTDB-1.2.2 was released on 2023-10-15.
>
> - Main work of project:
> * We are working on the table model providing data to users in a table view
> in the same way as relational databases which will greatly reduce the
> learning curve of IoTDB.
> * Collaborate with the Apache StreamPipes community to improve the IoTDB 
> plugin
>  and explore other means on integration.
> * New query optimization rules are added to greatly improve the query
> performance of IoTDB, like PredicatePushDown and AggregatePushDown.
> * Explain Analyze tool which can be used to analyze specific query
> performance better and is a good tool for troubleshooting performance problems
>
>
> ## Community Health:
> Overall community health is good.
> - Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
> - the mailing list traffic increases (226 emails compared to 76)
>
> Best,
> ---
> Xiangdong Huang

IoTDB quarter report (2024 Q2)

[Xiangdong Huang]

Hi all,

This is the draft of IoTDB quarter report

## Description:
The mission of Apache IoTDB is the creation and maintenance of software related
to an IoT native database with high performance for data management and analysis

## Project Status:
Current project status: Ongoing with high activity.
Issues for the board:
- there are some trademark issues but the community has resolved them.

## Membership Data:
Apache IoTDB was founded 2020-09-16 (4 years ago)
There are currently 63 committers and 29 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Steve Yurong Su on 2023-09-28.
- William Song was added as committer on 2024-03-22
- Caiyin Yang was added as committer on 2024-06-12

## Project Activity:
- Recent releases:
IOTDB-1.3.2 is on the way.
IOTDB-1.3.1 was released on 2024-04-22.
IOTDB-1.3.0 was released on 2024-01-01.
IOTDB-1.2.2 was released on 2023-10-15.

- Main work of project:
* We are working on the table model providing data to users in a table view
in the same way as relational databases which will greatly reduce the
learning curve of IoTDB.
* Collaborate with the Apache StreamPipes community to improve the IoTDB plugin
 and explore other means on integration.
* New query optimization rules are added to greatly improve the query
performance of IoTDB, like PredicatePushDown and AggregatePushDown.
* Explain Analyze tool which can be used to analyze specific query
performance better and is a good tool for troubleshooting performance problems


## Community Health:
Overall community health is good.
- Joined  "Digital Economy-Urban Open Source Tour"  meetup in Shanghai.
- the mailing list traffic increases (226 emails compared to 76)

Best,
---
Xiangdong Huang

ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.

[Trevor Hart]

Hello Team



I have a customer using my application with IoTDB 1.3.0 in the backend.



Auto create schema is enabled as there is no fixed template. Multiple devices 
are pushing data to IotDB.



I have seen some ALIGN BY DEVICE queries fail with the error in the subject ie



ALIGN BY DEVICE: the data types of the same measurement column should be the 
same across devices.



The "show timeseries" shows that from one device the value has been inferred as 
FLOAT and another as DOUBLE.


Per the documentation it is suggested that the default 
"floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why am 
I getting FLOAT? Is the default "floating_string_infer_type" not being honoured?



Currently in the config file "floating_string_infer_type" is commented out. 
Should I explicitly set this to DOUBLE?


Thanks 

Trevor Hart

Re: AW: version about odbc

[Haonan Hou]

There is another example of the rest-client-c-example without the pom.xml that 
demonstrate how to use rest api with c language. Should we consider it as 
document example, or a "compilable" example? 

I prefer it is a compilable module and I think it is possible to add a pom.xml 
and make it as a maven module.

Haonan

On 2024/06/07 10:20:21 Christofer Dutz wrote:
> The ODBC is not really a “compilable” example.
> It contains a description how you can use a commercial product to wrap our 
> JDBC driver in order to produce an ODBC driver.
> Also does it contain a compilable C# project file, that demonstates the usage 
> of the ODBC API.
> 
> However even if the example would be compiled as part of the build, you would 
> still need to manually prepare the ODBC wrapper.
> 
> I would consider this module more documentation than “compilable code”.
> 
> Chris
> 
> 
> Von: Yuan Tian 
> Datum: Freitag, 7. Juni 2024 um 12:14
> An: dev 
> Betreff: version about odbc
> Hi all,
> 
> I found that in master branch, we've already upgraded the version to 
> 1.3.3-SNAPSHOT, but the examples about odbc module is missed?
> 
> [cid:ii_lx4j3xy50]
> 
> Best regards,
> -
> Yuan Tian
> 

AW: Session API issues

[Christofer Dutz]

I guess the simplest solution would simply be to return an error if not all 
paths in the query are valid. Right now I added a check on the client side that 
throws an exception if the size of the paths is different from that of the 
fields.

Chris



Von: Christofer Dutz 
Datum: Sonntag, 9. Juni 2024 um 16:40
An: dev@iotdb.apache.org 
Betreff: Session API issues
Hi all,

I have been playing around with the session API and have been stumbling into 
some pits, that I think would be worth addressing.

The biggest issue I have had recently is using the executeAggregationQuery 
method, passing in pairs of queries and aggregations.
However, I noticed that if for example I pass in 3 queries with the middle one 
being invalid, that I simply get back a 2-field response and I have no way of 
seeing which field was invalid.

In general, I think the passing in multiple lists with each the same size sort 
of feels a bit odd. Instead of passing in one list of objects which again have 
multiple options would be a preferrable approach.
Also, would this allow us to add these as references to the responses. So 
having a Field returned by RowRecord.getFields() have a reference to the query 
object, would make the API better to handle.
We wouldn’t have to pass the fields when returning the result, this could be 
something the client does completely in the client to avoid unnecessary 
payload. I do think however for this to work, that we would need some sort of 
error handling, passing back something like an ErrorField, if a query was wrong 
or anything similar.

Chris

Session API issues

[Christofer Dutz]

Hi all,

I have been playing around with the session API and have been stumbling into 
some pits, that I think would be worth addressing.

The biggest issue I have had recently is using the executeAggregationQuery 
method, passing in pairs of queries and aggregations.
However, I noticed that if for example I pass in 3 queries with the middle one 
being invalid, that I simply get back a 2-field response and I have no way of 
seeing which field was invalid.

In general, I think the passing in multiple lists with each the same size sort 
of feels a bit odd. Instead of passing in one list of objects which again have 
multiple options would be a preferrable approach.
Also, would this allow us to add these as references to the responses. So 
having a Field returned by RowRecord.getFields() have a reference to the query 
object, would make the API better to handle.
We wouldn’t have to pass the fields when returning the result, this could be 
something the client does completely in the client to avoid unnecessary 
payload. I do think however for this to work, that we would need some sort of 
error handling, passing back something like an ErrorField, if a query was wrong 
or anything similar.

Chris

AW: version about odbc

[Christofer Dutz]

The ODBC is not really a “compilable” example.
It contains a description how you can use a commercial product to wrap our JDBC 
driver in order to produce an ODBC driver.
Also does it contain a compilable C# project file, that demonstates the usage 
of the ODBC API.

However even if the example would be compiled as part of the build, you would 
still need to manually prepare the ODBC wrapper.

I would consider this module more documentation than “compilable code”.

Chris


Von: Yuan Tian 
Datum: Freitag, 7. Juni 2024 um 12:14
An: dev 
Betreff: version about odbc
Hi all,

I found that in master branch, we've already upgraded the version to 
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[cid:ii_lx4j3xy50]

Best regards,
-
Yuan Tian

version about odbc

[Yuan Tian]

Hi all,

I found that in master branch, we've already upgraded the version to
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[image: image.png]

Best regards,
-
Yuan Tian

Re: Prepare releasing 1.3.2?

[Haonan Hou]

Ok, the version of master branch has updated to 1.3.3-SNAPSHOT.
Once Ratis 3.1.0 released, I'll prepare the rc/1.3.2 branch. 

Best,
Haonan


On 2024/06/03 02:07:55 Xinyu Tan wrote:
> Hi haonan
> 
> +1 for bumping the version of master branch to 1.3.3-SNAPSHOT.
> 
> As for the 1.3.2 release, currently ratis is releasing there new version 
> 3.1.0[1], we can rely on the official Ratis 3.1.0 release in the open source 
> version IoTDB 1.3.2 instead of the snapshot release to conform to the 
> guidelines of the Apache community.
> 
> Best
> -
> Xinyu Tan
> 
> [1] https://lists.apache.org/thread/q8gdfz1j1l4qv1v24z4dxcg01j9qs28n
> 
> On 2024/06/03 01:44:17 Haonan Hou wrote:
> > Hi all,
> > 
> > IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over 
> > bug fixes and improvements. I think we can start the process of releasing 
> > new version.
> > 
> > If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the 
> > version of master branch to 1.3.3-SNAPSHOT.
> > 
> > BR,
> > Haonan
> > 
> 

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[ITAMI SHO]

Dear Junzhi Peng and Sicheng Yu,

Thank you for sharing the exciting news about the new consensus protocols, 
FastIoTConsensus and IoTV2Consensus. Your efforts and contributions to 
improving the IoTDB project are greatly appreciated. We look forward to 
exploring these new features and providing feedback.

Best regards,

Itami Sho

2024年6月3日 18:28，Junzhi Peng  写道：

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Device Level TTL for IoTDB

[Xinyu Tan]

Hi Peichen

I'm very excited to see the implementation of device-level TTL. Before your 
work, if users wanted to set different TTLs for different devices, they had to 
model them into many databases, which was very inconvenient. After your work, 
users can set different TTLs within a single database as they wish, greatly 
enhancing the user experience for those users with TTL needs.

I am thrilled to see your work being merged and look forward to it creating 
greater value for users.

Best

Xinyu Tan

On 2024/06/03 14:10:05 BensonChou wrote:
> Hello, everyone.
> 
> I am BensonChou(Peichen Zhou), a committer of IoTDB. Over the past few 
> months, I have implemented a new feature, which is device-level TTL, and I am 
> so excited to share this new feature with you.
> 
> Previously, IoTDB only supported database-level TTL. For devices with 
> different collection frequencies and data validity periods under a single 
> database, we want to use a more granular device-level TTL to manage the data 
> lifecycle and optimize disk space as much as possible.
> 
> The new version of IoTDB will support device-level TTL management, allowing 
> users to set, modify, and unset TTLs in bulk. When setting TTL, the system 
> will apply it to all devices that match the specified path. If multiple TTLs 
> exist on a device path, the TTL closest to the device node will take effect. 
> Once device data expires, it will no longer be queryable. While the data in 
> disk files is not guaranteed to be immediately deleted, it will eventually be 
> deleted.
> 
> The related SQL syntax isas following:
> 
> 
> 
> SET TTL TO path value. Eg: SET TTL TO ROOT.DB 3600;
> 
> UNSET TTL TO path. Eg: UNSET TTL TO ROOT.DB;
> 
> SHOW ALL TTL
> Additionally, we have introduced the keyword `INF` to represent an infinite 
> TTL value.You can achieve more diversified configurations to meet 
> different scenarios and requirements through the configuration parameters 
> `default_ttl_in_ms`, `ttl_check_interval`, `ttl_rule_capacity
> `, `max_expired_time`, and `expired_data_ratio` in the configuration file. 
> 
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> BensonChou
> 
> Reference:
> 
> https://github.com/apache/iotdb/pull/12122

Re: [Proposal] Device Level TTL for IoTDB

[Xinyu Tan]

Hi Peichen

I'm very excited to see the implementation of device-level TTL. Before your 
work, if users wanted to set different TTLs for different devices, they had to 
model them into many databases, which was very inconvenient. After your work, 
users can set different TTLs within a single database as they wish, greatly 
enhancing the user experience for those users with TTL needs.

I am thrilled to see your work being merged and look forward to it creating 
greater value for users.

Best

Xinyu Tan

On 2024/06/03 14:10:05 BensonChou wrote:
> Hello, everyone.
> 
> I am BensonChou(Peichen Zhou), a committer of IoTDB. Over the past few 
> months, I have implemented a new feature, which is device-level TTL, and I am 
> so excited to share this new feature with you.
> 
> Previously, IoTDB only supported database-level TTL. For devices with 
> different collection frequencies and data validity periods under a single 
> database, we want to use a more granular device-level TTL to manage the data 
> lifecycle and optimize disk space as much as possible.
> 
> The new version of IoTDB will support device-level TTL management, allowing 
> users to set, modify, and unset TTLs in bulk. When setting TTL, the system 
> will apply it to all devices that match the specified path. If multiple TTLs 
> exist on a device path, the TTL closest to the device node will take effect. 
> Once device data expires, it will no longer be queryable. While the data in 
> disk files is not guaranteed to be immediately deleted, it will eventually be 
> deleted.
> 
> The related SQL syntax isas following:
> 
> 
> 
> SET TTL TO path value. Eg: SET TTL TO ROOT.DB 3600;
> 
> UNSET TTL TO path. Eg: UNSET TTL TO ROOT.DB;
> 
> SHOW ALL TTL
> Additionally, we have introduced the keyword `INF` to represent an infinite 
> TTL value.You can achieve more diversified configurations to meet 
> different scenarios and requirements through the configuration parameters 
> `default_ttl_in_ms`, `ttl_check_interval`, `ttl_rule_capacity
> `, `max_expired_time`, and `expired_data_ratio` in the configuration file. 
> 
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> BensonChou
> 
> Reference:
> 
> https://github.com/apache/iotdb/pull/12122

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xinyu Tan]

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
> 

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xinyu Tan]

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
> 

Re: [DISCUSS] GH Actions optimizations

[Yuan Tian]

ok..., if so, I think it's really nice.

On Wed, Jun 5, 2024 at 8:28 PM Christofer Dutz 
wrote:

> Also is the settings currently done that it only skips any tests in
> non-master branches … so if there’s a PR on a branch, that will be
> optimized, but every commit going back to master should have all tests run.
>
> Chris
>
>
> Von: Christofer Dutz 
> Datum: Mittwoch, 5. Juni 2024 um 14:22
> An: dev@iotdb.apache.org 
> Betreff: AW: [DISCUSS] GH Actions optimizations
> Well even with the re-run option still failures get counted and a
> dashboard like this:
>
> https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
> Is a lot better than having to continuously go back to your PRs and click
> on “re-run failed tests”
>
> Chris
>
> Von: Yuan Tian 
> Datum: Mittwoch, 5. Juni 2024 um 03:06
> An: dev@iotdb.apache.org 
> Betreff: Re: [DISCUSS] GH Actions optimizations
> Hi Chris,
>
> That's really good news. About the `surefire` part, I think we need to
> discuss more about it because some of the previously failed tests have
> indeed helped us discover concurrency issues in IoTDB. I am concerned that
> doing this might reduce the likelihood of exposing such bugs.
>
>
> Best regards,
> 
> Yuan Tian
>

AW: [DISCUSS] GH Actions optimizations

[Christofer Dutz]

Also is the settings currently done that it only skips any tests in non-master 
branches … so if there’s a PR on a branch, that will be optimized, but every 
commit going back to master should have all tests run.

Chris


Von: Christofer Dutz 
Datum: Mittwoch, 5. Juni 2024 um 14:22
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] GH Actions optimizations
Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

AW: [DISCUSS] GH Actions optimizations

[Christofer Dutz]

Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

Re: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Jialin Qiao]

Thank Chris!

Jialin Qiao

Christofer Dutz  于2024年6月4日周二 17:28写道：
>
> Ok … artifacts should be available …
>
> Chris
>
> Von: Christofer Dutz 
> Datum: Dienstag, 4. Juni 2024 um 11:26
> An: dev@iotdb.apache.org 
> Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
> Thrift 0.20.0)
> So, the vote passes with 7 +1 votes at least 5 I could verify as binding 
> votes.
>
> In the future, it would be great if people would add their name/apacheId etc. 
> to the votes to make it easier to verify and count for the RM.
>
> Releasing the maven artifacts right away.
>
> Thanks for all those who voted.
>
> Chris
>
>
> Von: Christofer Dutz 
> Datum: Dienstag, 4. Juni 2024 um 09:37
> An: dev@iotdb.apache.org 
> Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
> 0.20.0)
> Yeah.. Currently at community over code... Was planning on doing this in a 
> break today.
>
> Gesendet von Outlook für Android
> 
> From: Haonan Hou 
> Sent: Monday, June 3, 2024 4:11:04 AM
> To: dev@iotdb.apache.org 
> Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
> 0.20.0)
>
> Hi Chris,
>
> I think 72 hours has passed. These artifacts can be released now?
>
> Thanks,
> Haonan
>
> On 2024/05/28 14:40:39 Christofer Dutz wrote:
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the 
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible 
> > again), I also updated the Thrift version and staged a release of Thrift 
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so 
> > it should be quick to review.
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be 
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then finish 
> > changing the main repo to use these artifacts.
> >
> >
> > Chris
> >

[DISCUSS] GH Actions optimizations

[Christofer Dutz]

At today’s sessions at community over code EU in a session about “Apache build 
observability” we got a little spotlight on IoTDB.
The develocity folks pointed out Apache IoTDB as being the one Apache project 
that provides the biggest room for improvement by enabling some of their 
build-optimization tools.

I have created a PR : https://github.com/apache/iotdb/pull/12654

This enables predictive test-selection which should prioritize tests that are 
most probable to be affected by changes and to skip ones that are more 
definitely not going to be affected.

I was actively approached by the develocity folks here at the conference and 
together we created a PR that should enable this for us.
Also did I take the liberty to also enable surefire to automatically re-run 
failed tests for a maximum of 3 times before actually failing a test.

These changes should allow reducing the load we have on GH Actions by 40% 
(according to the dashboards develocity has)

Also did they recommend utilizing build cache extensions, which should even 
more greatly reduce the build time and the load on the GH runners.

If anyone wants to play around with the dashboards for IoTDB …. Please have a 
look here.

https://ge.apache.org/scans?search.rootProjectNames=*IoTDB*

There are very interesting dashboards that show us stuff like flaky tests:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY

Chris

AW: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Ok … artifacts should be available …

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 11:26
An: dev@iotdb.apache.org 
Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
Thrift 0.20.0)
So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

[RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xiangpeng Hu]

Hello Junzhi,


I would like to extend my heartfelt thanks to you and Sicheng Yu for your 
FastIoTConsensus and IoTV2Consensus protocols. It is a significant step forward 
for IoTDB.


I am well aware that the existing IoTConsensus has been performing 
exceptionally well in most scenarios, yet it has encountered availability 
issues. I appreciate your insight into the need to further sacrifice 
consistency for higher synchronization performance, which is a forward-thinking 
approach. I am also very excited about the opportunity to participate in the 
development and testing of these new consensus protocols.


Lastly, your hard work and innovation are greatly appreciated, and I look 
forward to seeing the real-world performance of FastIoTConsensus and 
IoTV2Consensus.


Best regards,
Xiangpeng Hu
 Replied Message 
| From | Junzhi Peng |
| Date | 6/3/2024 18:28 |
| To |  |
| Subject | [Proposal] New Consensus Protocals for IoTDB DataRegion |
Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Merge configuration files to iotdb-system.properties

[wenwei shu]

Hi,

Most of the changes in this PR are to modify the configuration file itself
and some script tools that will read the configuration file. Codes for
accessing the options are not merged.

Best regards,
Wenwei Shu

Christofer Dutz  于2024年6月1日周六 15:32写道：

> However,
>
> I think it would be good If the code for accessing the options would not
> be merged.
> Because that would move the project to being more of a mono-module type of
> project.
>
> Chris
>
> Von: Jialin Qiao 
> Datum: Samstag, 1. Juni 2024 um 08:07
> An: dev@iotdb.apache.org 
> Betreff: Re: [Proposal] Merge configuration files to
> iotdb-system.properties
> Hi,
>
> Good feature!  This will make it easy for users to set configs.
>
> Jialin Qiao
>
> wenwei shu  于2024年5月31日周五 21:23写道：
> >
> >  Hello everyone,
> >
> > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > working on merging `iotdb-confignode.properties`,
> > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > `iotdb-system.properties` file. For old version users who upgrade to a
> new
> > version, if they don't create the new configuration file themselves
> during
> > the upgrade, IoTDB will generate the new configuration file based on
> > several old configuration files after it is started. You can check this
> PR
> > to get more details: https://github.com/apache/iotdb/pull/12570.
> > Thank you for your reading.
> >
> > Best regards,
> > Wenwei Shu
>

[Proposal] New Consensus Protocals for IoTDB DataRegion

[Junzhi Peng]

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Jialin Qiao]

Hi,

> I think it would be good If the code for accessing the options would not be 
> merged.

+1, I think the code is not merged in Wenwei's PR.

Jialin Qiao

Caiyin Yang  于2024年6月3日周一 10:27写道：
>
> Hi Xinyu,
>
> If that's the case, then this merge seems to be very user-friendly.
>
> Best
> Caiyin Yang
>
> Xinyu Tan  于2024年6月3日周一 10:22写道：
>
> > Hi, Caiyin
> >
> > Following Wenwei's work, we plan to introduce an
> > iotdb-system.properties.template file that will contain all configuration
> > items, relevant comments, and optional settings, similar to the latest user
> > configuration manual. In the future, the iotdb-system.properties file will
> > only include explicitly configured items by the user (such as IP addresses
> > or manually configured storage engine parameters). Compared to the current
> > method of finding and modifying configurations in the lengthy
> > iotdb-confignode.properties and iotdb-datanode.properties files, this
> > approach seems more convenient. If you have any good ideas, feel free to
> > discuss your preferred method further!
> >
> > Best
> > 
> > Xinyu Tan
> >
> > On 2024/06/01 09:18:48 ycy wi wrote:
> > > Hi wenwei,
> > >
> > > I am curious whether this will ultimately lead to an oversized or
> > redundant configuration file. I might just want to modify some
> > configuration items in the dataNode. In theory, I only need to pay
> > attention to the configuration items in iotdb-datanode.properties. Will the
> > merge cause any confusion due to similar configuration items in
> > iotdb-confignode.properties?
> > >
> > > Best
> > > CaiyinYang
> > >
> > >
> > >
> > >
> > >
> > > > 2024年5月31日 21:20，wenwei shu  写道：
> > > >
> > > > Hello everyone,
> > > >
> > > > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > > > working on merging `iotdb-confignode.properties`,
> > > > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > > > `iotdb-system.properties` file. For old version users who upgrade to a
> > new
> > > > version, if they don't create the new configuration file themselves
> > during
> > > > the upgrade, IoTDB will generate the new configuration file based on
> > > > several old configuration files after it is started. You can check
> > this PR
> > > > to get more details: https://github.com/apache/iotdb/pull/12570.
> > > > Thank you for your reading.
> > > >
> > > > Best regards,
> > > > Wenwei Shu
> > >
> > >
> >

Re: [Proposal] Load Tsfile Functionality Optimisation

[ITAMI SHO]

Hi Xiangzhi,

Thank you for sharing your work on the load tsfile feature. I am currently 
working on the Load function as well, and I have a question for you. Have you 
considered enhancing the Load function directly within the core code, rather 
than relying on the script to load tsfiles one by one? Additionally , do you 
think there would be a significant performance difference between the original 
batch load method and your script of loading files individually?

Your insights on this matter would be very valuable, and I believe it could 
lead to further optimizations!

Best regards,
Itami Sho

> 2024年5月30日 11:11，孟祥志  写道：
> 
> Hello everyone,
> 
> I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
> with you a feature that I have been working on for the past few months.
> 
> load tsfile has always been an important feature for IoTDB, but due to the 
> original design, he has some flaws in it, when load a folder fails to import 
> some tsfile, it may cause the whole task fails, which is obviously a not very 
> user-friendly design, so I decided to modify his.
> 
> I'll modify the original load tsfile script to traverse the entire folder 
> first, find all the files, and then use the LOAD function to import 
> individual files; and I've added a few interesting changes along the way, 
> modifying the successful operation parameter to make it easier for people to 
> deal with successful files, and added a parameter for failed files to make it 
> easier for people to find those failed files, and finally added a new number 
> of threads to make the LOAD function work more quickly.
> 
> 
> I hope you are interested in this feature and would like to participate in 
> the development and testing. You can also leave your comments and suggestions 
> in this thread. Appreciate any suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> Xiangzhi Meng
> 
> Reference:
> https://github.com/apache/iotdb/pull/12541

Re: [Proposal] Active metaData Query

[ycy wi]

Hi ITAMI,

‘Active’ refers to the data that can be seen by query in the specified time 
range, which you can also understand as data that has been written in the 
specified time and not deleted later. 

Best
CaiyinYang





> 2024年6月3日 15:40，ITAMI SHO  写道：
> 
> question

Re: [Proposal] Active metaData Query

[ITAMI SHO]

Hi Caiyin,

Thank you for your contribution, I have a question regrading the definition of 
active metadata. Could you please provide more details on what constitutes 
“active metadata” in this context? Understanding this definition better will 
help us utilize this new feature!

Thank you again for your hard work and contribution.

Best regards,
Itami Sho

> 2024年6月3日 11:45，Caiyin Yang  写道：
> 
> Hello everyone, I'm Caiyin Yang, a contributor to IoTDB.
> 
> I have implemented an active metadata query feature in IoTDB in the past
> few months, which is an extension based on the original *SHOW/COUNT
> DEVICES/TIMESERIES *SQL. You only need to specify the time condition in the
> SQL statement (*for example: show timeseries where time > xx and time < xx*)
> to know which metadata exists in that interval.
> 
> The active metadata query first uses the statistical information in memory
> for optimization, and only performs disk scanning when the statistical
> information cannot be used.
> 
> To optimize the performance of disk scanning, I implemented the sequential
> scan of TsFile files called RegionScan.
> 
> In the scenario of full scanning of active data, compared with the original
> series query, there is better performance (preliminary tests have shown an
> improvement of nearly 20% - 30%). I will then carry out more comprehensive
> performance testing scenarios and welcome everyone to provide any feedback.
> 
> *PR Links:*
> https://github.com/apache/iotdb/pull/12446
> https://github.com/apache/iotdb/pull/12539
> https://github.com/apache/iotdb/pull/12607

Re: Canonical the semantics of auto create schema function

[ITAMI SHO]

Hi Yongzao Dan,

Thank you for your recent contribution to the Apache IoTDB community. Your work 
on clarifying the semantics of the auto-create schema function and addressing 
the associated issues has been incredibly valuable to IoTDB. Your dedication 
and attention to detail are greatly appreciated!

Best regards,
Itami Sho

> 2024年6月1日 12:43，Yongzao Dan <532741...@qq.com.INVALID> 写道：
> 
> Hi everyone,
> 
> 
>  I'm Yongzao Dan, a committer in the Apache IoTDB community.
> 
> 
>  The semantics ofauto create schema function has been confused 
> since the community updated the distribution version, where the IoTDB will 
> automatically create both corresponding database and time series schema when 
> enable this function and process write request. But the IoTDB will still 
> create a database when disable this function and process write request. In 
> fact, neither the database nor the time series schema should be created when 
> disable this function.
> 
> 
>  Thus I commited a PR to fix this neglected semantical bug during 
> version iteration. Added a new IT as well to prevent this function being 
> incorrectly modified in future 
> update:https://github.com/apache/iotdb/pull/12590
> 
> 
> Warm regards,
> Yongzao Dan

[Proposal] Active metaData Query

[Caiyin Yang]

Hello everyone, I'm Caiyin Yang, a contributor to IoTDB.

I have implemented an active metadata query feature in IoTDB in the past
few months, which is an extension based on the original *SHOW/COUNT
DEVICES/TIMESERIES *SQL. You only need to specify the time condition in the
SQL statement (*for example: show timeseries where time > xx and time < xx*)
to know which metadata exists in that interval.

The active metadata query first uses the statistical information in memory
for optimization, and only performs disk scanning when the statistical
information cannot be used.

To optimize the performance of disk scanning, I implemented the sequential
scan of TsFile files called RegionScan.

In the scenario of full scanning of active data, compared with the original
series query, there is better performance (preliminary tests have shown an
improvement of nearly 20% - 30%). I will then carry out more comprehensive
performance testing scenarios and welcome everyone to provide any feedback.

*PR Links:*
https://github.com/apache/iotdb/pull/12446
https://github.com/apache/iotdb/pull/12539
https://github.com/apache/iotdb/pull/12607

Re: [Proposal] Add the feature of importing and exporting metadata by CSV file

[Xiangpeng Hu]

Hi Rong Li,


This is a great new feature. Thank you for working on this. Registering 
metadata is usually a time-consuming operation in the case of massive 
sequences. I am curious about the efficiency of importing and exporting 
metadata. Have the scripts been tested on large-scale sequences? If so, could 
you share the relevant data?


Xiangpeng Hu
 Replied Message 
| From | 李蓉 |
| Date | 6/3/2024 10:04 |
| To | dev@iotdb.apache.org |
| Subject | [Proposal] Add the feature of importing and exporting metadata by 
CSV file |
Hello Everyone,
 
I am Rong Li. I am excited to share with you a feature that I have been working 
on for the past few months.

As we all know, IoTDB supports various forms of importing and exporting data, 
but does not support importing and exporting metadata. In practical use, there 
are also many scenarios that require importing and exporting metadata, so I 
added a metadata import and export script to achieve this function.
 
I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.
 
Thank you for your attention and support.

Best regards,

Rong Li

Reference:
https://github.com/apache/iotdb/pull/12623

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Caiyin Yang]

Hi Xinyu,

If that's the case, then this merge seems to be very user-friendly.

Best
Caiyin Yang

Xinyu Tan  于2024年6月3日周一 10:22写道：

> Hi, Caiyin
>
> Following Wenwei's work, we plan to introduce an
> iotdb-system.properties.template file that will contain all configuration
> items, relevant comments, and optional settings, similar to the latest user
> configuration manual. In the future, the iotdb-system.properties file will
> only include explicitly configured items by the user (such as IP addresses
> or manually configured storage engine parameters). Compared to the current
> method of finding and modifying configurations in the lengthy
> iotdb-confignode.properties and iotdb-datanode.properties files, this
> approach seems more convenient. If you have any good ideas, feel free to
> discuss your preferred method further!
>
> Best
> 
> Xinyu Tan
>
> On 2024/06/01 09:18:48 ycy wi wrote:
> > Hi wenwei,
> >
> > I am curious whether this will ultimately lead to an oversized or
> redundant configuration file. I might just want to modify some
> configuration items in the dataNode. In theory, I only need to pay
> attention to the configuration items in iotdb-datanode.properties. Will the
> merge cause any confusion due to similar configuration items in
> iotdb-confignode.properties?
> >
> > Best
> > CaiyinYang
> >
> >
> >
> >
> >
> > > 2024年5月31日 21:20，wenwei shu  写道：
> > >
> > > Hello everyone,
> > >
> > > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > > working on merging `iotdb-confignode.properties`,
> > > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > > `iotdb-system.properties` file. For old version users who upgrade to a
> new
> > > version, if they don't create the new configuration file themselves
> during
> > > the upgrade, IoTDB will generate the new configuration file based on
> > > several old configuration files after it is started. You can check
> this PR
> > > to get more details: https://github.com/apache/iotdb/pull/12570.
> > > Thank you for your reading.
> > >
> > > Best regards,
> > > Wenwei Shu
> >
> >
>

Re: Canonical the semantics of auto create schema function

[Xinyu Tan]

Hi rongzhao

It's really cool that with this change we're able to provide more explicit 
auto-creation semantics to our users.

Best
--
Xinyu Tan

On 2024/06/01 04:43:05 Yongzao Dan wrote:
> Hi everyone,
> 
> 
>  I'm Yongzao Dan, a committer in the Apache IoTDB community.
> 
> 
>  The semantics ofauto create schema function has been confused 
> since the community updated the distribution version, where the IoTDB will 
> automatically create both corresponding database and time series schema when 
> enable this function and process write request. But the IoTDB will still 
> create a database when disable this function and process write request. In 
> fact, neither the database nor the time series schema should be created when 
> disable this function.
> 
> 
>  Thus I commited a PR to fix this neglected semantical bug during 
> version iteration. Added a new IT as well to prevent this function being 
> incorrectly modified in future 
> update:https://github.com/apache/iotdb/pull/12590
> 
> 
> Warm regards,
> Yongzao Dan

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Xinyu Tan]

Hi, Caiyin

Following Wenwei's work, we plan to introduce an 
iotdb-system.properties.template file that will contain all configuration 
items, relevant comments, and optional settings, similar to the latest user 
configuration manual. In the future, the iotdb-system.properties file will only 
include explicitly configured items by the user (such as IP addresses or 
manually configured storage engine parameters). Compared to the current method 
of finding and modifying configurations in the lengthy 
iotdb-confignode.properties and iotdb-datanode.properties files, this approach 
seems more convenient. If you have any good ideas, feel free to discuss your 
preferred method further!

Best

Xinyu Tan

On 2024/06/01 09:18:48 ycy wi wrote:
> Hi wenwei,
> 
> I am curious whether this will ultimately lead to an oversized or redundant 
> configuration file. I might just want to modify some configuration items in 
> the dataNode. In theory, I only need to pay attention to the configuration 
> items in iotdb-datanode.properties. Will the merge cause any confusion due to 
> similar configuration items in iotdb-confignode.properties?
> 
> Best
> CaiyinYang
> 
> 
> 
> 
> 
> > 2024年5月31日 21:20，wenwei shu  写道：
> > 
> > Hello everyone,
> > 
> > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > working on merging `iotdb-confignode.properties`,
> > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > `iotdb-system.properties` file. For old version users who upgrade to a new
> > version, if they don't create the new configuration file themselves during
> > the upgrade, IoTDB will generate the new configuration file based on
> > several old configuration files after it is started. You can check this PR
> > to get more details: https://github.com/apache/iotdb/pull/12570.
> > Thank you for your reading.
> > 
> > Best regards,
> > Wenwei Shu
> 
> 

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Haonan Hou]

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris
> 

Re: Prepare releasing 1.3.2?

[Xinyu Tan]

Hi haonan

+1 for bumping the version of master branch to 1.3.3-SNAPSHOT.

As for the 1.3.2 release, currently ratis is releasing there new version 
3.1.0[1], we can rely on the official Ratis 3.1.0 release in the open source 
version IoTDB 1.3.2 instead of the snapshot release to conform to the 
guidelines of the Apache community.

Best
-
Xinyu Tan

[1] https://lists.apache.org/thread/q8gdfz1j1l4qv1v24z4dxcg01j9qs28n

On 2024/06/03 01:44:17 Haonan Hou wrote:
> Hi all,
> 
> IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over bug 
> fixes and improvements. I think we can start the process of releasing new 
> version.
> 
> If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the 
> version of master branch to 1.3.3-SNAPSHOT.
> 
> BR,
> Haonan
> 

[Proposal] Add the feature of importing and exporting metadata by CSV file

[李蓉]

Hello Everyone,
 
I am Rong Li. I am excited to share with you a feature that I have been working 
on for the past few months.

As we all know, IoTDB supports various forms of importing and exporting data, 
but does not support importing and exporting metadata. In practical use, there 
are also many scenarios that require importing and exporting metadata, so I 
added a metadata import and export script to achieve this function.
 
I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.
 
Thank you for your attention and support.

Best regards,

Rong Li

Reference:
https://github.com/apache/iotdb/pull/12623

Re: [Proposal] Merge configuration files to iotdb-system.properties

[wenwei shu]

Hi caiyin,

After merging, there are indeed more configuration items in one file, but i
think it will not cause confusion. Most configuration item names in
`iotdb-confignode.properties` or `iotdb-datanode.properties` begin with
`cn` or `dn`.

Best
Wenwei Shu

ycy wi  于2024年6月1日周六 17:19写道：

> Hi wenwei,
>
> I am curious whether this will ultimately lead to an oversized or
> redundant configuration file. I might just want to modify some
> configuration items in the dataNode. In theory, I only need to pay
> attention to the configuration items in iotdb-datanode.properties. Will the
> merge cause any confusion due to similar configuration items in
> iotdb-confignode.properties?
>
> Best
> CaiyinYang
>
>
>
>
>
> > 2024年5月31日 21:20，wenwei shu  写道：
> >
> > Hello everyone,
> >
> > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > working on merging `iotdb-confignode.properties`,
> > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > `iotdb-system.properties` file. For old version users who upgrade to a
> new
> > version, if they don't create the new configuration file themselves
> during
> > the upgrade, IoTDB will generate the new configuration file based on
> > several old configuration files after it is started. You can check this
> PR
> > to get more details: https://github.com/apache/iotdb/pull/12570.
> > Thank you for your reading.
> >
> > Best regards,
> > Wenwei Shu
>
>

Re: Prepare releasing 1.3.2?

[Yuan Tian]

Hi haonan,


+1(binding), looking forward to exciting v1.3.2

Best regards,

Yuan Tian



On Mon, Jun 3, 2024 at 9:45 AM Haonan Hou  wrote:

> Hi all,
>
> IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over
> bug fixes and improvements. I think we can start the process of releasing
> new version.
>
> If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the
> version of master branch to 1.3.3-SNAPSHOT.
>
> BR,
> Haonan
>

Prepare releasing 1.3.2?

[Haonan Hou]

Hi all,

IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over bug 
fixes and improvements. I think we can start the process of releasing new 
version.

If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the version 
of master branch to 1.3.3-SNAPSHOT.

BR,
Haonan

Re: [PR] chore(deps): Bump spring.version from 5.3.33 to 5.3.34 [iotdb-extras]

[via GitHub]

dependabot[bot] closed pull request #4: chore(deps): Bump spring.version from 
5.3.33 to 5.3.34
URL: https://github.com/apache/iotdb-extras/pull/4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] chore(deps): Bump spring.version from 5.3.33 to 5.3.36 [iotdb-extras]

[via GitHub]

dependabot[bot] opened a new pull request, #8:
URL: https://github.com/apache/iotdb-extras/pull/8

   Bumps `spring.version` from 5.3.33 to 5.3.36.
   Updates `org.springframework:spring-tx` from 5.3.33 to 5.3.36
   
   Release notes
   Sourced from https://github.com/spring-projects/spring-framework/releases;>org.springframework:spring-tx's
 releases.
   
   v5.3.36
   :lady_beetle: Bug Fixes
   
   Overridden aspect method runs twice https://redirect.github.com/spring-projects/spring-framework/issues/32868;>#32868
   @DateTimeFormat(iso = DateTimeFormat.ISO.DATE\_TIME) cannot 
convert UTC without milliseconds to java.util.Date https://redirect.github.com/spring-projects/spring-framework/issues/32860;>#32860
   Spring AOP fails against registered @Configurable aspect https://redirect.github.com/spring-projects/spring-framework/issues/32840;>#32840
   
   v5.3.35
   :star: New Features
   
   Accept ajc-compiled @Aspect classes for Spring AOP proxy 
usage https://redirect.github.com/spring-projects/spring-framework/issues/32818;>#32818
   
   :lady_beetle: Bug Fixes
   
   DeferredQueryInvocationHandler fails to unwrap QuerySqmImpl class 
outside of transaction https://redirect.github.com/spring-projects/spring-framework/issues/32770;>#32770
   MergedAnnotations search does not find container for 
repeatable annotation https://redirect.github.com/spring-projects/spring-framework/issues/32751;>#32751
   AnnotationConfigWebApplicationContext should propagate 
ApplicationStartup to BeanFactory https://redirect.github.com/spring-projects/spring-framework/issues/32749;>#32749
   Ignore non-String keys in 
PropertiesPropertySource.getPropertyNames() https://redirect.github.com/spring-projects/spring-framework/issues/32744;>#32744
   multiple subscribers not supported when using WebClient 
exchange https://redirect.github.com/spring-projects/spring-framework/issues/32728;>#32728
   Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Undertow 
2.3.10 https://redirect.github.com/spring-projects/spring-framework/issues/32698;>#32698
   
   :notebook_with_decorative_cover: Documentation
   
   Correct documentation on streaming with MockMvcWebTestClient https://redirect.github.com/spring-projects/spring-framework/issues/32723;>#32723
   Update links to HttpOnly documentation at OWASP in 
ResponseCookie https://redirect.github.com/spring-projects/spring-framework/issues/32668;>#32668
   
   :hammer: Dependency Upgrades
   
   Upgrade to Reactor 2020.0.44 https://redirect.github.com/spring-projects/spring-framework/issues/32788;>#32788
   
   v5.3.34
   :star: New Features
   
   Log column type for limited support message in 
JdbcUtils.getResultSetValue https://redirect.github.com/spring-projects/spring-framework/issues/32603;>#32603
   Avoid additional unnecessary Annotation array cloning in 
TypeDescriptor https://redirect.github.com/spring-projects/spring-framework/issues/32477;>#32477
   Avoid cloning empty Annotation array in 
TypeDescriptor https://redirect.github.com/spring-projects/spring-framework/issues/32466;>#32466
   
   :lady_beetle: Bug Fixes
   
   Refine scheme, userinfo, host and port parsing in UriComponentsBuilder 
https://redirect.github.com/spring-projects/spring-framework/issues/32618;>#32618
   MethodIntrospector.selectMethods() fails to detect bridge 
methods across ApplicationContexts https://redirect.github.com/spring-projects/spring-framework/issues/32588;>#32588
   JmsUtils.commitIfNecessary catches and ignores JMS 
IllegalStateException, losing message with ActiveMQ Artemis https://redirect.github.com/spring-projects/spring-framework/issues/32480;>#32480
   Consistently apply TaskDecorator to ManagedExecutorService as well https://redirect.github.com/spring-projects/spring-framework/issues/32457;>#32457
   
   :hammer: Dependency Upgrades
   
   Upgrade to Reactor 2020.0.43 https://redirect.github.com/spring-projects/spring-framework/issues/32594;>#32594
   
   
   
   
   Commits
   
   https://github.com/spring-projects/spring-framework/commit/2399806d5f0a0e5fcecd8dd57fbae493f8b8da3f;>2399806
 Release v5.3.36
   https://github.com/spring-projects/spring-framework/commit/eff9f5b92a0f4daa65f6f6276b0915879b24497c;>eff9f5b
 Select most specific advice method in case of override
   https://github.com/spring-projects/spring-framework/commit/f1fed9c174705aa4f2817c94fc519f17b670976b;>f1fed9c
 Polishing
   https://github.com/spring-projects/spring-framework/commit/d2e7cf439545cac51a464956017f1db433990eba;>d2e7cf4
 Default fallback parsing for UTC without milliseconds
   https://github.com/spring-projects/spring-framework/commit/47a7abee8f223266bfdd012a03b88f2f7b8f5a32;>47a7abe
 Polishing
   https://github.com/spring-projects/spring-framework/commit/ef2c140d3cbb5fe7dd53afd65c6ca061f617911a;>ef2c140
 Defensively catch and log pointcut parsing exceptions
   https://github.com/spring-projects/spring-framework/commit/de6cf845b102cffd81dd52013cf9dc3cfe0d04df;>de6cf84
 Next development 

Re: [PR] chore(deps): Bump spring.version from 5.3.33 to 5.3.34 [iotdb-extras]

[via GitHub]

dependabot[bot] commented on PR #4:
URL: https://github.com/apache/iotdb-extras/pull/4#issuecomment-2143521149

   Superseded by #8.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [Proposal] Merge configuration files to iotdb-system.properties

[ycy wi]

Hi wenwei,

I am curious whether this will ultimately lead to an oversized or redundant 
configuration file. I might just want to modify some configuration items in the 
dataNode. In theory, I only need to pay attention to the configuration items in 
iotdb-datanode.properties. Will the merge cause any confusion due to similar 
configuration items in iotdb-confignode.properties?

Best
CaiyinYang





> 2024年5月31日 21:20，wenwei shu  写道：
> 
> Hello everyone,
> 
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
> 
> Best regards,
> Wenwei Shu

AW: [Proposal] Merge configuration files to iotdb-system.properties

[Christofer Dutz]

However,

I think it would be good If the code for accessing the options would not be 
merged.
Because that would move the project to being more of a mono-module type of 
project.

Chris

Von: Jialin Qiao 
Datum: Samstag, 1. Juni 2024 um 08:07
An: dev@iotdb.apache.org 
Betreff: Re: [Proposal] Merge configuration files to iotdb-system.properties
Hi,

Good feature!  This will make it easy for users to set configs.

Jialin Qiao

wenwei shu  于2024年5月31日周五 21:23写道：
>
>  Hello everyone,
>
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
>
> Best regards,
> Wenwei Shu

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Jialin Qiao]

Hi,

Good feature!  This will make it easy for users to set configs.

Jialin Qiao

wenwei shu  于2024年5月31日周五 21:23写道：
>
>  Hello everyone,
>
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
>
> Best regards,
> Wenwei Shu

[Proposal] Merge configuration files to iotdb-system.properties

[wenwei shu]

 Hello everyone,

I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
working on merging `iotdb-confignode.properties`,
`iotdb-datanode.properties` and `iotdb-common.properties` into a new
`iotdb-system.properties` file. For old version users who upgrade to a new
version, if they don't create the new configuration file themselves during
the upgrade, IoTDB will generate the new configuration file based on
several old configuration files after it is started. You can check this PR
to get more details: https://github.com/apache/iotdb/pull/12570.
Thank you for your reading.

Best regards,
Wenwei Shu

AW: [Proposal] Data Subscription Client on IoTDB

[Christofer Dutz]

Hi Yuchen,

do I understand it correctly, that with this, a client could subscribe to data 
and receive this in TsFile format? This plus the ability of an IoTDB instance 
subscribing to such a TsFile subscription would perfectly align with my plans 
for 2024 allowing TsFile libraries on PLCs sending data to IoTDB as well as 
simplify data-collection gateways.

Chris

Von: Yuchen Ding 
Datum: Freitag, 31. Mai 2024 um 10:14
An: dev@iotdb.apache.org 
Betreff: Re: [Proposal] Data Subscription Client on IoTDB
Hello everyone,

I am VGalaxies. Recently, I have been working on providing TsFile subscription 
support for IoTDB data subscription client. The background of this feature is 
to achieve data file export and backup for multi-replica clusters using data 
subscription. Using the existing data subscription client, the subscribed data 
is in the form of SessionDataSet. The server needs to parse TsFile, and the 
client needs to rewrite TsFile. Raw data is transmitted over the network, 
without leveraging the high compression features of TsFile. Therefore, we hope 
to support exporting TsFile using the TsFile client SDK in data subscriptions.

In terms of functionality, TsFile subscription support for the data 
subscription client includes three steps. First, create a topic with a data 
presentation format of TsFile, by specifying the topic format as TsFileHandler. 
Second, when creating a consumer, specify the directory where the subscribed 
TsFile will be saved using the fileSaveDir parameter. Third, obtain the 
corresponding handler based on the type of SubscriptionMessage, which is 
SubscriptionTsFileHandler. SubscriptionTsFileHandler encapsulates operations 
like cp, mv, and rm from the Java standard library, and can also iterate data 
through the TsFile SDK, i.e., TsFileReader. Users can achieve TsFile 
subscription with minimal configuration.

Technically, we have restructured the Message Payload of the pipeSubscribe RPC 
poll type to support the transmission of data in both SessionDataSet and TsFile 
formats. The transmission of a TsFile file is divided into multiple events, 
including tsfile init event, tsfile piece event, and tsfile seal event. The 
reliable transmission of TsFile files is achieved through the interaction of 
these events between the DN side and the client side.

Currently, TsFile subscription still has some limitations. For example, when 
the format of the topic is TsFileHandler, there are certain constraints on its 
path and time configuration. We will continue to optimize these issues in 
future iterations.

I have initially implemented support for TsFile subscription in this PR[1]. I 
hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
VGalaxies

Reference:
1. https://github.com/apache/iotdb/pull/12326

On 2024/04/08 03:06:59 VGalaxies wrote:
> Hello everyone,
>
> I am VGalaxies, a new contributor to Apache IoTDB. I am excited to
> share with you a new feature that I have been working on for the past
> few months.
>
> The data subscription client is a new way to access data within IoTDB,
> distinct from the traditional method of querying data using SQL-like
> syntax. In scenarios where real-time data, quick response to data
> changes, and building highly event-driven systems are required, data
> subscription has greater advantages over data querying. For example,
> in the following two scenarios:
>
> 1. Replace extensive polling queries for large amounts of data: Avoid
> significant impacts on the performance of existing systems when
> querying frequently or when there are many data points. Also, avoid
> problems with determining the query scope and ensure downstream
> receives accurate full data.
> 2. Facilitate downstream system integration: It's easier to integrate
> with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc.
> There's no need to customize the logic of IoTDB's data change capture
> for each big data component separately, simplifying integration
> component design and making it easier for users.
>
> The IoTDB subscription client references some features defined by some
> message queue products like Kafka. It consists of 3 core concepts:
> Topic, Consumer, and Consumer Group.
>
> - Topic is a logical concept used by the IoTDB subscription client to
> classify data, serving as a channel for data publication. Producers
> publish data to specific topics, while consumers subscribe to these
> topics to receive related data. In the IoTDB subscription client,
> topics describe the sequence characteristics, time characteristics,
> presentation forms, and optional custom processing logic of subscribed
> data.
> - Consumer is an application or service in the IoTDB subscription
> client responsible for receiving and processing data published 

Re: [Proposal] Data Subscription Client on IoTDB

[Yuchen Ding]

Hello everyone,

I am VGalaxies. Recently, I have been working on providing TsFile subscription 
support for IoTDB data subscription client. The background of this feature is 
to achieve data file export and backup for multi-replica clusters using data 
subscription. Using the existing data subscription client, the subscribed data 
is in the form of SessionDataSet. The server needs to parse TsFile, and the 
client needs to rewrite TsFile. Raw data is transmitted over the network, 
without leveraging the high compression features of TsFile. Therefore, we hope 
to support exporting TsFile using the TsFile client SDK in data subscriptions.

In terms of functionality, TsFile subscription support for the data 
subscription client includes three steps. First, create a topic with a data 
presentation format of TsFile, by specifying the topic format as TsFileHandler. 
Second, when creating a consumer, specify the directory where the subscribed 
TsFile will be saved using the fileSaveDir parameter. Third, obtain the 
corresponding handler based on the type of SubscriptionMessage, which is 
SubscriptionTsFileHandler. SubscriptionTsFileHandler encapsulates operations 
like cp, mv, and rm from the Java standard library, and can also iterate data 
through the TsFile SDK, i.e., TsFileReader. Users can achieve TsFile 
subscription with minimal configuration.

Technically, we have restructured the Message Payload of the pipeSubscribe RPC 
poll type to support the transmission of data in both SessionDataSet and TsFile 
formats. The transmission of a TsFile file is divided into multiple events, 
including tsfile init event, tsfile piece event, and tsfile seal event. The 
reliable transmission of TsFile files is achieved through the interaction of 
these events between the DN side and the client side.

Currently, TsFile subscription still has some limitations. For example, when 
the format of the topic is TsFileHandler, there are certain constraints on its 
path and time configuration. We will continue to optimize these issues in 
future iterations.

I have initially implemented support for TsFile subscription in this PR[1]. I 
hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
VGalaxies

Reference:
1. https://github.com/apache/iotdb/pull/12326

On 2024/04/08 03:06:59 VGalaxies wrote:
> Hello everyone,
> 
> I am VGalaxies, a new contributor to Apache IoTDB. I am excited to
> share with you a new feature that I have been working on for the past
> few months.
> 
> The data subscription client is a new way to access data within IoTDB,
> distinct from the traditional method of querying data using SQL-like
> syntax. In scenarios where real-time data, quick response to data
> changes, and building highly event-driven systems are required, data
> subscription has greater advantages over data querying. For example,
> in the following two scenarios:
> 
> 1. Replace extensive polling queries for large amounts of data: Avoid
> significant impacts on the performance of existing systems when
> querying frequently or when there are many data points. Also, avoid
> problems with determining the query scope and ensure downstream
> receives accurate full data.
> 2. Facilitate downstream system integration: It's easier to integrate
> with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc.
> There's no need to customize the logic of IoTDB's data change capture
> for each big data component separately, simplifying integration
> component design and making it easier for users.
> 
> The IoTDB subscription client references some features defined by some
> message queue products like Kafka. It consists of 3 core concepts:
> Topic, Consumer, and Consumer Group.
> 
> - Topic is a logical concept used by the IoTDB subscription client to
> classify data, serving as a channel for data publication. Producers
> publish data to specific topics, while consumers subscribe to these
> topics to receive related data. In the IoTDB subscription client,
> topics describe the sequence characteristics, time characteristics,
> presentation forms, and optional custom processing logic of subscribed
> data.
> - Consumer is an application or service in the IoTDB subscription
> client responsible for receiving and processing data published to
> specific topics. Consumers retrieve data from the queue and perform
> corresponding processing. The IoTDB subscription client provides two
> types of consumers: pull consumer and push consumer.
> - Consumer Group is a collection of consumers. When different
> consumers in the same consumer group subscribe to the same topic,
> these consumers share the processing progress of data under this
> topic. Each data under this topic can only be processed by one
> consumer within the group, ensuring that 

[Proposal] Support different charsets in IoTDB

[ITAMI SHO]

Hi all,
I’m Itami Sho, a contributor to Apache IoTDB. Currently, IoTDB supports only 
the UTF-8 charset. To enhance the system's flexibility and usability across 
different environments, I have implemented a feature that handles character set 
conversion when the user system's charset does not match IoTDB's default UTF-8.

The main functionality of this feature involves converting input character sets 
to UTF-8 and output character sets to the user-specified charset.

This update will primarily focus on modifying the JDBC interfaces. During the 
initialization of a connection, users can now specify the charset parameter in 
the URL, as demonstrated below:

try (Connection connection = 
DriverManager.getConnection("jdbc:iotdb://127.0.0.1:6667?charset=GB18030", 
"root", "root"); Statement statement = connection.createStatement()) { // Your 
code here }

In future work, I plan to extend support for this feature to other interfaces 
such as Session, REST API, MQTT, and more.

You can check my PR to get more code details: 
https://github.com/apache/iotdb/pull/12618

I look forward to your feedback and suggestions to further improve this 
feature. Your input will be invaluable in ensuring that this enhancement meets 
the needs of the community : )

Best regards,

Itami Sho

[Proposal] Query mode in extraction of Streaming Engine

[??]

Hello everyone,
I'm Caideyipi, and is developing the Streaming Engine module. Here I wish to 
share a new feature I introduced to the "Extraction" phase of Streaming Engine.
Formerly, the Streaming Engine is long restricted to unbounded stream. For 
instance, if users want to extract the historical data only, they can only 
judge the end themselves by querying the receiver side, and drop the pipe 
manually. The inconvenience is apparent in not only the normal scenarios, but 
the data subscription and the tsFile export service, which may need a terminal 
event to halt the entire procedure.
Hence, a bound is introduced in the extractor phase. Now users can use 
"source.mode" in "with source" clause ("source" can be omitted) to decide 
whether to send the terminal event and drop the pipe automatically when the 
transmission is completed. When "source.mode" == "query" (default is 
"subscribe"), when the historical data is all processed/transmitted, a terminal 
event is sent to the subsequent steps. When all the pipe tasks has completed, 
the ConfigNode will collect the information in pipe heartbeats, and drop the 
pipe if all the pipeTasks have completed.
Fervently hope you like the feature, and glad to see any ideas or suggestions.
Thanks for your reading and support.

Sincerely,

Caideyipi

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Xiangdong Huang]

+1 binding
---
Xiangdong Huang


ITAMI SHO  于2024年5月29日周三 16:23写道：
>
> +1
>
> It will increase the stability of our builds!
>
> Itami Sho
>
> > 2024年5月28日 22:40，Christofer Dutz  写道：
> >
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the 
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible 
> > again), I also updated the Thrift version and staged a release of Thrift 
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so 
> > it should be quick to review.
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be 
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then finish 
> > changing the main repo to use these artifacts.
> >
> >
> > Chris
>

Re: [Proposal] Load Tsfile Functionality Optimisation

[Xiangpeng Hu]

Hello Xiangzhi,


This is a interesting and helpful new feature. Thank you for your dedication 
and hard work in enhancing Apache IoTDB!


Best regards,
|
Xiangpeng Hu
|
 Replied Message 
| From | ycy wi |
| Date | 5/30/2024 11:47 |
| To |  |
| Subject | Re: [Proposal] Load Tsfile Functionality Optimisation |
Hi Xiangzhi,

Thank you for your work! Load function is something I frequently use when 
testing, and your optimization should greatly enhance my user experience!

By the way, I would like to know what the general causes of failure might be, 
and whether there is an option to retry or handle the failed TsFile?

Best
CaiyinYang



2024年5月30日 11:11，孟祥志  写道：

Hello everyone,

I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
with you a feature that I have been working on for the past few months.

load tsfile has always been an important feature for IoTDB, but due to the 
original design, he has some flaws in it, when load a folder fails to import 
some tsfile, it may cause the whole task fails, which is obviously a not very 
user-friendly design, so I decided to modify his.

I'll modify the original load tsfile script to traverse the entire folder 
first, find all the files, and then use the LOAD function to import individual 
files; and I've added a few interesting changes along the way, modifying the 
successful operation parameter to make it easier for people to deal with 
successful files, and added a parameter for failed files to make it easier for 
people to find those failed files, and finally added a new number of threads to 
make the LOAD function work more quickly.


I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,

Xiangzhi Meng

Reference:
https://github.com/apache/iotdb/pull/12541

Re: [Proposal] Load Tsfile Functionality Optimisation

[ycy wi]

Hi Xiangzhi,

Thank you for your work! Load function is something I frequently use when 
testing, and your optimization should greatly enhance my user experience!

By the way, I would like to know what the general causes of failure might be, 
and whether there is an option to retry or handle the failed TsFile?

Best
CaiyinYang



> 2024年5月30日 11:11，孟祥志  写道：
> 
> Hello everyone,
> 
> I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
> with you a feature that I have been working on for the past few months.
> 
> load tsfile has always been an important feature for IoTDB, but due to the 
> original design, he has some flaws in it, when load a folder fails to import 
> some tsfile, it may cause the whole task fails, which is obviously a not very 
> user-friendly design, so I decided to modify his.
> 
> I'll modify the original load tsfile script to traverse the entire folder 
> first, find all the files, and then use the LOAD function to import 
> individual files; and I've added a few interesting changes along the way, 
> modifying the successful operation parameter to make it easier for people to 
> deal with successful files, and added a parameter for failed files to make it 
> easier for people to find those failed files, and finally added a new number 
> of threads to make the LOAD function work more quickly.
> 
> 
> I hope you are interested in this feature and would like to participate in 
> the development and testing. You can also leave your comments and suggestions 
> in this thread. Appreciate any suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> Xiangzhi Meng
> 
> Reference:
> https://github.com/apache/iotdb/pull/12541

[Proposal] Load Tsfile Functionality Optimisation

[孟祥志]

Hello everyone,

I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
with you a feature that I have been working on for the past few months.

load tsfile has always been an important feature for IoTDB, but due to the 
original design, he has some flaws in it, when load a folder fails to import 
some tsfile, it may cause the whole task fails, which is obviously a not very 
user-friendly design, so I decided to modify his.

I'll modify the original load tsfile script to traverse the entire folder 
first, find all the files, and then use the LOAD function to import individual 
files; and I've added a few interesting changes along the way, modifying the 
successful operation parameter to make it easier for people to deal with 
successful files, and added a parameter for failed files to make it easier for 
people to find those failed files, and finally added a new number of threads to 
make the LOAD function work more quickly.


I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,

Xiangzhi Meng

Reference:
https://github.com/apache/iotdb/pull/12541

[Proposal] New Columns in "Show Pipes" on IoTDB

[??]

Hello everyone,
I'm Caideyipi, a contributor to IoTDB, currently working on Streaming Engine. 
Here I wish to share a new feature I introduced to IoTDB "Show Pipes" SQL.
Previously, the "Show Pipes" SQL contains 7 columns, like 
"ID|CreationTime|State|PipeSource|PipeProcessor|PipeSink|ExceptionMessage". By 
that, we can easily see the pipes' definitions, statuses, and exceptions if 
exist. However, the pipe's current progress is left unclear, and users must 
query the receiver or browse the dashboard to acquire the progress. In some 
typical scenarios, not only the dashboard is absent due to resource limitation, 
but the receiver is inaccessible for secrecy concerns. In that circumstance, 
the progress may not be easily known by users.
Hence, two new columns, "RemainingEventCount" and "EstimatedRemainingSeconds", 
are introduced in show pipes result. They are also introduced to the dashboard. 
Detailed illustrations are provided below.
- "RemainingEventCount" is that how many events are left 
unprocessed/untransferred in the pipe framework, consisting of:
   1. The tsfile, insertion, raw tablet, heartbeat and schema events.
   2. Events in the config region and all the schema regions and data regions.
- "EstimatedRemainingSeconds" is the estimated elapsed seconds when the 
Streaming Engine has processed/transferred all the existing data. It's 
calculated by the "RemainingEventCount" of each region, and uses each region's 
average event commit rate in one minute in calculation. The commit rate will 
apply the "single exponential smoothing" method when updated to get more 
precise result, and the largest displayed time is 1 year.
To avoid affecting the cluster's performance, the two values are updated in the 
pipe's heartbeat and may have some delay. As an improvement, in a cluster 
having only one dataNode, if the pipe comprising only data region and schema 
region, it's calculation is done at the dataNode and is refreshed on query.
I heartily hope that you are interested in this feature. Any participation in 
testing, and any comments or suggestions in this thread are welcome.

Thanks for your reading and support.

Sincerely,

Caideyipi

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[ITAMI SHO]

+1

It will increase the stability of our builds!

Itami Sho

> 2024年5月28日 22:40，Christofer Dutz  写道：
> 
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[ycy wi]

+1 
It will significantly improved our build stability and cross-platform support.

> 2024年5月28日 22:40，Christofer Dutz  写道：
> 
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris

Re: Could anyone please approve this PR?

[Jialin Qiao]

This is merged :)


Jialin Qiao

Christofer Dutz  于2024年5月28日周二 22:23写道：
>
> Hi all,
>
> in the iotdb-bin-resources I have also updated the Cmake version and thrift 
> version in order to be able to deploy new versions of thrift artifacts.
>
> https://github.com/apache/iotdb-bin-resources/pull/61
>
> Would be cool if I could get some approval.
>
> Chris

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Jialin Qiao]

+1 (binding)


Jialin Qiao

Yuan Tian  于2024年5月29日周三 09:07写道：
>
> +1 (binding)
>
> On Tue, May 28, 2024 at 10:40 PM Christofer Dutz 
> wrote:
>
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible
> > again), I also updated the Thrift version and staged a release of Thrift
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> >
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so
> > it should be quick to review.
> >
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then
> > finish changing the main repo to use these artifacts.
> >
> >
> > Chris
> >

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Yuan Tian]

+1 (binding)

On Tue, May 28, 2024 at 10:40 PM Christofer Dutz 
wrote:

> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible
> again), I also updated the Thrift version and staged a release of Thrift
> binaries for all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
>
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so
> it should be quick to review.
>
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then
> finish changing the main repo to use these artifacts.
>
>
> Chris
>

AW: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

+1 (binding)

Chris

[OK] Checked the signatures
[OK] Checked the hashes
[OK] Checked the content of the source bundle
[OK] Checked that the staged binaries work on their target platforms



Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 16:40
An: dev@iotdb.apache.org 
Betreff: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)
Hi all,

I after fixing some issues in the cmake-maven-plugin, have I updated the 
iotdb-bin-resources module to use these.
Now we can stage Apache Thrift binaries for all platforms.

In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), I 
also updated the Thrift version and staged a release of Thrift binaries for all 
of our supported platforms in Nexus.

I’ve staged the artifacts at:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Tag Name: iotdb-tools-thrift-v0.20.0.0
https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0

The source bundle generally consists of a pom.xml and an assembly xml … so it 
should be quick to review.
https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip

I’m intentionally not staging it on the release-svn, as nobody should be 
required to consume this module but us ourselves.

The vote stays open for 72hours … assuming enough support, I’ll then finish 
changing the main repo to use these artifacts.


Chris

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Haonan Hou]

+1

Haonan Hou

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris
> 

[DISCUSS] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Hi all,

this vote is not on switching to Thrift 0.20.0. In order to try out Thrift we 
however need to have the Thrift binaries available.
That’s why I’ve started a VOTE on publishing these.

The alternative would be to work with intermediate maven repos, which just 
makes things a lot more difficult.

Chris

[VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Hi all,

I after fixing some issues in the cmake-maven-plugin, have I updated the 
iotdb-bin-resources module to use these.
Now we can stage Apache Thrift binaries for all platforms.

In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), I 
also updated the Thrift version and staged a release of Thrift binaries for all 
of our supported platforms in Nexus.

I’ve staged the artifacts at:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Tag Name: iotdb-tools-thrift-v0.20.0.0
https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0

The source bundle generally consists of a pom.xml and an assembly xml … so it 
should be quick to review.
https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip

I’m intentionally not staging it on the release-svn, as nobody should be 
required to consume this module but us ourselves.

The vote stays open for 72hours … assuming enough support, I’ll then finish 
changing the main repo to use these artifacts.


Chris

Could anyone please approve this PR?

[Christofer Dutz]

Hi all,

in the iotdb-bin-resources I have also updated the Cmake version and thrift 
version in order to be able to deploy new versions of thrift artifacts.

https://github.com/apache/iotdb-bin-resources/pull/61

Would be cool if I could get some approval.

Chris

AW: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Ok … and done:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Here are all 6 types of binaries.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 11:01
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] Investigate updating to Thrift 0.20.0?
Ok …

So, I’m currently working on staging the last of the binaries (thrift for 
windows x86_64).
However, in order to make this work I think we would also need to update the 
CMake version.
I spent a lot of time over the last weekends figuring out how to support Cmake 
on all of these platforms with the maintainer of the cmake-maven-plugin.
Now we have a version that works on all systems I have.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

AW: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Ok …

So, I’m currently working on staging the last of the binaries (thrift for 
windows x86_64).
However, in order to make this work I think we would also need to update the 
CMake version.
I spent a lot of time over the last weekends figuring out how to support Cmake 
on all of these platforms with the maintainer of the cmake-maven-plugin.
Now we have a version that works on all systems I have.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

Re: Missing binaries for Windows on Aarch64 systems

[Christofer Dutz]

Yeah.

But I could also produce a new set of binaries for 0.14.1...we'd have to 
re-release them too.

Chris

Gesendet von Outlook für Android

From: Jialin Qiao 
Sent: Tuesday, May 28, 2024 3:39:59 AM
To: dev@iotdb.apache.org 
Subject: Re: Missing binaries for Windows on Aarch64 systems

Hi,

Is this will also be fixed by upgrading thrift to 0.20?

Jialin Qiao

Christofer Dutz  于2024年5月24日周五 22:58写道：
>
> Hi all,
>
> I was just verifying the build using the most recent cmake-maven-plugin works 
> and stumbled over something.
>
> Currently we don’t have thrift binaries for combinations of systems. It seems 
> we’re missing windows-aarch64.
>
> Should we plan fixing this and doing a release 0.14.1.1?
>
> Chris

Re: Upgrading 0.12 -> 1.3

[Trevor Hart]

Thank you Jialin. I didnt realise the data was not migrated as part of the 
upgrade.



That is good to see there is a resource to migrate. I will use that approach.



Thanks 

Trevor Hart




 On Tue, 28 May 2024 13:35:41 +1200 Jialin Qiao  
wrote ---



Hi, 
 
The architecture of 1.x is different from 0.13. The upgrading can not 
be in situ. You need to deploy an 1.3 instance, then transfer the 
data. 
There are two ways to transfer data: 
1. Using Session to query from the 0.13, then write into 1.3. 
2. Using TsFile API to rewrite the TsFile, then load TsFiles to 1.3. 
 
1. is easier. You could refer to this: 
https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/DataMigrationExample.java
 
 
Jialin Qiao

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Haonan Hou]

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
> 
> Hi all,
> 
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
> 
> https://github.com/apache/thrift/blob/master/CHANGES.md
> 
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
> 
> 
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
> 
> Chris

Re: Missing binaries for Windows on Aarch64 systems

[Jialin Qiao]

Hi,

Is this will also be fixed by upgrading thrift to 0.20?

Jialin Qiao

Christofer Dutz  于2024年5月24日周五 22:58写道：
>
> Hi all,
>
> I was just verifying the build using the most recent cmake-maven-plugin works 
> and stumbled over something.
>
> Currently we don’t have thrift binaries for combinations of systems. It seems 
> we’re missing windows-aarch64.
>
> Should we plan fixing this and doing a release 0.14.1.1?
>
> Chris

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Jialin Qiao]

Hi,

Looks good :D
If the upgrading does not bring any change to our users, +1 for upgrading.
You could prepare a PR, then we perform some validations.

Jialin Qiao

Christofer Dutz  于2024年5月28日周二 03:41写道：
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>   *   Mac – aarch64
>   *   Mac – x86_64
>   *   Linux  – aarch64
>   *   Linux  – x86_64
>   *   Windows  – aarch64
>   *   Windows  – x86_64
>
> Chris

Re: Upgrading 0.12 -> 1.3

[Jialin Qiao]

Hi,

The architecture of 1.x is different from 0.13. The upgrading can not
be in situ. You need to deploy an 1.3 instance, then transfer the
data.
There are two ways to transfer data:
1. Using Session to query from the 0.13, then write into 1.3.
2. Using TsFile API to rewrite the TsFile, then load TsFiles to 1.3.

1. is easier. You could refer to this:
https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/DataMigrationExample.java

Jialin Qiao

Trevor Hart  于2024年5月28日周二 06:27写道：
>
> Hello
>
>
>
> I have an application that is running against IoTDB version 0.12 which I 
> would like to upgrade to 1.3
>
>
>
> However I have some timeseries paths that contain numbers (eg below)
>
>
> root.ABC.health.url.17.code
>
>
>
> From the documentation it seems this is not supported beyond 0.13
>
>
>
> To migrate this data I am proposing this process;
>
>
>
> 1. Use "select into" to migrate to a new path ie root.ABC.health.url.17.code 
> -> root.ABC.health.url.U17.code
>
> 2. Delete the old timeseries ie root.ABC.health.url.17.code
>
> 3. Upgrade IoTDB
>
>
> Is this the correct/best approach?
>
>
> Thanks
>
> Trevor Hart