Re: [Proposal] Load Tsfile Functionality Optimisation
Hello Xiangzhi, This is a interesting and helpful new feature. Thank you for your dedication and hard work in enhancing Apache IoTDB! Best regards, | Xiangpeng Hu | Replied Message | From | ycy wi | | Date | 5/30/2024 11:47 | | To | | | Subject | Re: [Proposal] Load Tsfile Functionality Optimisation | Hi Xiangzhi, Thank you for your work! Load function is something I frequently use when testing, and your optimization should greatly enhance my user experience! By the way, I would like to know what the general causes of failure might be, and whether there is an option to retry or handle the failed TsFile? Best CaiyinYang 2024年5月30日 11:11，孟祥志 写道： Hello everyone, I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share with you a feature that I have been working on for the past few months. load tsfile has always been an important feature for IoTDB, but due to the original design, he has some flaws in it, when load a folder fails to import some tsfile, it may cause the whole task fails, which is obviously a not very user-friendly design, so I decided to modify his. I'll modify the original load tsfile script to traverse the entire folder first, find all the files, and then use the LOAD function to import individual files; and I've added a few interesting changes along the way, modifying the successful operation parameter to make it easier for people to deal with successful files, and added a parameter for failed files to make it easier for people to find those failed files, and finally added a new number of threads to make the LOAD function work more quickly. I hope you are interested in this feature and would like to participate in the development and testing. You can also leave your comments and suggestions in this thread. Appreciate any suggestion/feedback & contribution. Thank you for your attention and support. Best regards, Xiangzhi Meng Reference: https://github.com/apache/iotdb/pull/12541
Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)
+1 binding --- Xiangdong Huang ITAMI SHO 于2024年5月29日周三 16:23写道： > > +1 > > It will increase the stability of our builds! > > Itami Sho > > > 2024年5月28日 22:40，Christofer Dutz 写道： > > > > Hi all, > > > > I after fixing some issues in the cmake-maven-plugin, have I updated the > > iotdb-bin-resources module to use these. > > Now we can stage Apache Thrift binaries for all platforms. > > > > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible > > again), I also updated the Thrift version and staged a release of Thrift > > binaries for all of our supported platforms in Nexus. > > > > I’ve staged the artifacts at: > > https://repository.apache.org/content/repositories/orgapacheiotdb-1158 > > > > Tag Name: iotdb-tools-thrift-v0.20.0.0 > > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0 > > > > The source bundle generally consists of a pom.xml and an assembly xml … so > > it should be quick to review. > > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip > > > > I’m intentionally not staging it on the release-svn, as nobody should be > > required to consume this module but us ourselves. > > > > The vote stays open for 72hours … assuming enough support, I’ll then finish > > changing the main repo to use these artifacts. > > > > > > Chris >
[Proposal] Query mode in extraction of Streaming Engine
Hello everyone,
I'm Caideyipi, and is developing the Streaming Engine module. Here I wish to
share a new feature I introduced to the "Extraction" phase of Streaming Engine.
Formerly, the Streaming Engine is long restricted to unbounded stream. For
instance, if users want to extract the historical data only, they can only
judge the end themselves by querying the receiver side, and drop the pipe
manually. The inconvenience is apparent in not only the normal scenarios, but
the data subscription and the tsFile export service, which may need a terminal
event to halt the entire procedure.
Hence, a bound is introduced in the extractor phase. Now users can use
"source.mode" in "with source" clause ("source" can be omitted) to decide
whether to send the terminal event and drop the pipe automatically when the
transmission is completed. When "source.mode" == "query" (default is
"subscribe"), when the historical data is all processed/transmitted, a terminal
event is sent to the subsequent steps. When all the pipe tasks has completed,
the ConfigNode will collect the information in pipe heartbeats, and drop the
pipe if all the pipeTasks have completed.
Fervently hope you like the feature, and glad to see any ideas or suggestions.
Thanks for your reading and support.
Sincerely,
Caideyipi
[Proposal] Support different charsets in IoTDB
Hi all,
I’m Itami Sho, a contributor to Apache IoTDB. Currently, IoTDB supports only
the UTF-8 charset. To enhance the system's flexibility and usability across
different environments, I have implemented a feature that handles character set
conversion when the user system's charset does not match IoTDB's default UTF-8.
The main functionality of this feature involves converting input character sets
to UTF-8 and output character sets to the user-specified charset.
This update will primarily focus on modifying the JDBC interfaces. During the
initialization of a connection, users can now specify the charset parameter in
the URL, as demonstrated below:
try (Connection connection =
DriverManager.getConnection("jdbc:iotdb://127.0.0.1:6667?charset=GB18030",
"root", "root"); Statement statement = connection.createStatement()) { // Your
code here }
In future work, I plan to extend support for this feature to other interfaces
such as Session, REST API, MQTT, and more.
You can check my PR to get more code details:
https://github.com/apache/iotdb/pull/12618
I look forward to your feedback and suggestions to further improve this
feature. Your input will be invaluable in ensuring that this enhancement meets
the needs of the community : )
Best regards,
Itami Sho
