[Proposal] Load Tsfile Functionality Optimisation
Hello everyone, I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share with you a feature that I have been working on for the past few months. load tsfile has always been an important feature for IoTDB, but due to the original design, he has some flaws in it, when load a folder fails to import some tsfile, it may cause the whole task fails, which is obviously a not very user-friendly design, so I decided to modify his. I'll modify the original load tsfile script to traverse the entire folder first, find all the files, and then use the LOAD function to import individual files; and I've added a few interesting changes along the way, modifying the successful operation parameter to make it easier for people to deal with successful files, and added a parameter for failed files to make it easier for people to find those failed files, and finally added a new number of threads to make the LOAD function work more quickly. I hope you are interested in this feature and would like to participate in the development and testing. You can also leave your comments and suggestions in this thread. Appreciate any suggestion/feedback & contribution. Thank you for your attention and support. Best regards, Xiangzhi Meng Reference: https://github.com/apache/iotdb/pull/12541
Re: [Proposal] Load Tsfile Functionality Optimisation
Hi Xiangzhi, Thank you for your work! Load function is something I frequently use when testing, and your optimization should greatly enhance my user experience! By the way, I would like to know what the general causes of failure might be, and whether there is an option to retry or handle the failed TsFile? Best CaiyinYang > 2024年5月30日 11:11，孟祥志 写道： > > Hello everyone, > > I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share > with you a feature that I have been working on for the past few months. > > load tsfile has always been an important feature for IoTDB, but due to the > original design, he has some flaws in it, when load a folder fails to import > some tsfile, it may cause the whole task fails, which is obviously a not very > user-friendly design, so I decided to modify his. > > I'll modify the original load tsfile script to traverse the entire folder > first, find all the files, and then use the LOAD function to import > individual files; and I've added a few interesting changes along the way, > modifying the successful operation parameter to make it easier for people to > deal with successful files, and added a parameter for failed files to make it > easier for people to find those failed files, and finally added a new number > of threads to make the LOAD function work more quickly. > > > I hope you are interested in this feature and would like to participate in > the development and testing. You can also leave your comments and suggestions > in this thread. Appreciate any suggestion/feedback & contribution. > > Thank you for your attention and support. > > Best regards, > > Xiangzhi Meng > > Reference: > https://github.com/apache/iotdb/pull/12541
Re: [Proposal] Load Tsfile Functionality Optimisation
Hello Xiangzhi, This is a interesting and helpful new feature. Thank you for your dedication and hard work in enhancing Apache IoTDB! Best regards, | Xiangpeng Hu | Replied Message | From | ycy wi | | Date | 5/30/2024 11:47 | | To | | | Subject | Re: [Proposal] Load Tsfile Functionality Optimisation | Hi Xiangzhi, Thank you for your work! Load function is something I frequently use when testing, and your optimization should greatly enhance my user experience! By the way, I would like to know what the general causes of failure might be, and whether there is an option to retry or handle the failed TsFile? Best CaiyinYang 2024年5月30日 11:11，孟祥志 写道： Hello everyone, I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share with you a feature that I have been working on for the past few months. load tsfile has always been an important feature for IoTDB, but due to the original design, he has some flaws in it, when load a folder fails to import some tsfile, it may cause the whole task fails, which is obviously a not very user-friendly design, so I decided to modify his. I'll modify the original load tsfile script to traverse the entire folder first, find all the files, and then use the LOAD function to import individual files; and I've added a few interesting changes along the way, modifying the successful operation parameter to make it easier for people to deal with successful files, and added a parameter for failed files to make it easier for people to find those failed files, and finally added a new number of threads to make the LOAD function work more quickly. I hope you are interested in this feature and would like to participate in the development and testing. You can also leave your comments and suggestions in this thread. Appreciate any suggestion/feedback & contribution. Thank you for your attention and support. Best regards, Xiangzhi Meng Reference: https://github.com/apache/iotdb/pull/12541
Re: [Proposal] Load Tsfile Functionality Optimisation
Hi Xiangzhi, Thank you for sharing your work on the load tsfile feature. I am currently working on the Load function as well, and I have a question for you. Have you considered enhancing the Load function directly within the core code, rather than relying on the script to load tsfiles one by one? Additionally , do you think there would be a significant performance difference between the original batch load method and your script of loading files individually? Your insights on this matter would be very valuable, and I believe it could lead to further optimizations! Best regards, Itami Sho > 2024年5月30日 11:11，孟祥志 写道： > > Hello everyone, > > I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share > with you a feature that I have been working on for the past few months. > > load tsfile has always been an important feature for IoTDB, but due to the > original design, he has some flaws in it, when load a folder fails to import > some tsfile, it may cause the whole task fails, which is obviously a not very > user-friendly design, so I decided to modify his. > > I'll modify the original load tsfile script to traverse the entire folder > first, find all the files, and then use the LOAD function to import > individual files; and I've added a few interesting changes along the way, > modifying the successful operation parameter to make it easier for people to > deal with successful files, and added a parameter for failed files to make it > easier for people to find those failed files, and finally added a new number > of threads to make the LOAD function work more quickly. > > > I hope you are interested in this feature and would like to participate in > the development and testing. You can also leave your comments and suggestions > in this thread. Appreciate any suggestion/feedback & contribution. > > Thank you for your attention and support. > > Best regards, > > Xiangzhi Meng > > Reference: > https://github.com/apache/iotdb/pull/12541
