Hi Cheongu, Q1: As you've mentioned, you can change the time precision from milliseconds to microseconds to support more data points in one second. In IoTDB, we don't have a way to automatically do that thing.
Q2: DeviceId + time is the primary key in IoTDB, so I wonder why you have so many duplicate timestamps in one car. Q3: The loading time is too long, I think you use the wrong insert method, "7.6M, 43.6M, and 168.4M" is not that large, using the right loading method and concurrently loading won't take that much time. BTW, I think you can firstly preprocess your data to make them not have duplicate timestamps in one car or you can just explain why your data have so many duplicate timestamps in one car. Best regards, ------------------------ Yuan Tian On Thu, Jan 18, 2024 at 10:12 AM 김천구 <[email protected]> wrote: > Dear Apache IoTDB Development Team > > Greetings, My name is Cheongu Kim and I have a question about IoTDB. > Previously I asked about duplicate tuples at the same time. > And I received a suggestion that using an offset is a solution to my > problem. > e.g., > origin data: <car_1, 1705396048000, 2.0>, <car_1, 1705396048000, 2.1> > to: <car_1, 1705396048000, 2.0>, <car_1, 1705396048001, 2.1> > > While it is possible to adjust the resolution down to milliseconds, > it is not feasible to modify it to seconds as it may yield different query > results > (Because the examination starts from the second level), > Which means only 1000 duplicates can be processed for the same timestamp. > (using offset 0.000s ~ 0.999s) > And my dataset contains a maximum of 2572 occurrences of duplication for > the same timestamp. > > Secondly, when using offsets, the addition of the duplicate-checking > process to the data loading phase significantly extends the loading time. > Considering that my dataset sizes are 17.6M, 43.6M, and 168.4M > respectively, it seems impractical to use the offset method, as quick data > loading is crucial for me to complete my tests. > (the estimated loading times are 23h, 58h, and 224h) > > So here is my question. > Q1) Is there a way to apply an offset to the timestamp in microseconds in > IoTDB? > If possible, please provide guidance. > > Q2) Is time the only unique key column, or could you provide information on > creating key columns other than time to distinguish duplicates at the > same time? > > Q3) If possible, could you recommend an efficient method for loading a > dataset that already contains measured results and has a significant amount > of duplicated timestamps, ensuring data integrity? > > Thank you for your time, and I look forward to a positive and helpful > response. > > Best regards, > ----------------------------------- > Cheongu Kim > [email protected] >
