> 1. Checking before insert if the timestamp already exists and remedy on the > client before resend > 2. Moving to Nanosecond and introducing some insignificant time value to keep > timestamp values unique. Yes these maybe the best solutions for a specific application.
Analysis for IoTDB: - Rejecting the write when receiving an existing timestamp in IoTDB is time-costly (IoTDB needs to check historical data). I think we will do not check it until we find a low-latency method. - Allowing multiple value versions for a timestamp may introduce a chain reaction and there may be a lot of codes that should be modified, which is a huge work. There is a new idea (but I have no time to implement it...) - Add a parameter in IoTDB: replace_strategy: first, last, avg etc... - when an existing timestamp arrives, IoTDB accepts it - when IoTDB runs LSM to merge data and meets multiple values for a timestamp, then handles it according to the replace_startegy. The solution may also introduce some work to do... and we need to think carefully the impact to the query process. Need to survey whether this is a common requirement. Best, ----------------------------------- Xiangdong Huang Trevor Hart <tre...@ope.nz> 于2024年5月14日周二 09:55写道： > > Hello Yuan > > > > Correct, the first timestamp and values should be retained. > > > > I realise this is does not align with the current design. I was just asking > whether there was an existing option to operate to block duplicates. > > > > In a normal RDBMS if you try to insert with a duplicate the insert will fail > with a PK violation. It would be great in some circumstances if IotDB at > least had the option to fail this way. > > > > I am considering some options such as; > > > > 1. Checking before insert if the timestamp already exists and remedy on the > client before resend > > 2. Moving to Nanosecond and introducing some insignificant time value to keep > timestamp values unique. > > > > I have already done something similar to #2 with storing IIS web log files as > they are recorded in seconds and not milliseconds. > > > > Thanks > > Trevor Hart > > > > > ---- On Tue, 14 May 2024 13:29:02 +1200 Yuan Tian <jackietie...@gmail.com> > wrote --- > > > > Hi Trevor, > > By "rejects duplicates", you mean you want to keep the first duplicate > timestamp and its corresponding values?(because the following duplicated > ones will be rejected) > > Best regards, > -------------------- > Yuan Tian > > On Mon, May 13, 2024 at 6:24 PM Trevor Hart <mailto:tre...@ope.nz> wrote: > > > > > > > > > > > Correct. I’m not disputing that. What I’m asking is that it > > would be good to have a configuration that either allows overwrites or > > rejects duplicates.My scenario is request log data from a server (the > > device). As it may be processing multiple requests at once there is a > > chance that there could be colliding time stamps.As it stands now I would > > need to check if the timestamp exists before inserting the data. Which > > obviously affects throughput. Thanks Trevor Hart ---- On Fri, 10 May > > 2024 00:33:40 +1200 Jialin Qiao<mailto:qiaojia...@apache.org> wrote ---- > > Hi, > > In IoT or IIoT scenarios, we thought each data point represent a metric of > > a timestamp.In which case you need to store duplicated values? Take this > > for an example: Time, root.sg1.car1.speed 1, 1 1, 2 Could a car has > > different speed at time 1? Jialin Qiao Yuan Tian < > > mailto:jackietie...@gmail.com> 于2024年5月9日周四 18:51写道： > > Hi Trevor, > > Now > > we > > will override the duplicate timestamp with a newer one. There is > nothing > > we can do about it now. > > Best regards, > ------------------- > Yuan Tian > > > > On Wed, May 8, 2024 at 5:31 PM Trevor Hart <mailto:tre...@ope.nz> > > > > wrote: > > > > > Hello > > > > > > > > I’m aware that when inserting a duplicate timestamp > > the values will be > > overwritten. This will obviously result in data > > loss. > > > > > > > > Is there a config/setting to reject or throw an error > > on duplicate > > inserts? Although highly unlikely I would prefer to be > > alerted to the > > situation rather than lose data. > > > > > > > > I read > > through the documentation but couldn’t find anything. > > > > > > > > > > Thanks > > > > Trevor Hart > > > > > > > > > > > > > >
