Hi,
> I wonder whether we could index the file by its name. (naming the tsfile
by date)
I think it is a good idea, but maybe not very easy to implement. If we can
organize the data like this, then it is very very regular and very easy to
access or delete expired data...
> we would need is a tree strucutre where each node has start time / end
time for "everything" in the file.
This is also a good idea.
When we are discussing the granularity of "device", we are worrying about
the size of the index, actually.
So, we do not care whether there is a so called "sub device", we just care
how many entities will be indexed.
Suppose an IoTDB instance can bear 1 million index entries <some_id ->
(start time, end time)>, and given a tree schema, if there are about 1
million nodes from level 0 to level 3, then we can index the nodes on
level3 (so level 3 is so-called "device" in current version).
Meantime, index the nodes from level0 to level2, as Julian proposed, is
also beneficial.
The nature of the above idea is letting IoTDB decides which are "devices"
automatically.
At the beginning of this discussion, I just want to let user claim which
are "devices" (or, which prefixes of Paths have time indexes.. but this
kind of description may be not user friendly..). As it is more easy.... but
may carry risk if the user set too many devices.
Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
runhus...@foxmail.com <runhus...@foxmail.com> 于2020年7月20日周一 下午7:47写道：
> Hi，
>
> > I wonder whether we could index the file by its name. (naming the tsfile
> by date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
>
> So, how to deal with the out of order data? Could you give more details.
>
>
>
> Thanks!
>
> runhus...@foxmail.com
>
>
> From: Jialin Qiao
> Date: 2020-07-20 18:21
> To: dev
> Subject: Re: [Discuss] How to delivery the device concept to users
> Hi,
>
> > The question I would ask is why "devices" hurt us.
>
> I'd like to introduce this a bit. For each storage group, we flush the
> memtable into TsFiles one by one. For each TsFile, we maintain a temporal
> index on device level in memory. Suppose there are 3 devices in one TsFile,
> the index is like this:
>
> start time array: long[3] = {1, 1, 2}
> end time array: long[3] = {5, 6, 10}
> devicesToIndexInArray: Map<String, Integer> = {"root.sg.d1" -> 0,
> "root.sg.d2" -> 1, "root.sg.d3" -> 2}
>
> If we have millions of devices, for each TsFile, this index will reach
> dozens of MB in memory. Although we could introduce the persistence of the
> index. It is still recommended to decrease the number of devices.
>
> I wonder whether we could index the file by its name. (naming the tsfile
> by date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -----原始邮件-----
> > 发件人: "Julian Feinauer" <j.feina...@pragmaticminds.de>
> > 发送时间: 2020-07-20 17:34:40 (星期一)
> > 收件人: "dev@iotdb.apache.org" <dev@iotdb.apache.org>
> > 抄送:
> > 主题: Re: [Discuss] How to delivery the device concept to users
> >
> > Hey Jialin, xinagdong,
> >
> > very good question!
> >
> > And I tend to agree with Xiangdong.
> > If the users do it that way it probably makes most sense for them.
> > The question I would ask is why "devices" hurt us (I know a bit about
> the implementation of course but probably we have to adopt our datamodel
> also a bit in the future).
> >
> > Generally speaking, form e it also makes sense tob e allowed to have
> "subcategories" below my devices as my devices usually are "big".
> > And technically speaking in the current version this is totally possible
> to have nested structures below devices or measurements (but these will
> then again be devices).
> >
> > So my question is:
> > - Do we really need the static construct of a "device" or can we
> probably use a different datastructure where I "select" my device only at
> query time and we just select everything under that tree as ist
> measurements or "sub-measurements" in cases of nesting.
> >
> > WDYT?
> >
> > Julian
> >
> > Am 20.07.20, 09:34 schrieb "Xiangdong Huang" <saint...@gmail.com>:
> >
> > Hi,
> >
> > This is a quite good topic!
> >
> > 1. maybe we should hear more users opinions.
> >
> > For me, I think emphasize the concept of "device" is good. We can
> even
> > expose the concept in our APIs.
> >
> > 2.
> >
> > > A more efficient way is
> > > root.sg.device1.measurement1_int0
> > > root.sg.device1.measurement1_int1
> > > root.sg.device1.measurement1_int2
> > > root.sg.device1.measurement2_long
> >
> > I think the more efficient way is:
> >
> > root.sg.device1.measurement1.0
> > root.sg.device1.measurement1.1
> > root.sg.device1.measurement1.2
> > root.sg.device1.measurement2
> >
> > And, as you said "a device has a sensor that collects some data in
> array
> > format (int[3]) and some in long type",
> > will the user query just one element from the int[3]? If not, a
> better
> > schema is:
> >
> > root.sg.device1.measurement1 (the dataType is int[])
> > root.sg.device1.measurement2 (the dataType is long)
> >
> > Best,
> > -----------------------------------
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > Jialin Qiao <qj...@mails.tsinghua.edu.cn> 于2020年7月20日周一 下午3:28写道：
> >
> > > Hi
> > >
> > > Recently, I find that some users create timeseries do not
> following the
> > > real world semantic of device
> > >
> > >
> > > E.g., a device has a sensor that collects some data in array format
> > > (int[3]) and some in long type.
> > >
> > >
> > > Many users will create timeseries like this:
> > >
> > >
> > > root.sg.device1.measurement1.int0
> > > root.sg.device1.measurement1.int1
> > > root.sg.device1.measurement1.int2
> > > root.sg.device1.measurement2.long
> > >
> > >
> > > As a consequence, there will be two devices instead of one device.
> This
> > > will cause the real number of devices is much bigger than the real
> devices
> > > they thought. The drawback is: more devices leads to more memory
> > > consumption.
> > >
> > >
> > > A more efficient way is
> > >
> > >
> > > root.sg.device1.measurement1_int0
> > > root.sg.device1.measurement1_int1
> > > root.sg.device1.measurement1_int2
> > > root.sg.device1.measurement2_long
> > >
> > >
> > > In this schema, there will be only one device and 4 measurements.
> > >
> > >
> > > The problem is we extract the device id automatically. Users
> usually do
> > > not have a clear concept about "device". Should we emphasize the
> concept of
> > > device by letting users create device manually?
> > >
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > --
> > > Jialin Qiao
> > > School of Software, Tsinghua University
> > >
> > > 乔嘉林
> > > 清华大学 软件学院
> >
>
