Hi, This PR has opened for several days. I will merge this PR into the master.
By the way, I notice that there is no document about TsFile API. Actually we have previous versions about the API introduction and example on https://github.com/thulab/tsfile/wiki , but I am not sure whether they catch up with the latest version of API... I open a JIRA ticket, https://issues.apache.org/jira/browse/IOTDB-40 . Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Xiangdong Huang <[email protected]> 于2019年3月6日周三 上午12:53写道： > Hi, > > I have added a new TsFileIOWriter, which supports to recover data from a > broken TsFile (I mean, a TsFile that is not closed correctly). The PR is > https://github.com/apache/incubator-iotdb/pull/87. > > You can use the new feature by: > ``` > NativeRestorableIOWriter rWriter = new NativeRestorableIOWriter(file); > TsFileWriter writer = new TsFileWriter(rWriter); > ``` > 1. If the file is a complete TsFile (e.g., the FileMetadata has been > persistent into the file and the tail magic string is correct), then you > can not write data into this TsFile anymore (actually I can remove all > FileMetadata and then let you continue to write, but it needs more coding > work...), `TsFileWriter writer = new TsFileWriter(rWriter);` will throw a > IoException. You can check it before you create TsFileWriter by using > `rWriter.canWrite()`. > > 2. ChunkGroup is the basic unit that can be recover. That is, if a > ChunkGroup is complete, i.e., its chunkGroupFooter is complete, I will > remain it, otherwise I will truncate the chunk group. However, I can not > know whether a chunkGroupFooter is complete (if only one byte lose, the > deserialize method does not throw exception, because something like > `deserializeToInt(inputstream)` does not throw exception (it will use 3 > bytes to convert them to a integer). So, when I say "if a ChunkGroup is > complete", I mean that not only its chunkGroupFooter is serialized > successfully, but also there is at least one more byte is serialized into > the file followed the footer. > > 3. I have tested many cases, such only a magic head string is persisted, > only a ChunkHeader is persisted, the first ChunkHeader is not persisted > completely, only some Chunks are persisted, a ChunkGroupFooter is > persisted, Two ChunkGroupFooter are persisted, and a complete TsFile is > persisted, etc.. (`NativeRestorableIOWriterTest.java`, hope I have cover > all cases) > > Hope TsFile can be used in edge device management applications ASAP! > > Now please enjoy the new feature. > > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Julian Feinauer <[email protected]> 于2019年3月5日周二 下午8:10写道： > >> Hey, >> >> thank you fort he link... I did not know of this.. this is exactly what I >> was looking for! >> >> Julian >> >> PS.: Looking forward to your PR : ) >> >> Am 05.03.19, 12:26 schrieb "Xiangdong Huang" <[email protected]>: >> >> Hi, >> >> 1. We have a document to introduce that: >> https://cwiki.apache.org/confluence/display/IOTDB/TsFile+Format >> >> 2. The new API for recovering data is almost done. I am writing the >> UTs >> now. Maybe I can submit a PR tonight (if everything is fine...) >> >> Best, >> ----------------------------------- >> Xiangdong Huang >> School of Software, Tsinghua University >> >> 黄向东 >> 清华大学 软件学院 >> >> >> Julian Feinauer <[email protected]> 于2019年3月5日周二 下午6:00写道： >> >> > Hi Xiangdong, >> > >> > that sounds excellent. >> > Do you have a short overview of how the file format is designed on >> disk? >> > I know that its somewhat similar to parquet but I did not find more >> > details. >> > Basically what would suffice for us would be something like >> skipping an >> > invalid column group (or how you name it) and go on with the next, >> or so. >> > >> > Julian >> > >> > Am 04.03.19, 13:21 schrieb "Xiangdong Huang" <[email protected]>: >> > >> > Hi, >> > >> > If so, I think I need to add a new API to allow you continue to >> write >> > data >> > in an existing but not closed correctly TsFile. Then >> everything is >> > fine >> > for you :D >> > >> > Best, >> > ----------------------------------- >> > Xiangdong Huang >> > School of Software, Tsinghua University >> > >> > 黄向东 >> > 清华大学 软件学院 >> > >> > >> > Julian Feinauer <[email protected]> 于2019年3月4日周一 >> 下午8:08写道： >> > >> > > Hey Xiangdong, >> > > >> > > thanks for the great explanation. >> > > And in fact, I agree with you that it would be best if we >> start to >> > play >> > > around with it and reply all our findings or wishes back to >> this >> > list (in >> > > fact that proved to be beneficial in plc4x as well). >> > > >> > > You confirm my thoughts about the two "levels" of APIs (DB >> and file) >> > and >> > > the file api is exactly what we looked for for our use case. >> > > As we do not care much about data loss (when an edge device >> fails >> > its... >> > > gone). >> > > The crucial point for us is that no corrupt files can be >> generated. >> > > This means I'm fine when the last data submitted is lost but >> I'm not >> > fine >> > > if we can get to a situation where the last datafile is >> completely >> > lost >> > > (well, perhaps this could be acceptable). >> > > >> > > @tim: Perhaps its best when you give some more information to >> > Xiangdong >> > > about our idea, and we can also point to our current code in >> github >> > > >> > > Julian >> > > >> > > Am 04.03.19, 13:03 schrieb "Xiangdong Huang" < >> [email protected]>: >> > > >> > > Hi, >> > > >> > > TsFile API is not deprecated. In fact, it is designed for >> this >> > > scenario and >> > > MapReduce/Spark computing. >> > > >> > > If you just use Reader and Writer API, there is something >> you >> > need to >> > > know: >> > > >> > > Let's suppose your block size is x Bytes, >> > (tsfile-format.properties: >> > > group_size_in_byte). >> > > >> > > 1. If you write data and a shutdown occurs, then all data >> that is >> > > flushed >> > > on disk is ok, and you can read the data ( class >> > > org.apache.iotdb.tsfile.TsFileSequenceRead is an example, >> but >> > you need >> > > to >> > > change it a little. I think I can write an example.) >> > > >> > > 2. Actually, TsFile has the ability to allow you continue >> to >> > write >> > > data at >> > > the end of the incomplete file. However, We do not >> provide this >> > API >> > > now... >> > > If needed, I can add the API. >> > > >> > > 3. In this scenario, you will lose at most x Bytes data. >> If you >> > do not >> > > accept that, something like WAL is needed. (It is not very >> > complex, >> > > but I >> > > am not sure that whether it should be an embedded >> function for >> > TsFile). >> > > >> > > Up to now, we can consider that TsFile API is suitable >> for your >> > > scenario >> > > (even though we need to add a little more API if you >> desire). >> > And you >> > > can >> > > get the ability to compress data, and query data from the >> TsFile >> > rather >> > > than scan the data from the head to the tail. >> > > >> > > However, TsFile has one constraint: You can not write >> > out-of-order data >> > > into a TsFile, otherwise the query API may return >> incomplete >> > result. >> > > But I think it is ok for real applications, because I do >> not >> > think >> > > that a >> > > device can generate out-of-order data.... >> > > >> > > For example, If you write two devices' data into one >> TsFile, it >> > is ok >> > > if >> > > you write data like: >> > > - d1.t1, d1.t2, d2.t1, d2.t2, d2.t3, d1.t4, d1.t5 .... >> > > or: >> > > - d1.m1.t1, d1.m1.t2, d1.m2.t1, d1.m2.t2, d2.m1.t1 ... >> > > >> > > But you can not write data like: >> > > - d1.m1.t2, d1.m1.t1 ... >> > > >> > > I think it is a good chance to improve TsFile to make it >> more >> > suitable >> > > for >> > > real applications, so please do not hesitate to tell me >> more >> > about >> > > what you >> > > think TsFile should want to have? >> > > >> > > Best, >> > > ----------------------------------- >> > > Xiangdong Huang >> > > School of Software, Tsinghua University >> > > >> > > 黄向东 >> > > 清华大学 软件学院 >> > > >> > > >> > > Julian Feinauer <[email protected]> >> 于2019年3月4日周一 >> > 下午7:17写道： >> > > >> > > > Hi Xiangdong, >> > > > >> > > > thanks for the info. >> > > > How is it in the case when you use the Reader / Writer >> API for >> > the >> > > tsfiles >> > > > directly (or should this be considered "deprecated")? >> > > > Can these files come to corrupted state? >> > > > >> > > > One Situation where we have to deal with these >> situations is >> > "at the >> > > edge" >> > > > when we have devices inside large machines. >> > > > Usually at the end of the shift these machines (and >> therefore >> > our >> > > device) >> > > > is powered off hard, so no shutdown or >> de-initialization is >> > possible. >> > > > >> > > > Best >> > > > Julian >> > > > >> > > > Am 04.03.19, 12:14 schrieb "Xiangdong Huang" < >> > [email protected]>: >> > > > >> > > > Hi, >> > > > >> > > > IoTDB can support either on a server with 7*24 or a >> > RaspberryPi. >> > > We >> > > > have >> > > > tested both the two scenario. >> > > > >> > > > When you shutdown an IoTDB instance in force (e.g., >> power >> > off) >> > > and >> > > > restart >> > > > it again, no data loses ( if you enable the WAL). >> > > > >> > > > However, currently we do not optimize the time cost >> of the >> > > restart >> > > > process. >> > > > It is an important feature that we need to do, >> because we >> > hope >> > > IoTDB >> > > > can >> > > > support data management either on the edge devices >> or the >> > data >> > > center. >> > > > >> > > > And, the default configuration is not so suitable >> for >> > running on >> > > the >> > > > edge >> > > > device. (e.g., block size is 128MB, which is too >> large for >> > a >> > > > RaspberryPi, >> > > > and will slow down the restart process because >> there are >> > too >> > > much WAL >> > > > data >> > > > on disk). >> > > > >> > > > Best, >> > > > ----------------------------------- >> > > > Xiangdong Huang >> > > > School of Software, Tsinghua University >> > > > >> > > > 黄向东 >> > > > 清华大学 软件学院 >> > > > >> > > > >> > > > Tim Mitsch <[email protected]> >> 于2019年3月4日周一 >> > > 下午6:53写道： >> > > > >> > > > > Hello development-team >> > > > > >> > > > > First of all thanks for developing this kind of >> > interesting >> > > project >> > > > and >> > > > > bringing it into apache incubator. >> > > > > >> > > > > I have a question regarding the place of >> operation and >> > > robustness: >> > > > > >> > > > > * Is iotDB concepted as application on a >> server >> > which is >> > > running >> > > > 24/7 >> > > > > or >> > > > > * Is it also possible to run it on a device >> like >> > > RaspberryPi or >> > > > IPC, >> > > > > where operation can interrupt. >> > > > > I’m asking because i’m searching for solution for >> a >> > temporary >> > > > storage that >> > > > > is robust against spontaneous interrupt, e.g. >> switch off >> > > electricity >> > > > > without regular shutdown of OS – have u tested >> something >> > like >> > > this >> > > > yet? >> > > > > >> > > > > Best regards >> > > > > Tim >> > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > >> > >> > >> >> >>
