Re: Operation and robustness of iotDB

2019-06-26 Thread Xiangdong Huang
Hi, If someone uses NativeRestorableIOWriter, please notice this PR [1]. The behavior of the class changes a little: 1. if append == false, the writer will be closed after it is created. a. if the tsfile is broken, the behavior of the class is fixing the broken tsfile and make it sealed. b

Re: Operation and robustness of iotDB

2019-03-08 Thread Xiangdong Huang
Hi, This PR has opened for several days. I will merge this PR into the master. By the way, I notice that there is no document about TsFile API. Actually we have previous versions about the API introduction and example on https://github.com/thulab/tsfile/wiki , but I am not sure whether they catch

Re: Operation and robustness of iotDB

2019-03-07 Thread Xiangdong Huang
t; >> Thanks >> XuYi >> >> iPhoneから送信 >> >> 2019/03/08 1:41、Julian Feinauer のメール: >> >> > Hi, >> > >> > Yes this is what I meant. >> > >> > Julian >> > >> > Von mei

Re: Operation and robustness of iotDB

2019-03-07 Thread Xiangdong Huang
What do you think of this @xiangdong > > Thanks > XuYi > > iPhoneから送信 > > 2019/03/08 1:41、Julian Feinauer のメール: > > > Hi, > > > > Yes this is what I meant. > > > > Julian > > > > Von mei

Re: Operation and robustness of iotDB

2019-03-07 Thread Julian Feinauer
Julian Feinauer のメール: > Hi, > > Yes this is what I meant. > > Julian > > Von meinem Mobiltelefon gesendet > > > Ursprüngliche Nachricht -------- > Betreff: Re: Operation and robustness of iotDB > Von: 徐毅

Re: AW: Operation and robustness of iotDB

2019-03-07 Thread Xu yi
t I meant. > > Julian > > Von meinem Mobiltelefon gesendet > > > Ursprüngliche Nachricht -------- > Betreff: Re: Operation and robustness of iotDB > Von: 徐毅 > An: dev@iotdb.apache.org > Cc: > > Hi, > In the definition of ChunkGroup, what is

AW: Operation and robustness of iotDB

2019-03-07 Thread Julian Feinauer
Hi, Yes this is what I meant. Julian Von meinem Mobiltelefon gesendet Ursprüngliche Nachricht Betreff: Re: Operation and robustness of iotDB Von: 徐毅 An: dev@iotdb.apache.org Cc: Hi, In the definition of ChunkGroup, what is the meaning of 'share one time signal'

Re: Operation and robustness of iotDB

2019-03-07 Thread 徐毅
Hi, In the definition of ChunkGroup, what is the meaning of 'share one time signal'? Do these measurements share same timestamps? Thanks XuYi On 3/8/2019 01:11,Julian Feinauer wrote: Hey Xiangdong, hey all, I like the documentation much. The only thing I'm a bit unsure is about the names (as th

Re: Operation and robustness of iotDB

2019-03-07 Thread Julian Feinauer
Hey Xiangdong, hey all, I like the documentation much. The only thing I'm a bit unsure is about the names (as there is no clarification). So, before I update it with any wrong information I would like to ensure that I have the correct understanding. I assume that most naming is similar to Parqu

Re: Operation and robustness of iotDB

2019-03-06 Thread Xiangdong Huang
est code and > give you feedback! > > Thank you! > Julian > > Von meinem Mobiltelefon gesendet > > > ---- Ursprüngliche Nachricht ---- > Betreff: Re: Operation and robustness of iotDB > Von: Xiangdong Huang > An: dev@iotdb.apache.org > Cc: > > Hi, > &

AW: Operation and robustness of iotDB

2019-03-05 Thread Julian Feinauer
endet Ursprüngliche Nachricht Betreff: Re: Operation and robustness of iotDB Von: Xiangdong Huang An: dev@iotdb.apache.org Cc: Hi, I have added a new TsFileIOWriter, which supports to recover data from a broken TsFile (I mean, a TsFile that is not closed correctly). The PR is

Re: Operation and robustness of iotDB

2019-03-05 Thread Xiangdong Huang
Hi, I have added a new TsFileIOWriter, which supports to recover data from a broken TsFile (I mean, a TsFile that is not closed correctly). The PR is https://github.com/apache/incubator-iotdb/pull/87. You can use the new feature by: ``` NativeRestorableIOWriter rWriter = new NativeRestorableI

Re: Operation and robustness of iotDB

2019-03-05 Thread Julian Feinauer
Hey, thank you fort he link... I did not know of this.. this is exactly what I was looking for! Julian PS.: Looking forward to your PR : ) Am 05.03.19, 12:26 schrieb "Xiangdong Huang" : Hi, 1. We have a document to introduce that: https://cwiki.apache.org/confluence/display/

Re: Operation and robustness of iotDB

2019-03-05 Thread Xiangdong Huang
Hi, 1. We have a document to introduce that: https://cwiki.apache.org/confluence/display/IOTDB/TsFile+Format 2. The new API for recovering data is almost done. I am writing the UTs now. Maybe I can submit a PR tonight (if everything is fine...) Best, --- Xiangdong

Re: Operation and robustness of iotDB

2019-03-05 Thread Julian Feinauer
Hi Xiangdong, that sounds excellent. Do you have a short overview of how the file format is designed on disk? I know that its somewhat similar to parquet but I did not find more details. Basically what would suffice for us would be something like skipping an invalid column group (or how you name

Re: Operation and robustness of iotDB

2019-03-04 Thread Xiangdong Huang
Hi, If so, I think I need to add a new API to allow you continue to write data in an existing but not closed correctly TsFile. Then everything is fine for you :D Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Julian Feinauer 于

Re: Operation and robustness of iotDB

2019-03-04 Thread Julian Feinauer
Hey Xiangdong, thanks for the great explanation. And in fact, I agree with you that it would be best if we start to play around with it and reply all our findings or wishes back to this list (in fact that proved to be beneficial in plc4x as well). You confirm my thoughts about the two "levels"

Re: Operation and robustness of iotDB

2019-03-04 Thread Xiangdong Huang
Hi, TsFile API is not deprecated. In fact, it is designed for this scenario and MapReduce/Spark computing. If you just use Reader and Writer API, there is something you need to know: Let's suppose your block size is x Bytes, (tsfile-format.properties: group_size_in_byte). 1. If you write data a

Re: Operation and robustness of iotDB

2019-03-04 Thread Julian Feinauer
Hi Xiangdong, thanks for the info. How is it in the case when you use the Reader / Writer API for the tsfiles directly (or should this be considered "deprecated")? Can these files come to corrupted state? One Situation where we have to deal with these situations is "at the edge" when we have de

Re: Operation and robustness of iotDB

2019-03-04 Thread Xiangdong Huang
Hi, IoTDB can support either on a server with 7*24 or a RaspberryPi. We have tested both the two scenario. When you shutdown an IoTDB instance in force (e.g., power off) and restart it again, no data loses ( if you enable the WAL). However, currently we do not optimize the time cost of the resta

Operation and robustness of iotDB

2019-03-04 Thread Tim Mitsch
Hello development-team First of all thanks for developing this kind of interesting project and bringing it into apache incubator. I have a question regarding the place of operation and robustness: * Is iotDB concepted as application on a server which is running 24/7 or * Is it also poss