[jira] [Closed] (IOTDB-357) NullPointerException in ActiveTimeSeriesCounter

[Jialin Qiao (Jira)]

 [ 
https://issues.apache.org/jira/browse/IOTDB-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jialin Qiao closed IOTDB-357.
-
Resolution: Fixed

> NullPointerException in ActiveTimeSeriesCounter
> ---
>
> Key: IOTDB-357
> URL: https://issues.apache.org/jira/browse/IOTDB-357
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Jialin Qiao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Start a empty IoTDB, execute 3 sql:
> {code:java}
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d2.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA;
> insert into root.turbine.d2(timestamp,s0) values(2,25.3);{code}
> Kill the server and restart, then you will see:
> {code:java}
> 11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group 
> root.turbine- registers active time series root.turbine.d2.s0 
> failed11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group 
> root.turbine- registers active time series root.turbine.d2.s0 
> failedjava.lang.NullPointerException: null at 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.offer(ActiveTimeSeriesCounter.java:65)
>  at 
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:87)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
>  at 
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)11:37:45.158 
> [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Update 
> root.turbine- active ratio failedjava.lang.NullPointerException: null at 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.updateActiveRatio(ActiveTimeSeriesCounter.java:79)
>  at 
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:91)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
>  at 
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> {code}
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (IOTDB-360) A bug of deadlock in `CompressionRatio`

[Tianan Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/IOTDB-360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tianan Li closed IOTDB-360.
---

> A bug of deadlock in `CompressionRatio`
> ---
>
> Key: IOTDB-360
> URL: https://issues.apache.org/jira/browse/IOTDB-360
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Tianan Li
>Priority: Blocker
>  Labels: pull-request-available
> Attachments: a0fb49ad609f2f0a5dfbbc9daf65bbff.jpg
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In the class of `CompressionRatio`,  it may cause a deadlock in very low 
> probability.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [jira] [Created] (IOTDB-357) NullPointerException in ActiveTimeSeriesCounter

[李天安]

Hi,

I have found that it’s a hidden bug in method `redoLogs()` of class 
`TsFileRecoverPerformer`.

I have pull a request to fix that. The pr link is 
https://github.com/apache/incubator-iotdb/pull/642 


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> 2019年12月10日 上午11:41，Jialin Qiao (Jira)  写道：
> 
> Jialin Qiao created IOTDB-357:
> -
> 
> Summary: NullPointerException in ActiveTimeSeriesCounter
> Key: IOTDB-357
> URL: https://issues.apache.org/jira/browse/IOTDB-357
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Jialin Qiao
> 
> 
> Start a empty IoTDB, execute 3 sql:
> {code:java}
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d2.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA;
> insert into root.turbine.d2(timestamp,s0) values(2,25.3);{code}
> Kill the server and restart, then you will see:
> {code:java}
> 11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group 
> root.turbine- registers active time series root.turbine.d2.s0 
> failed11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group 
> root.turbine- registers active time series root.turbine.d2.s0 
> failedjava.lang.NullPointerException: null at 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.offer(ActiveTimeSeriesCounter.java:65)
>  at 
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:87)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
>  at 
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)11:37:45.158 
> [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Update 
> root.turbine- active ratio failedjava.lang.NullPointerException: null at 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.updateActiveRatio(ActiveTimeSeriesCounter.java:79)
>  at 
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:91)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
>  at 
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> {code}
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)

[Discuss] about collecting info to know who are using IoTDB

[Xiangdong Huang]

Hi,

I notice that ShardingSpehere incubating project has a page for collecting
who are using IoTDB[1].  They use github issue to collect the data [2].

I think it is helpful for the project graduation

So, if IoTDB also collects the info, how to do that? Any ideas?

1. We do not open the issue module on Github.
2. For Chinese users, many of them do not have a Apache JIRA account and
opening the jira may requires some skills in China because of Chinese
network problem.

So, how about opening a tickets on jira for collecting users out of China,
and opening a online document, e.g., QQ document, to collect users in China
(But it is hard to record that who maintains the document. (Everyone can
modify the document, while on JIRA and github, the contents is binding with
accounts)).

[1] https://shardingsphere.apache.org/community/en/poweredby/
[2] https://github.com/sharding-sphere/sharding-sphere/issues/234

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

[jira] [Created] (IOTDB-360) A bug of deadlock in `CompressionRatio`

[Tianan Li (Jira)]

Tianan Li created IOTDB-360:
---

 Summary: A bug of deadlock in `CompressionRatio`
 Key: IOTDB-360
 URL: https://issues.apache.org/jira/browse/IOTDB-360
 Project: Apache IoTDB
  Issue Type: Bug
Reporter: Tianan Li
 Attachments: a0fb49ad609f2f0a5dfbbc9daf65bbff.jpg

In the class of `CompressionRatio`,  it may cause a deadlock in very low 
probability.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Code refactoring of Query

[Lei Rui]

Hi,


+1 with Jialin Qiao. 
I think the ongoing query refactor work will cover the following points:
1) refactor the ordinary query, by 
a) changing all operators in query plans to be batch operators
b) replanning the reading of data from storage, taking into consideration 
the interactions both between the sequence and overflow data and within 
overflow data


2) refactor the aggregate and groupBy queries, by
a) abstracting new aggregate operators or interfaces to enable both point, 
page and chunk level aggregate reading
b) replanning the reading of data from storage, taking into consideration 
the interactions both between the sequence and overflow data and within 
overflow data


Lastly, sorry @atoiLiu I don't quite get what you mean. `Limit` query is 
already realized I think.




Best,
Lei Rui




On 12/10/2019 22:37，atoiLiu wrote：
hi,
Offset command is sometimes found in SQL, so hopefully it will also provide a 
jump query


在 2019年12月10日，下午7:16，Jialin Qiao  写道：

Hi,

Code refactoring is inevitable when building a large system. The read/write
of TsFile, storage engine of the server have been refactored. Now, it's
time to refactor the query engine in the server.

Currently, the query is in a tuple-at-a-time manner. In the meantime, the
interface is in chaos.
To improve the query speed, a batch-at-a-time(vectorization) iteration is
needed.

I have opened a branch with Lei Rui: f_batch_reader. The existing
interfaces are simplified and IBatchReader is added. Welcome to work on
this branch for query optimization.

Thanks,
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

Re: [jira] [Created] (IOTDB-346) StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime

[李天安]

Hi, 

After I replicate your experiments, I found it’s a bug of renaming tsfile when 
it has an out-of-order file name with current tsfiles. I will add a new ut test 
and pull a request to master soon.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> 2019年12月9日 上午12:14，Lei Rui (Jira)  写道：
> 
> Lei Rui created IOTDB-346:
> -
> 
> Summary: StorageGroupProcessor.sequenceFileList is ordered by 
> fileName rather than dataTime
> Key: IOTDB-346
> URL: https://issues.apache.org/jira/browse/IOTDB-346
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Lei Rui
> 
> 
> `StorageGroupProcessor.sequenceFileList` is ordered by fileName rather than 
> by time of data, as reflected in the `StorageGroupProcessor.getAllFiles` 
> method code:
> {code:java}
> tsFiles.sort(this::compareFileName);
> {code}
> 
> I use the following examples to expose the bug when the order of fileName is 
> inconsistent with that of dataTime.
> 
> First, for preparation, I created three tsfiles using the following sql:
> {code:java}
> SET STORAGE GROUP TO root.ln.wf01.wt01
> CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, 
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE, 
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32, 
> ENCODING=PLAIN
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(1, 1.1, false, 11)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(2, 2.2, true, 22)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(3, 3.3, false, 33)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(4, 4.4, false, 44)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(5, 5.5, false, 55)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(100, 100.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(150, 200.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(200, 300.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(250, 400.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(300, 500.5, false, 550)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(10, 10.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(20, 20.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(30, 30.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(40, 40.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(50, 50.5, false, 550)
> flush
> {code}
> The tsfiles created are organized in the following directory structure:
> {code:java}
> |data
> |--sequence
> |root.ln.wf01.wt01
> |--1575813520203-101-0.tsfile
> |--1575813520203-101-0.tsfile.resource
> |--1575813520669-103-0.tsfile
> |--1575813520669-103-0.tsfile.resource
> |--unsequence
> |root.ln.wf01.wt01
> |--1575813521063-105-0.tsfile
> |--1575813521063-105-0.tsfile.resource
> {code}
> ||File Name||Data Time||
> |(a) 1575813520203-101-0.tsfile|1-5|
> |(c) 1575813521063-105-0.tsfile|10-50|
> |(b) 1575813520669-103-0.tsfile|100-300|
> 
> Note how the order of fileName is inconsistent with that of dataTime.
> 
> By the way, if you look into the code, you will know how the file name is 
> generated:
> {code:java}
> System.currentTimeMillis() + IoTDBConstant.TSFILE_NAME_SEPARATOR + 
> versionController.nextVersion() + IoTDBConstant.TSFILE_NAME_SEPARATOR + "0" + 
> TSFILE_SUFFIX
> {code}
> 
> Then, I loaded the three tsfiles into another brand new IoTDB. I did two 
> experiments with different loading orders each.
> 
> In the first experiment, the tsfiles were loaded in their data time order. 
> That is,
> {code:java}
> IoTDB> load 1575813520203-101-0.tsfile // tsfile (a), with data time 1-5
> IoTDB> load 1575813521063-105-0.tsfile // tsfile (c), with data time 10-50
> IoTDB> load 1575813520669-103-0.tsfile // tsfile (b), with data time 
> 100-300{code}
> After loading successfully, I did the following query in the same client 
> window and got the wrong result:
> {code:java}
> IoTDB> select * from root
> +---+-+-+-+
> |   Time|root.ln.wf01.wt01.temperature| 
> root.ln.wf01.wt01.status|   root.ln.wf01.wt01.hardware|
> 

Re: [jira] [Created] (IOTDB-357) NullPointerException in ActiveTimeSeriesCounter

[Jialin Qiao]

master (0.10.0-SNAPSHOT)

Xiangdong Huang  于2019年12月10日周二 下午11:02写道：

> Hi, which version? 0.9.0?
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao (Jira)  于2019年12月10日周二 上午11:41写道：
>
> > Jialin Qiao created IOTDB-357:
> > -
> >
> >  Summary: NullPointerException in ActiveTimeSeriesCounter
> >  Key: IOTDB-357
> >  URL: https://issues.apache.org/jira/browse/IOTDB-357
> >  Project: Apache IoTDB
> >   Issue Type: Bug
> > Reporter: Jialin Qiao
> >
> >
> > Start a empty IoTDB, execute 3 sql:
> > {code:java}
> > SET STORAGE GROUP TO root.turbine;
> > CREATE TIMESERIES root.turbine.d2.s0 WITH DATATYPE=DOUBLE,
> > ENCODING=GORILLA;
> > insert into root.turbine.d2(timestamp,s0) values(2,25.3);{code}
> > Kill the server and restart, then you will see:
> > {code:java}
> > 11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR
> > org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group
> > root.turbine- registers active time series root.turbine.d2.s0
> > failed11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR
> > org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group
> > root.turbine- registers active time series root.turbine.d2.s0
> > failedjava.lang.NullPointerException: null at
> >
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.offer(ActiveTimeSeriesCounter.java:65)
> > at
> >
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:87)
> > at
> >
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
> > at
> >
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
> > at
> >
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
> > at
> >
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
> > at
> >
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
> > at
> >
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)11:37:45.158
> > [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR
> > org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Update
> > root.turbine- active ratio failedjava.lang.NullPointerException: null at
> >
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.updateActiveRatio(ActiveTimeSeriesCounter.java:79)
> > at
> >
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:91)
> > at
> >
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
> > at
> >
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
> > at
> >
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
> > at
> >
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
> > at
> >
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
> > at
> >
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > {code}
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
>


-- 
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

Re: StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime

[atoiLiu]

Hi，

I think the semantics of load are the same as insert, except this insert is a 
sealed file, so I think it should be dumped into iotdb as an unseq file and 
sorted in memory with the original files.

This may cause queries to be very slow, but we should prompt the user to do a 
merge command ??

> 在 2019年12月10日，下午9:04，Xiangdong Huang  写道：
> 
> Hi,
> 
> I think it is a bug in the `load` function now, and needs to be fixed
> quickly.
> 
> Firstly, let's consider that there is no `load` function.
> In this case, the files will have the same order no matter you use which
> device's timeline as the ordering dimension.
> 
> (Second, in your case, can we put the tsfile 105 into the sequence files?
> Condition: all devices in a flushing memetable can be set in a time hole of
> the sequence files.)
> 
> Third, lets's consider that if the `load` function is enable.
> 
> The worest case is that you add a file  which has two devices (device 1 and
> device2), and if you use device1's timeline to order files, it is between
> F2 and F3, while it is between F1 and F2 if you use device2's timeline.
> 
> device1: F1   F2   _HOLE__ F3
> device2: F1  __HOLE__ F2  F3
> 
> Then, why not split the file into two files?
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
> 黄向东
> 清华大学 软件学院
> 
> 
> Jialin Qiao  于2019年12月10日周二 下午7:05写道：
> 
>> Hi,
>> 
>> Things become complicated when the load file feature is introduced in
>> IoTDB. The newly added data file may contain many devices with different
>> time intervals. Therefore, one order of TsFileResources is insufficient.
>> A possible solution is to sort the TsFileResources temporarily when
>> querying.
>> 
>> Thanks,
>> Jialin Qiao
>> 
>> Lei Rui (Jira)  于2019年12月9日周一 上午12:14写道：
>> 
>>> Lei Rui created IOTDB-346:
>>> -
>>> 
>>> Summary: StorageGroupProcessor.sequenceFileList is ordered
>> by
>>> fileName rather than dataTime
>>> Key: IOTDB-346
>>> URL: https://issues.apache.org/jira/browse/IOTDB-346
>>> Project: Apache IoTDB
>>>  Issue Type: Bug
>>>Reporter: Lei Rui
>>> 
>>> 
>>> `StorageGroupProcessor.sequenceFileList` is ordered by fileName rather
>>> than by time of data, as reflected in the
>>> `StorageGroupProcessor.getAllFiles` method code:
>>> {code:java}
>>> tsFiles.sort(this::compareFileName);
>>> {code}
>>> 
>>> I use the following examples to expose the bug when the order of fileName
>>> is inconsistent with that of dataTime.
>>> 
>>> First, for preparation, I created three tsfiles using the following sql:
>>> {code:java}
>>> SET STORAGE GROUP TO root.ln.wf01.wt01
>>> CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN,
>>> ENCODING=PLAIN
>>> CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE,
>>> ENCODING=PLAIN
>>> CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32,
>>> ENCODING=PLAIN
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(1, 1.1, false, 11)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(2, 2.2, true, 22)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(3, 3.3, false, 33)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(4, 4.4, false, 44)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(5, 5.5, false, 55)
>>> flush
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(100, 100.1, false, 110)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(150, 200.2, true, 220)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(200, 300.3, false, 330)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(250, 400.4, false, 440)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(300, 500.5, false, 550)
>>> flush
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(10, 10.1, false, 110)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(20, 20.2, true, 220)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(30, 30.3, false, 330)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(40, 40.4, false, 440)
>>> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
>>> values(50, 50.5, false, 550)
>>> flush
>>> {code}
>>> The tsfiles created are organized in the following directory structure:
>>> {code:java}
>>> |data
>>> |--sequence
>>> |root.ln.wf01.wt01
>>> |--1575813520203-101-0.tsfile
>>> |--1575813520203-101-0.tsfile.resource
>>> |--1575813520669-103-0.tsfile
>>> |--1575813520669-103-0.tsfile.resource
>>> |--unsequence
>>> |root.ln.wf01.wt01
>>> 

Re: [jira] [Created] (IOTDB-358) Questions about the `group by` down-samplying clause

[Xiangdong Huang]

Hi,

> The answer is NO. If it calculated twice, it's a bug.

So, it is a typo of the document, the correct version should be [0, 1d),
[1d, 2d).

> The new group by statement is like this:  select count(*) from root where
> time >= 1 and time <= 100 group by ([0, 100], 20ms, 20ms)
> The 0 in [0, 100] is used as the start point. This statement should
satisfy
> most scenarios.

Ah... so you give up the ability that letting users define the start
point?  Besides, I  feel confusing about: `time >= 1 and time <= 100 group
by ([0, 100]`, if `1` is for the time filter, and `0` is for the start time
point, so why we have two `100`? What it is mean about `time >= 1 and time
<= 99 group by ([0, 100]`, or `time >= 1 and time <= 100 group by ([0, 99]`
?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2019年12月10日周二 下午6:55写道：

> Hi,
>
> > My question is, is the data point at 1d calculated twice?
>
> The answer is NO. If it calculated twice, it's a bug.
>
> > it loses the ability to define the start point of the timestamp for
> splitting the time window.
>
> The new group by statement is like this:  select count(*) from root where
> time >= 1 and time <= 100 group by ([0, 100], 20ms, 20ms)
> The 0 in [0, 100] is used as the start point. This statement should satisfy
> most scenarios.
>
> Thanks,
> Jialin Qiao
>
> xiangdong Huang (Jira)  于2019年12月10日周二 下午4:50写道：
>
> > xiangdong Huang created IOTDB-358:
> > -
> >
> >  Summary: Questions about the `group by` down-samplying
> clause
> >  Key: IOTDB-358
> >  URL: https://issues.apache.org/jira/browse/IOTDB-358
> >  Project: Apache IoTDB
> >   Issue Type: Bug
> > Affects Versions: 0.9.0
> > Reporter: xiangdong Huang
> >
> >
> > When reading issue  #331, I read the `group by` document again and find
> > some confusing contents:
> >
> > V0.9:
> >
> > [http://iotdb.apache.org/#/Documents/0.9.0/chap5/sec2]
> >  # (minor), typo, Fig 5.2 rather than Fig 3.2
> >  # The first parameter of the GROUP BY statement above is the time
> > interval for dividing the time axis. Taking this parameter (1d) as time
> > interval and the default origin as the dividing origin, the time axis is
> > divided into several continuous intervals, which are [0,1d], [1d, 2d],
> [2d,
> > 3d], etc.
> >
> > My question is, is the data point at 1d calculated twice?
> >
> >
> >
> > V0.10:
> > [http://iotdb.apache.org/#/Documents/progress/chap5/sec2]
> >
> > the syntax is quite different with v0.9.0. I originally thought it was an
> > improvement comparing with 0.9, but when I read the document, I find it
> > loses the ability to define the start point of the timestamp for
> splitting
> > the time window.
> >
> > Can someone tell me why to design like this?
> >
> >
> >
> >
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
>
>
> --
> —
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>

Re: [jira] [Created] (IOTDB-359) [0.9.0] TsFile Sketch Tool prints wrong results when the tsfile is very large

[Xiangdong Huang]

Hi,

Does this bug only affects the sketch-print-tool, or the TsFile actually?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Lei Rui (Jira)  于2019年12月10日周二 下午5:53写道：

> Lei Rui created IOTDB-359:
> -
>
>  Summary: [0.9.0] TsFile Sketch Tool prints wrong results when
> the tsfile is very large
>  Key: IOTDB-359
>  URL: https://issues.apache.org/jira/browse/IOTDB-359
>  Project: Apache IoTDB
>   Issue Type: Bug
> Reporter: Lei Rui
>
>
> I used the TsFile sketch tool on a TsFile sized 2.2G, and the printed
> result is wrong for that the ChunkGroupMetaDatas are not printed in their
> address ascending order.
>
> I analyzed this bug and find that `int` is used when comparing the address
> of ChunkGroupMetaDatas. However, the address should be long data type.
>
> I'll fix this bug soon :)
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

Re: [jira] [Created] (IOTDB-357) NullPointerException in ActiveTimeSeriesCounter

[Xiangdong Huang]

Hi, which version? 0.9.0?
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao (Jira)  于2019年12月10日周二 上午11:41写道：

> Jialin Qiao created IOTDB-357:
> -
>
>  Summary: NullPointerException in ActiveTimeSeriesCounter
>  Key: IOTDB-357
>  URL: https://issues.apache.org/jira/browse/IOTDB-357
>  Project: Apache IoTDB
>   Issue Type: Bug
> Reporter: Jialin Qiao
>
>
> Start a empty IoTDB, execute 3 sql:
> {code:java}
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d2.s0 WITH DATATYPE=DOUBLE,
> ENCODING=GORILLA;
> insert into root.turbine.d2(timestamp,s0) values(2,25.3);{code}
> Kill the server and restart, then you will see:
> {code:java}
> 11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group
> root.turbine- registers active time series root.turbine.d2.s0
> failed11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group
> root.turbine- registers active time series root.turbine.d2.s0
> failedjava.lang.NullPointerException: null at
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.offer(ActiveTimeSeriesCounter.java:65)
> at
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:87)
> at
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
> at
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
> at
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
> at
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
> at
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
> at
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)11:37:45.158
> [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Update
> root.turbine- active ratio failedjava.lang.NullPointerException: null at
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.updateActiveRatio(ActiveTimeSeriesCounter.java:79)
> at
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:91)
> at
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
> at
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
> at
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
> at
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
> at
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
> at
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

Re: Code refactoring of Query

[atoiLiu]

hi,
Offset command is sometimes found in SQL, so hopefully it will also provide a 
jump query


> 在 2019年12月10日，下午7:16，Jialin Qiao  写道：
> 
> Hi,
> 
> Code refactoring is inevitable when building a large system. The read/write
> of TsFile, storage engine of the server have been refactored. Now, it's
> time to refactor the query engine in the server.
> 
> Currently, the query is in a tuple-at-a-time manner. In the meantime, the
> interface is in chaos.
> To improve the query speed, a batch-at-a-time(vectorization) iteration is
> needed.
> 
> I have opened a branch with Lei Rui: f_batch_reader. The existing
> interfaces are simplified and IBatchReader is added. Welcome to work on
> this branch for query optimization.
> 
> Thanks,
> —
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院

[jira] [Closed] (IOTDB-190) switch to ANTLR v4

[Yuyuan KANG (Jira)]

 [ 
https://issues.apache.org/jira/browse/IOTDB-190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuyuan KANG closed IOTDB-190.
-
Resolution: Fixed

> switch to ANTLR v4
> --
>
> Key: IOTDB-190
> URL: https://issues.apache.org/jira/browse/IOTDB-190
> Project: Apache IoTDB
>  Issue Type: Improvement
>Reporter: Yuyuan KANG
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, the system uses ANTLR v3 for query parsing. However, ANTLR v4 has 
> some advantages over ANTLR v3.
>  # ANTLRv4 uses visitor and listener mechanism，which allows the grammer 
> definition independent from programming language (e.g. Java). So it is much 
> easier to maintain. 
>  # In Antlr v3, developers explicitly defines the structure of the tree to 
> build using operator '->'. However, Antlr v4 hides the tree structure from 
> developers, which simplifies grammer definition and increases the code 
> readability.
>  # ANTLR v3’s LL(* ) parsing strategy is weaker than v4’s ALL(* ), so v3 
> sometimes
>  relied on backtracking to properly parse input phrases.
>  # ANTLR v3 cannot handle left-recursive rules, while ANTLR v4 can now handle 
> direct left recursion. Therefore, ANTLR v4 can handle more grammer rules.
>  # ANTLR v4 works faster than ANTLR v3.
> So it is necessary to switch to ANTLR v4



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Docker image uploaded under the account of apache

[Xiangdong Huang]

Hi all,

Um I was trying to contact the owners of these docker repos.

Sadly, only hxdiotdb (it is me..) and xingtanzjr are contacted.

I also checked these repos.

[1], [2] and [3] are fine, both of them mentioned the image is for Apache
IoTDB, and showed iotdb's website.

I can not contact to [4] and [5] (DockerHub does not share owners' info,
even the email).

[4] is indeed an Apache IoTDB image, but sounds [5] is not an Apache IoTDB
image.

[1] https://hub.docker.com/r/hxdiotdb/iotdb <
https://hub.docker.com/r/hxdiotdb/iotdb>
[2] https://hub.docker.com/r/dmonesr/iotdb <
https://hub.docker.com/r/dmonesr/iotdb>
[3] https://hub.docker.com/r/xingtanzjr/iotdb
 <
https://hub.docker.com/r/xingtanzjr/iotdb>
[4] https://hub.docker.com/r/fudanyuan/iotdb <
https://hub.docker.com/r/fudanyuan/iotdb>
[5] https://hub.docker.com/r/zongxiongchen/iotdb-core <
https://hub.docker.com/r/zongxiongchen/iotdb-core>

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Xiangdong Huang  于2019年12月9日周一 下午10:04写道：

> Hi,
>
> > Nor do they need to, often a friendly email will achieve a good result
> or an outcome that works for everyone. If a solution can’t be found then
> brand can deal with it.
>
> If contacting with the owner, do we need to CC to the private@ mailing
> list or this mailing list?
>
> Best,
> --
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> Kevin A. McGrail  于2019年12月9日周一 下午9:35写道：
>
>> What defines a major branding issue?  Are they to police the internet?
>> How are they to defermine they are major or not?  These are very vague
>> instructions to a podling about what to do.
>>
>> On 12/9/2019 2:58 AM, Justin Mclean wrote:
>> > Hi,
>> >
>> >> I understand that request but that isn't a graduation issue if someone
>> >> doesn't listen to them.
>> > But they do need to contact them and look out for issues like this, I
>> didn’t say that there needs to be no branding issues at graduation=, just
>> that the podling is aware of any major ones and be dealing with them.
>> >
>> > Thanks,
>> > Justin
>>
>> --
>> Kevin A. McGrail
>> kmcgr...@apache.org
>>
>> Member, Apache Software Foundation
>> Chair Emeritus Apache SpamAssassin Project
>> https://www.linkedin.com/in/kmcgrail - 703.798.0171
>>
>>

Re: [jira] [Created] (IOTDB-346) StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime

[Xiangdong Huang]

Hi,

I think it is a bug in the `load` function now, and needs to be fixed
quickly.

Firstly, let's consider that there is no `load` function.
In this case, the files will have the same order no matter you use which
device's timeline as the ordering dimension.

(Second, in your case, can we put the tsfile 105 into the sequence files?
Condition: all devices in a flushing memetable can be set in a time hole of
the sequence files.)

Third, lets's consider that if the `load` function is enable.

The worest case is that you add a file  which has two devices (device 1 and
device2), and if you use device1's timeline to order files, it is between
F2 and F3, while it is between F1 and F2 if you use device2's timeline.

device1: F1   F2   _HOLE__ F3
device2: F1  __HOLE__ F2  F3

Then, why not split the file into two files?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2019年12月10日周二 下午7:05写道：

> Hi,
>
> Things become complicated when the load file feature is introduced in
> IoTDB. The newly added data file may contain many devices with different
> time intervals. Therefore, one order of TsFileResources is insufficient.
> A possible solution is to sort the TsFileResources temporarily when
> querying.
>
> Thanks,
> Jialin Qiao
>
> Lei Rui (Jira)  于2019年12月9日周一 上午12:14写道：
>
> > Lei Rui created IOTDB-346:
> > -
> >
> >  Summary: StorageGroupProcessor.sequenceFileList is ordered
> by
> > fileName rather than dataTime
> >  Key: IOTDB-346
> >  URL: https://issues.apache.org/jira/browse/IOTDB-346
> >  Project: Apache IoTDB
> >   Issue Type: Bug
> > Reporter: Lei Rui
> >
> >
> > `StorageGroupProcessor.sequenceFileList` is ordered by fileName rather
> > than by time of data, as reflected in the
> > `StorageGroupProcessor.getAllFiles` method code:
> > {code:java}
> > tsFiles.sort(this::compareFileName);
> > {code}
> > 
> > I use the following examples to expose the bug when the order of fileName
> > is inconsistent with that of dataTime.
> >
> > First, for preparation, I created three tsfiles using the following sql:
> > {code:java}
> > SET STORAGE GROUP TO root.ln.wf01.wt01
> > CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN,
> > ENCODING=PLAIN
> > CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE,
> > ENCODING=PLAIN
> > CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32,
> > ENCODING=PLAIN
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(1, 1.1, false, 11)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(2, 2.2, true, 22)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(3, 3.3, false, 33)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(4, 4.4, false, 44)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(5, 5.5, false, 55)
> > flush
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(100, 100.1, false, 110)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(150, 200.2, true, 220)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(200, 300.3, false, 330)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(250, 400.4, false, 440)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(300, 500.5, false, 550)
> > flush
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(10, 10.1, false, 110)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(20, 20.2, true, 220)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(30, 30.3, false, 330)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(40, 40.4, false, 440)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> > values(50, 50.5, false, 550)
> > flush
> > {code}
> > The tsfiles created are organized in the following directory structure:
> > {code:java}
> > |data
> > |--sequence
> > |root.ln.wf01.wt01
> > |--1575813520203-101-0.tsfile
> > |--1575813520203-101-0.tsfile.resource
> > |--1575813520669-103-0.tsfile
> > |--1575813520669-103-0.tsfile.resource
> > |--unsequence
> > |root.ln.wf01.wt01
> > |--1575813521063-105-0.tsfile
> > |--1575813521063-105-0.tsfile.resource
> > {code}
> > ||File Name||Data Time||
> > |(a) 1575813520203-101-0.tsfile|1-5|
> > |(c) 1575813521063-105-0.tsfile|10-50|
> > |(b) 1575813520669-103-0.tsfile|100-300|
> >
> > Note how the order of fileName is inconsistent with that of dataTime.
> >
> > By the way, if you look into the code, you will know how the file name is
> > generated:
> > {code:java}
> 

[jira] [Closed] (IOTDB-338) Meet error when disconnecting with the server

[Jialin Qiao (Jira)]

 [ 
https://issues.apache.org/jira/browse/IOTDB-338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jialin Qiao closed IOTDB-338.
-
Resolution: Fixed

[https://github.com/apache/incubator-iotdb/pull/637]

> Meet error when disconnecting with the server
> -
>
> Key: IOTDB-338
> URL: https://issues.apache.org/jira/browse/IOTDB-338
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Jialin Qiao
>Priority: Major
>
> The client-thread in server meet error when the client disconnect with server:
>  
> {code:java}
> //代码占位符
> 09:09:42.140 [pool-2-IoTDB-JDBC-Client-thread-1] ERROR 
> org.apache.thrift.server.TThreadPoolServer - Thrift error occurred during 
> processing of message.09:09:42.140 [pool-2-IoTDB-JDBC-Client-thread-1] ERROR 
> org.apache.thrift.server.TThreadPoolServer - Thrift error occurred during 
> processing of message.org.apache.thrift.transport.TTransportException: null 
> at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425) 
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321) 
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
>  at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Code refactoring of Query

[Jialin Qiao]

Hi,

Code refactoring is inevitable when building a large system. The read/write
of TsFile, storage engine of the server have been refactored. Now, it's
time to refactor the query engine in the server.

Currently, the query is in a tuple-at-a-time manner. In the meantime, the
interface is in chaos.
To improve the query speed, a batch-at-a-time(vectorization) iteration is
needed.

I have opened a branch with Lei Rui: f_batch_reader. The existing
interfaces are simplified and IBatchReader is added. Welcome to work on
this branch for query optimization.

Thanks,
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

Re: [jira] [Created] (IOTDB-346) StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime

[Jialin Qiao]

Hi,

Things become complicated when the load file feature is introduced in
IoTDB. The newly added data file may contain many devices with different
time intervals. Therefore, one order of TsFileResources is insufficient.
A possible solution is to sort the TsFileResources temporarily when
querying.

Thanks,
Jialin Qiao

Lei Rui (Jira)  于2019年12月9日周一 上午12:14写道：

> Lei Rui created IOTDB-346:
> -
>
>  Summary: StorageGroupProcessor.sequenceFileList is ordered by
> fileName rather than dataTime
>  Key: IOTDB-346
>  URL: https://issues.apache.org/jira/browse/IOTDB-346
>  Project: Apache IoTDB
>   Issue Type: Bug
> Reporter: Lei Rui
>
>
> `StorageGroupProcessor.sequenceFileList` is ordered by fileName rather
> than by time of data, as reflected in the
> `StorageGroupProcessor.getAllFiles` method code:
> {code:java}
> tsFiles.sort(this::compareFileName);
> {code}
> 
> I use the following examples to expose the bug when the order of fileName
> is inconsistent with that of dataTime.
>
> First, for preparation, I created three tsfiles using the following sql:
> {code:java}
> SET STORAGE GROUP TO root.ln.wf01.wt01
> CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN,
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE,
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32,
> ENCODING=PLAIN
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(1, 1.1, false, 11)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(2, 2.2, true, 22)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(3, 3.3, false, 33)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(4, 4.4, false, 44)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(5, 5.5, false, 55)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(100, 100.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(150, 200.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(200, 300.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(250, 400.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(300, 500.5, false, 550)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(10, 10.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(20, 20.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(30, 30.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(40, 40.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(50, 50.5, false, 550)
> flush
> {code}
> The tsfiles created are organized in the following directory structure:
> {code:java}
> |data
> |--sequence
> |root.ln.wf01.wt01
> |--1575813520203-101-0.tsfile
> |--1575813520203-101-0.tsfile.resource
> |--1575813520669-103-0.tsfile
> |--1575813520669-103-0.tsfile.resource
> |--unsequence
> |root.ln.wf01.wt01
> |--1575813521063-105-0.tsfile
> |--1575813521063-105-0.tsfile.resource
> {code}
> ||File Name||Data Time||
> |(a) 1575813520203-101-0.tsfile|1-5|
> |(c) 1575813521063-105-0.tsfile|10-50|
> |(b) 1575813520669-103-0.tsfile|100-300|
>
> Note how the order of fileName is inconsistent with that of dataTime.
>
> By the way, if you look into the code, you will know how the file name is
> generated:
> {code:java}
> System.currentTimeMillis() + IoTDBConstant.TSFILE_NAME_SEPARATOR +
> versionController.nextVersion() + IoTDBConstant.TSFILE_NAME_SEPARATOR + "0"
> + TSFILE_SUFFIX
> {code}
> 
> Then, I loaded the three tsfiles into another brand new IoTDB. I did two
> experiments with different loading orders each.
>
> In the first experiment, the tsfiles were loaded in their data time order.
> That is,
> {code:java}
> IoTDB> load 1575813520203-101-0.tsfile // tsfile (a), with data time 1-5
> IoTDB> load 1575813521063-105-0.tsfile // tsfile (c), with data time 10-50
> IoTDB> load 1575813520669-103-0.tsfile // tsfile (b), with data time
> 100-300{code}
> After loading successfully, I did the following query in the same client
> window and got the wrong result:
> {code:java}
> IoTDB> select * from root
>
> +---+-+-+-+
> |   Time|root.ln.wf01.wt01.temperature|
>  root.ln.wf01.wt01.status|   root.ln.wf01.wt01.hardware|
>
> 

Re: Add data to TSExecuteStatementResp

[Jialin Qiao]

Hi,

I have reviewed this PR and give some suggestions. This is a good
optimization.

Thanks,
Jialin Qiao

Xiangdong Huang  于2019年12月10日周二 上午8:45写道：

> Hi,
>
> I think the idea is ok. And I quick review the codes, sounds no problem.
> Does anyone who is more familiar with the query process can review the
> codes?
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> atoiLiu  于2019年12月9日周一 下午5:49写道：
>
> > Currently, when executing an SQL statement through JDBC, it is done in
> two
> > steps:
> >
> > 1. Query metadata set hasResultSet = true
> >
> > 2. When the client determines that hasResultSet = true, it will initiate
> > the next data query and call the fetchResult method of the server
> >
> > I think this step can be optimized to reduce one request to the server,
> > especially if the data is empty or the total amount of data is less than
> > the fetchSize
> >
> > So I added the TSQueryDataSet to TSExecuteStatementResp so that the
> client
> > could traverse the data directly.
> >
> >
> >
> > I hope my idea can contribute to the community. Can anyone review it for
> > me?
> >
> > Pr:
> > https://github.com/apache/incubator-iotdb/pull/631 <
> > https://github.com/apache/incubator-iotdb/pull/631>
> >
> >
> > In addition, I found a new problem.
> > When I input a random random SQL in client, the server would throw an
> > antlr error, which could not be caught by the expected SQLParserException
> > and was kindly prompted to the user.
> > I think adding  try...catch to the parseSQLToPhysicalPlan should solve
> > this problem.  I don’t know if i am right ,so sorry.
>


-- 
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

Re: [jira] [Created] (IOTDB-358) Questions about the `group by` down-samplying clause

[Jialin Qiao]

Hi,

> My question is, is the data point at 1d calculated twice?

The answer is NO. If it calculated twice, it's a bug.

> it loses the ability to define the start point of the timestamp for
splitting the time window.

The new group by statement is like this:  select count(*) from root where
time >= 1 and time <= 100 group by ([0, 100], 20ms, 20ms)
The 0 in [0, 100] is used as the start point. This statement should satisfy
most scenarios.

Thanks,
Jialin Qiao

xiangdong Huang (Jira)  于2019年12月10日周二 下午4:50写道：

> xiangdong Huang created IOTDB-358:
> -
>
>  Summary: Questions about the `group by` down-samplying clause
>  Key: IOTDB-358
>  URL: https://issues.apache.org/jira/browse/IOTDB-358
>  Project: Apache IoTDB
>   Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: xiangdong Huang
>
>
> When reading issue  #331, I read the `group by` document again and find
> some confusing contents:
>
> V0.9:
>
> [http://iotdb.apache.org/#/Documents/0.9.0/chap5/sec2]
>  # (minor), typo, Fig 5.2 rather than Fig 3.2
>  # The first parameter of the GROUP BY statement above is the time
> interval for dividing the time axis. Taking this parameter (1d) as time
> interval and the default origin as the dividing origin, the time axis is
> divided into several continuous intervals, which are [0,1d], [1d, 2d], [2d,
> 3d], etc.
>
> My question is, is the data point at 1d calculated twice?
>
>
>
> V0.10:
> [http://iotdb.apache.org/#/Documents/progress/chap5/sec2]
>
> the syntax is quite different with v0.9.0. I originally thought it was an
> improvement comparing with 0.9, but when I read the document, I find it
> loses the ability to define the start point of the timestamp for splitting
> the time window.
>
> Can someone tell me why to design like this?
>
>
>
>
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>


-- 
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

Re: Considering the first Meetup of IoTDB in Beijing

[Jialin Qiao]

Hi,

I would like to give a talk about the architecture and some components of
IoTDB.

Thanks,
Jialin Qiao

Xiangdong Huang  于2019年12月10日周二 下午5:24写道：

> Hi all,
>
> Some guys from Apache Pulsar community contacted me to host a meetup
> together in Beijing on 4 Jun 2020 in Beijing (using Chinese).
>
> Does someone want to share a talk about IoTDB? (How to use IoTDB, IoTDB's
> design etc..)
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>


-- 
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

[jira] [Created] (IOTDB-359) [0.9.0] TsFile Sketch Tool prints wrong results when the tsfile is very large

[Lei Rui (Jira)]

Lei Rui created IOTDB-359:
-

 Summary: [0.9.0] TsFile Sketch Tool prints wrong results when the 
tsfile is very large
 Key: IOTDB-359
 URL: https://issues.apache.org/jira/browse/IOTDB-359
 Project: Apache IoTDB
  Issue Type: Bug
Reporter: Lei Rui


I used the TsFile sketch tool on a TsFile sized 2.2G, and the printed result is 
wrong for that the ChunkGroupMetaDatas are not printed in their address 
ascending order.

I analyzed this bug and find that `int` is used when comparing the address of 
ChunkGroupMetaDatas. However, the address should be long data type.

I'll fix this bug soon :)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Considering the first Meetup of IoTDB in Beijing

[Xiangdong Huang]

Hi all,

Some guys from Apache Pulsar community contacted me to host a meetup
together in Beijing on 4 Jun 2020 in Beijing (using Chinese).

Does someone want to share a talk about IoTDB? (How to use IoTDB, IoTDB's
design etc..)

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

[jira] [Created] (IOTDB-358) Questions about the `group by` down-samplying clause

[xiangdong Huang (Jira)]

xiangdong Huang created IOTDB-358:
-

 Summary: Questions about the `group by` down-samplying clause
 Key: IOTDB-358
 URL: https://issues.apache.org/jira/browse/IOTDB-358
 Project: Apache IoTDB
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: xiangdong Huang


When reading issue  #331, I read the `group by` document again and find some 
confusing contents:

V0.9:

[http://iotdb.apache.org/#/Documents/0.9.0/chap5/sec2]
 # (minor), typo, Fig 5.2 rather than Fig 3.2 
 # The first parameter of the GROUP BY statement above is the time interval for 
dividing the time axis. Taking this parameter (1d) as time interval and the 
default origin as the dividing origin, the time axis is divided into several 
continuous intervals, which are [0,1d], [1d, 2d], [2d, 3d], etc.

        My question is, is the data point at 1d calculated twice? 

        

V0.10:
[http://iotdb.apache.org/#/Documents/progress/chap5/sec2]

the syntax is quite different with v0.9.0. I originally thought it was an 
improvement comparing with 0.9, but when I read the document, I find it loses 
the ability to define the start point of the timestamp for splitting the time 
window.

Can someone tell me why to design like this?

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [jira] [Created] (IOTDB-335) Separate query execution of the same timeseries with different aggregate functions

[Xiangdong Huang]

+1 for the idea.
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2019年12月5日周四 下午6:42写道：

> Hi,
>
> This is a good optimization :)
>
> Thanks,
> Jialin Qiao
>
> Lei Rui (Jira)  于2019年12月3日周二 下午2:20写道：
>
> > Lei Rui created IOTDB-335:
> > -
> >
> >  Summary: Separate query execution of the same timeseries
> with
> > different aggregate functions
> >  Key: IOTDB-335
> >  URL: https://issues.apache.org/jira/browse/IOTDB-335
> >  Project: Apache IoTDB
> >   Issue Type: Improvement
> > Reporter: Lei Rui
> >
> >
> > For example,
> >
> > select avg(s1),sum(s1) from root.sg.d0
> >
> > select first_value(s1),first_time(s1) from root.sg.d0
> >
> > select count(s1), sum(s1) from root.sg.d0 group by ([1,100],2ms)
> >
> >
> >
> > Probably the results of the same timeseries with different aggregate
> > functions can be achieved in one query execution process.
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
>
>
> --
> —
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>