[Proposal] Add the feature of importing and exporting metadata by CSV file

[李蓉]

Hello Everyone,
 
I am Rong Li. I am excited to share with you a feature that I have been working 
on for the past few months.

As we all know, IoTDB supports various forms of importing and exporting data, 
but does not support importing and exporting metadata. In practical use, there 
are also many scenarios that require importing and exporting metadata, so I 
added a metadata import and export script to achieve this function.
 
I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.
 
Thank you for your attention and support.

Best regards,

Rong Li

Reference:
https://github.com/apache/iotdb/pull/12623

Re:Re: Treat predicate whose time series doesn't exist as false

[李伟豪]

Hi,
We are developping this feature now, we introduce the Null ConstantOperand to 
denote timeseries that do not exist in Where/Having clasuse. 
For example:
select * from root.sg.d1 where s1 > 1 (root.sg.d1.s1 is not existed) 
After analyze, the where predicate will be refactor as : NULL > 1, and the 
final result of this predicate will always be NULL, no rows can be selected 
(only true the row will be selected), this query returns empty block.
 

Additionally, we tidy up the null-related operation rules, refers to MySQL and 
SQL Server

1. Arithmetic:+, -, *, /, %
The resulst will be NULL if any Operand is NULL.
2.Compare: =, >, <, like, in, between...(don't include `is null`)
The resulst will be NULL if any Operand is NULL.
3.Logic: not, and, or
In the following table, `？` (Undefined) stands for NULL
If the table are not transmit, you can see 
https://learn.microsoft.com/zh-cn/sql/connect/ado-net/sql/handle-null-values?view=sql-server-ver16
|A|B|A ∨ B|A ∧ B|¬ A|
|T|T|T|T|F|
|T|?|T|?|F|
|T|F|T|F|F|
|?|T|T|?|?|
|?|?|?|?|?|
|?|F|?|F|?|
|F|T|T|F|T|
|F|?|?|F|T|
|F|F|F|F|T|
在 2022-10-09 09:49:58，"Jialin Qiao"  写道：
>+1, return empty is better than throwing an exception.
>—
>Jialin Qiao
>Apache IoTDB PMC
>
>冯 庆新  于2022年10月8日周六 21:25写道：
>>
>> Hi,
>> I support you in this improvement.
>>
>> 获取 Outlook for iOS
>> 
>> 发件人: Zhou Yifu 
>> 发送时间: Friday, October 7, 2022 3:48:43 PM
>> 收件人: dev@iotdb.apache.org 
>> 主题: 回复: Treat predicate whose time series doesn't exist as false
>>
>> Yes, I agree with you. Query should not return an exception except for a 
>> syntax error.
>>
>> Thanks,
>> Yifu Zhou
>>
>> 从 Windows 版邮件发送
>>
>> 发件人: Yuan Tian
>> 发送时间: 2022年10月7日 14:56
>> 收件人: dev
>> 主题: Treat predicate whose time series doesn't exist as false
>>
>> Hi,
>>
>> Currently in IoTDB, if users write a non-exist time series in
>> select clause, we will simply ignore that and don't display it in
>> result columns instead of throwing exceptions. However, if users write
>> a non-exist time series in where clause, we will throw an exception
>> like: `the path 'root.test.g_0.d1.s2' in WHERE clause does not exist`.
>> It's inconsistent with the select clause, so we prepare to just treat
>> predicate whose time series doesn't exist as false instead of throwing
>> exception, same as InfluxDB.
>> How do you guys think?
>> The JIRA is https://issues.apache.org/jira/browse/IOTDB-4576
>>
>>
>> Best,
>> --
>> Yuan Tian
>>

Re:Re: Treat predicate whose time series doesn't exist as false

[李伟豪]

Hi,
We are developping this feature now, we introduce the Null ConstantOperand to 
denote timeseries that do not exist in Where/Having clasuse. 
For example:
select * from root.sg.d1 where s1 > 1 (root.sg.d1.s1 is not existed) 
After analyze, the where predicate will be refactor as : NULL > 1, and the 
final result of this predicate will always be NULL, no rows can be selected 
(only true the row will be selected), this query returns empty block.
 

Additionally, we tidy up the null-related operation rules, refers to MySQL and 
SQL Server

1. Arithmetic:+, -, *, /, %
The resulst will be NULL if any Operand is NULL.
2.Compare: =, >, <, like, in, between...(don't include `is null`)
The resulst will be NULL if any Operand is NULL.
3.Logic: not, and, or
In the following figure, `U` (Undefined) stands for NULL

在 2022-10-09 09:49:58，"Jialin Qiao"  写道：
>+1, return empty is better than throwing an exception.
>—
>Jialin Qiao
>Apache IoTDB PMC
>
>冯 庆新  于2022年10月8日周六 21:25写道：
>>
>> Hi,
>> I support you in this improvement.
>>
>> 获取 Outlook for iOS
>> 
>> 发件人: Zhou Yifu 
>> 发送时间: Friday, October 7, 2022 3:48:43 PM
>> 收件人: dev@iotdb.apache.org 
>> 主题: 回复: Treat predicate whose time series doesn't exist as false
>>
>> Yes, I agree with you. Query should not return an exception except for a 
>> syntax error.
>>
>> Thanks,
>> Yifu Zhou
>>
>> 从 Windows 版邮件发送
>>
>> 发件人: Yuan Tian
>> 发送时间: 2022年10月7日 14:56
>> 收件人: dev
>> 主题: Treat predicate whose time series doesn't exist as false
>>
>> Hi,
>>
>> Currently in IoTDB, if users write a non-exist time series in
>> select clause, we will simply ignore that and don't display it in
>> result columns instead of throwing exceptions. However, if users write
>> a non-exist time series in where clause, we will throw an exception
>> like: `the path 'root.test.g_0.d1.s2' in WHERE clause does not exist`.
>> It's inconsistent with the select clause, so we prepare to just treat
>> predicate whose time series doesn't exist as false instead of throwing
>> exception, same as InfluxDB.
>> How do you guys think?
>> The JIRA is https://issues.apache.org/jira/browse/IOTDB-4576
>>
>>
>> Best,
>> --
>> Yuan Tian
>>

答复: Release about 1.0.0

[李思佳]

That's really a good news !

It would be great if we could release deployment recommendations (For example, 
which scenarios are appropriate for the cluster version, how many nodes can 
manage how much metadata and data, how to plan the cluster size by estimating 
the amount of users' data, etc) and performance comparison reports (compared to 
the 0.x version) together at launch.

BR,
---
Sijia Li

-邮件原件-
发件人: Jialin Qiao  
发送时间: 2022年10月11日 14:36
收件人: dev@iotdb.apache.org
主题: Release about 1.0.0

Hi all,

After nearly 4 years since we join the ASF in 2018 and one year development of 
the Cluster version, we released several versions, from
0.8 to 0.14.

Now, we prepare to release a formal cluster version in the end of next month, 
and I propose that we call this version 1.0.0.

Thank all for contributing to the cluster version, what do you think?

Thanks,
—
Jialin Qiao
Apache IoTDB PMC

apply to join iotdb

[李龙]

Hi，i am lilong from yonyou special database technology R & D department
my jira id：jeck1s...@163.com
my confluence id：jeck1s...@163.com

Permission request

[李蓉]

Hi,I'm rongli,from Shaanxi Province.I have a wide range of interests and 
hope to get along with everyone in harmony.It's a pleasure to meet you all，and 
I hope to learn from each other in the future.
my jira id is rongli and my Confluence id : RongLi .

答复: New function: show region

[李思佳]

That sounds like a nice feature.
By the way, if my region is migrating or not ready, can I see its status?
As you said, can I see which regions are moving when my datanode has changed or 
in rebalanced?  For example, Region a is moving from node A to node B?

BR,
---
Sijia Li

-邮件原件-
发件人: Jialin Qiao  
发送时间: 2022年6月14日 17:31
收件人: dev@iotdb.apache.org
主题: New function: show region

Hi,

I created an issue to query the region distribution [1].

This could be used in some scenarios:
(1) To see if the region migration is successful after removing a datanode
(2) To see if the region is well-distributed in datanodes

The query result could contain 5 columns:

RegionID, RegionType, DataNodeID, RPCAddress, RPCPort

Any suggestions?

[1] https://issues.apache.org/jira/browse/IOTDB-3488

Thanks,
—
Jialin Qiao
Apache IoTDB PMC

答复: Re: Re: Flush function in cluster

[李思佳]

In fact, this is because we cannot compare tsFiles to determine whether the 
replica data is consistent.

If the user flush ensures that all copies are flushed, then the next restart, 
we only need to check whether the operation after this flush is consistent and 
update it.

Otherwise, when the follower is much behind the leader and we need to catch up 
via tsfile, is there a copy of the all data files?

BR,
---
Sijia Li


-邮件原件-
发件人: Xiangdong Huang  
发送时间: 2022年5月23日 11:52
收件人: dev 
主题: Re: Re: Flush function in cluster

> " flush can reduce memory and speed up the restart process" , this
assumes that all copies have been flushed synchronously, so we can ensure that 
the data files are logically consistent at this point.

Sorry that maybe I lag behind current cluster design..
Do we need "all copies have been flushed synchronously, so we can ensure that 
the data files are logically consistent at this point" ? why? because of the 
raft protocol?


---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


李思佳  于2022年5月23日周一 11:47写道：

> " flush can reduce memory and speed up the restart process" , this 
> assumes that all copies have been flushed synchronously, so we can 
> ensure that the data files are logically consistent at this point.
>
> The operation of datanode flushing should be the process of resource 
> release before the node is shutdown(but this does not guarantee that 
> all copies are logically consistent at this point). For example, 
> shutdownHook requires the default disk flushing and resource release. 
> We need to provide a flush command scenario, perhaps because our node 
> shutdown operation is not incomplete?
>
> BR,
> ---
> Sijia Li
>
>
> -邮件原件-
> 发件人: Xiangdong Huang 
> 发送时间: 2022年5月23日 11:37
> 收件人: dev 
> 主题: Re: Flush function in cluster
>
> I think distinguishing flushing on one node or on the cluster has its 
> meaning.
>
> As you said, flush can reduce memory and speed up the restart process. 
> So, how about if the DBA just wants to restart one node..
>
> However, the default behavior can be discussed: flush on one node by 
> default or on the whole cluster by default.
>
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> 李思佳  于2022年5月23日周一 11:28写道：
>
> > Sorry, I don't understand what the purpose and use of flushing 
> > current datanode is.
> >
> > IMO, flush all should mean that all storage group could be flushed, 
> > in another word, flush sg is a subset of flush all.
> >
> > For users, distributed is a black box, while SG is an exposed structure.
> > Therefore, for cli commands, there is no need to be aware of the 
> > relationship between the datanode and the self-created SG.
> >
> > In addition, the Flush operation may speed up our restart recovery 
> > process. For example, when we flush an SG successfully, we can label 
> > the associated data files to indicate that all copies are consistent 
> > at that point in time(here are flush and write priorities). During 
> > the next restart, we can use this flag to quickly skip the verification 
> > step.
> >
> > In summary, here are my questions and thoughts:
> > 1. Is it necessary to flush a dataNode? What are the benefits of this?
> > 2. Can the Flush operation affect the consensus group or WAL for a 
> > quick restart?
> >
> > BR,
> > ---
> > Sijia Li
> >
> >
> > -邮件原件-
> > 发件人: Jialin Qiao 
> > 发送时间: 2022年5月23日 11:07
> > 收件人: dev@iotdb.apache.org
> > 主题: Flush function in cluster
> >
> > Hi,
> >
> > Flush is a frequently used command in IoTDB, which flushes memtable 
> > into disk and closes all tsfiles.
> >
> > In the new cluster, we need to redefine this function [1].
> >
> > * flush: flushing current datanode
> >
> > * flush all/cluster: flushing all datanodes
> >
> > * flush sg: flush all DataRegions of a storage group
> >
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IOTDB-3099
> >
> > —
> > Jialin Qiao
> > Apache IoTDB PMC
> >
>

答复: Re: Flush function in cluster

[李思佳]

" flush can reduce memory and speed up the restart process" , this assumes that 
all copies have been flushed synchronously, so we can ensure that the data 
files are logically consistent at this point.  

The operation of datanode flushing should be the process of resource release 
before the node is shutdown(but this does not guarantee that all copies are 
logically consistent at this point). For example, shutdownHook requires the 
default disk flushing and resource release. We need to provide a flush command 
scenario, perhaps because our node shutdown operation is not incomplete?  

BR,
---
Sijia Li


-邮件原件-
发件人: Xiangdong Huang  
发送时间: 2022年5月23日 11:37
收件人: dev 
主题: Re: Flush function in cluster

I think distinguishing flushing on one node or on the cluster has its meaning.

As you said, flush can reduce memory and speed up the restart process. So, how 
about if the DBA just wants to restart one node..

However, the default behavior can be discussed: flush on one node by default or 
on the whole cluster by default.

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


李思佳  于2022年5月23日周一 11:28写道：

> Sorry, I don't understand what the purpose and use of flushing current 
> datanode is.
>
> IMO, flush all should mean that all storage group could be flushed, in 
> another word, flush sg is a subset of flush all.
>
> For users, distributed is a black box, while SG is an exposed structure.
> Therefore, for cli commands, there is no need to be aware of the 
> relationship between the datanode and the self-created SG.
>
> In addition, the Flush operation may speed up our restart recovery 
> process. For example, when we flush an SG successfully, we can label 
> the associated data files to indicate that all copies are consistent 
> at that point in time(here are flush and write priorities). During the 
> next restart, we can use this flag to quickly skip the verification step.
>
> In summary, here are my questions and thoughts:
> 1. Is it necessary to flush a dataNode? What are the benefits of this?
> 2. Can the Flush operation affect the consensus group or WAL for a 
> quick restart?
>
> BR,
> ---
> Sijia Li
>
>
> -邮件原件-
> 发件人: Jialin Qiao 
> 发送时间: 2022年5月23日 11:07
> 收件人: dev@iotdb.apache.org
> 主题: Flush function in cluster
>
> Hi,
>
> Flush is a frequently used command in IoTDB, which flushes memtable 
> into disk and closes all tsfiles.
>
> In the new cluster, we need to redefine this function [1].
>
> * flush: flushing current datanode
>
> * flush all/cluster: flushing all datanodes
>
> * flush sg: flush all DataRegions of a storage group
>
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/IOTDB-3099
>
> —
> Jialin Qiao
> Apache IoTDB PMC
>

答复: Flush function in cluster

[李思佳]

In addition, if flush current datanode operation is not supported, can our 
command use the previous rules, such as flush and flush SG  

BR,
---
Sijia Li


-邮件原件-
发件人: 李思佳  
发送时间: 2022年5月23日 11:28
收件人: dev@iotdb.apache.org
主题: 答复: Flush function in cluster

Sorry, I don't understand what the purpose and use of flushing current datanode 
is.

IMO, flush all should mean that all storage group could be flushed, in another 
word, flush sg is a subset of flush all.

For users, distributed is a black box, while SG is an exposed structure. 
Therefore, for cli commands, there is no need to be aware of the relationship 
between the datanode and the self-created SG.  

In addition, the Flush operation may speed up our restart recovery process. For 
example, when we flush an SG successfully, we can label the associated data 
files to indicate that all copies are consistent at that point in time(here are 
flush and write priorities). During the next restart, we can use this flag to 
quickly skip the verification step.

In summary, here are my questions and thoughts:
1. Is it necessary to flush a dataNode? What are the benefits of this?  
2. Can the Flush operation affect the consensus group or WAL for a quick 
restart?  

BR,
---
Sijia Li


-邮件原件-
发件人: Jialin Qiao  
发送时间: 2022年5月23日 11:07
收件人: dev@iotdb.apache.org
主题: Flush function in cluster

Hi,

Flush is a frequently used command in IoTDB, which flushes memtable into disk 
and closes all tsfiles.

In the new cluster, we need to redefine this function [1].

* flush: flushing current datanode

* flush all/cluster: flushing all datanodes

* flush sg: flush all DataRegions of a storage group


What do you think?

[1] https://issues.apache.org/jira/browse/IOTDB-3099

—
Jialin Qiao
Apache IoTDB PMC

答复: Flush function in cluster

[李思佳]

Sorry, I don't understand what the purpose and use of flushing current datanode 
is.

IMO, flush all should mean that all storage group could be flushed, in another 
word, flush sg is a subset of flush all.

For users, distributed is a black box, while SG is an exposed structure. 
Therefore, for cli commands, there is no need to be aware of the relationship 
between the datanode and the self-created SG.  

In addition, the Flush operation may speed up our restart recovery process. For 
example, when we flush an SG successfully, we can label the associated data 
files to indicate that all copies are consistent at that point in time(here are 
flush and write priorities). During the next restart, we can use this flag to 
quickly skip the verification step.

In summary, here are my questions and thoughts:
1. Is it necessary to flush a dataNode? What are the benefits of this?  
2. Can the Flush operation affect the consensus group or WAL for a quick 
restart?  

BR,
---
Sijia Li


-邮件原件-
发件人: Jialin Qiao  
发送时间: 2022年5月23日 11:07
收件人: dev@iotdb.apache.org
主题: Flush function in cluster

Hi,

Flush is a frequently used command in IoTDB, which flushes memtable into disk 
and closes all tsfiles.

In the new cluster, we need to redefine this function [1].

* flush: flushing current datanode

* flush all/cluster: flushing all datanodes

* flush sg: flush all DataRegions of a storage group


What do you think?

[1] https://issues.apache.org/jira/browse/IOTDB-3099

—
Jialin Qiao
Apache IoTDB PMC

Re: About replacing byteBuffer

[李思佳]

In fact, we can do that. There's no copy if bytebuf passed to the 
compositeByteBuf are wrapped by ByteBuffer.

The problem is that we might not force every module to do this(may be checked 
only by review code). If anyone uses Netty's byteBuf directly and does not 
manage memory properly(and we don't have good surveillance to find this 
problem), this could result in a memory leak.  

Maybe, how to make better use of Netty, provide more monitoring capabilities, 
and constrain the development standards of all modules, this may need further 
consideration.  

BR,
---
Sijia Li

-邮件原件-
发件人: Xiangdong Huang  
发送时间: 2022年5月17日 8:57
收件人: dev 
主题:Re: About replacing byteBuffer

If we introduce Netty, data copy when scaling a bytebuf is not what we want.
Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit of 
pooledByteBuf?

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2022年5月16日周一 12:35写道：

> Hi,
>
> The serialization interface needs to be refactored afterward.
>
> Before that, using ByteArrayOutputStream is easier.
>
> Thanks,
> —
> Jialin Qiao
> Apache IoTDB PMC
>
>
> 李思佳  于2022年5月16日周一 11:44写道：
>
> > Hi all,
> >
> > When I was developing the snapshot interface for the configNode 
> > module, I noticed that the parameters received by the serialization 
> > interface were all defined as ByteBuffer, which seemed to have some 
> > problems. For
> example,
> > the external main process has no way of knowing how big the buffer 
> > will
> be.
> > We can only estimate a large value to allocate memory.
> >
> >  Then I looked at the serialization interfaces of other modules, and 
> > it seemed that most modules did the same thing. This could be a 
> > problem once the actual size of the buffer exceeds our estimate. So 
> > I did a quick
> survey
> > of Netty's byteBuf last week, and here's the Chinese version of the
> results<
> > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>.
> >
> >   At the same time, we found that the consensus module also has some 
> > ByteBuf requirements. But byteBuf doesn't seem to be enough to give 
> > us precise control over the size of the memory pool, and we may need 
> > to wrap it if we decide to use it.
> >
> >   Finally, we decided to use stream type instead of byteBuffer in 
> > configNode for the time being. I will start this work to see if this 
> > is
> the
> > better way this week. If any idea, please let me know.
> >
> >   By the way, Netty’s ByteBuf provides powerful tool operations that 
> > we will not discard outright, but rather as an option.
> >
> > BR,
> > ---
> > Sijia Li
> >
> >
>

About replacing byteBuffer

[李思佳]

Hi all,

When I was developing the snapshot interface for the configNode module, I 
noticed that the parameters received by the serialization interface were all 
defined as ByteBuffer, which seemed to have some problems. For example, the 
external main process has no way of knowing how big the buffer will be. We can 
only estimate a large value to allocate memory.

 Then I looked at the serialization interfaces of other modules, and it seemed 
that most modules did the same thing. This could be a problem once the actual 
size of the buffer exceeds our estimate. So I did a quick survey of Netty's 
byteBuf last week, and here's the Chinese version of the 
results.

  At the same time, we found that the consensus module also has some ByteBuf 
requirements. But byteBuf doesn't seem to be enough to give us precise control 
over the size of the memory pool, and we may need to wrap it if we decide to 
use it.

  Finally, we decided to use stream type instead of byteBuffer in configNode 
for the time being. I will start this work to see if this is the better way 
this week. If any idea, please let me know.

  By the way, Netty’s ByteBuf provides powerful tool operations that we will 
not discard outright, but rather as an option.

BR,
---
Sijia Li

账户权限申请

[李**]

self-introduction：I'm WeiHao Li, a development engineer of IoTDB team of 
Timecho.


Jira ID：Weihao Li


Confluence ID：Weihao Li

答复: Re: Rocksdb-based metadata storage

[李思佳]

Thank you for your reminding.  

Currently we are working on code issues, such as the unit tests involved and 
the questions raised by the code reviewer.

This feature is currently only available to community developers. When the 
issues about code are resolved and merged into the master branch, we will start 
to work on performance testing about this feature(maybe we also need some work 
on tuning the configuration for optimum performance), and then we will add 
operating instructions and performance test reports.

If possible, we also welcome colleagues from the test team to test together. In 
the testing process, different purposes and angles will make our products 
better and stronger.

BR,
---
Sijia Li

-邮件原件-
发件人: Xiangdong Huang  
发送时间: 2022年3月23日 20:58
收件人: dev 
主题: Re: Rocksdb-based metadata storage

Hi Sijia,

Nice work.

But better to add the user guide along with the codes (tell users how to use 
and when to use).

It is also welcome to share some of your use practice/experiments here or on 
Github PR (in English) in either some slides or some pictures.

Best

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


李思佳  于2022年3月21日周一 11:45写道：

> Hi all,
>
> We have submitted a formal PR based on the RocksDB metadata storage scheme.
>
> This solution is used to solve the problem of too many time series, 
> resulting in insufficient memory, frequent full GC, and service 
> unavailability.
>
> Parts of the interface are still a work in progress, and we'd love to 
> see others work with us to improve it. If you have any time, you are 
> welcome to test and supplement : )
>
> Here's the link: https://github.com/apache/iotdb/pull/5295
>
> BR,
> ---
> Sijia Li
>
>

Rocksdb-based metadata storage

[李思佳]

Hi all,

We have submitted a formal PR based on the RocksDB metadata storage scheme.

This solution is used to solve the problem of too many time series, resulting 
in insufficient memory, frequent full GC, and service unavailability.

Parts of the interface are still a work in progress, and we'd love to see 
others work with us to improve it. If you have any time, you are welcome to 
test and supplement : )

Here's the link: https://github.com/apache/iotdb/pull/5295

BR,
---
Sijia Li

CatchupTask may cause a cluster crash?

[李思佳]

Hi all,

After a node in the cluster went offline and then rejoined the cluster, I found 
that my cluster was in a crash state.
Therefore, I did some check and analysis, there are some problems in the 
SnapshotCatcup module.

Relevant descriptions are recorded here: 
https://github.com/apache/iotdb/issues/4352

If you have encountered the same problem, or are familiar with this module, 
feel free to discuss. Thanks.

BR,
---
Sijia Li

答复: Re: Enable wiki in Github

[李思佳]

Hi，+1

In addition, can we add the version number or release information in our 
documents?  
Sometimes it's hard to figure out which version of the code a design document 
in wiki corresponds to.
(the document may be too new, maybe it's stale)


Thanks

Sijia Li


On 09/23/2021 09:47，Chao Wang wrote：
+1




Thanks!


Chao Wang
BONC ltd
ccgow...@163.com
On 9/23/2021 09:46，Haiming Zhu wrote：
Hi, +1 :)

Best,

Haiming Zhu
School of Software, Tsinghua University

朱海铭
清华大学 软件学院


钟文豪  于2021年9月23日周四 上午9:45写道：

Hi,
+1
Thanks.zhongwenhao





At 2021-09-21 21:26:21, "Xiangdong Huang"  wrote:
Hi,

Using Github Wiki can indeed make it easy to update documents.

If we move the wiki from confluence to Github wiki, we still need to sync the 
wiki documents in some place of ASF.

As Github wiki is also a git codebase
(https://github.com/YOUR_USERNAME/YOUR_REPOSITORY.wiki.git),
if we enable it for iotdb (i.e., github.com/apache/iotdb.wiki.git),
will ASF have a corresponding git codebase automatically?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

黄向东
清华大学 软件学院

Willem Jiang  于2021年9月20日周一 上午10:12写道：

Here is one concern we need to address.
Although we are using Github as our daily development platform,  ASF is still 
syncing the data to ASF infrastructure through email forwarding or git sync.
In this way we still have a chance to commit the code through ASF infra.
If we move the wiki from confluence to Github wiki, we still need to sync the 
wiki documents in some place of ASF.

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Sun, Sep 19, 2021 at 4:55 PM Jialin Qiao 
wrote:

Hi,

We have tried a lot about how to manage our documents (UserGuide, SystemDesign, 
ContributeGuide...).

First, we manage these docs as many .md files in our repo, however, update it 
is troublesome, need to submit a pr and the website update is not in time.

Next, we try the confluence, However, the editor of confluence is not friendly. 
md is good.

Therefore, we try to use Github issues to manage these things. E.g., join the 
community https://github.com/apache/iotdb/issues/1995.

Finally, we realize that GitHub wiki may be the best place to manage such 
documents,  it is the nearest to our code, users do not need to go to another 
place (all in Github). As for the website, we could read the .md files from 
GitHub wiki.

What do you think?

If everyone recognized it, I will submit an issue to the INFRA.

Thanks,
—
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together

[李思佳]

Hi,

This is Sijia Li, I am a big data engineer of Qihoo 360.

I'd like to participate in this work with my colleague @Jianyun Cheng.

I hope that we can have a pleasant cooperation.

Thanks

-邮件原件-
发件人: Xiangdong Huang  
发送时间: 2021年8月27日 14:33
收件人: dev 
主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch 
together

Hi,

I have worked on "cluster-" branch for several days, but still not finished.
As I have no much time to finish it, I'd like to ask cooperators working on it 
together.
Please let me know if you want to join.

"cluster-" branch is not for fixing any bug in the cluster module, but just for 
refactoring the codes to make the structure similar to the server module.

Some ideas in the branch:

1. split Thrift RPC service and RPC implementation.
In the server module, we have extracted a class called ThriftService, which is 
responsible for starting a RPC service. We just need to ingest a RPC 
implementation to it.
However, in the cluster module, some RPC classes mixed them, which causes code 
duplication, code inconsistencies, and hard to understand.

2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for 
serving meta raft group, which should not control the whole server too deep. 
So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed 
from ClusterMain), and ClusterIoTDB is responsible for creating them. 
MetaGroupMember only can modify these fields, rather than creating them.

That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than 
MetaGroupMember.

BTW, MetaGroupMember will be renamed to MetaGroupEngine.

3. Similar with the relationship between StorageEngine and StorageProcessor in 
the server module, DataGroupMember can be considered as StorageProcessor, and 
we need a DataGroupEngine to control them.

4. I am considering how to refactor the ClientPool. there are too many in the 
codes. But I have no clear idea now.

The 2nd and 3rd refactors will lead to many unexpected issues as I am not also 
very familiar with the codes. So, I have to check all UT and ITs one by one.

I think after the refactoring, developers who know the server module will be 
easy to understand the codes in the cluster module.

And, once the ClusterIoTDB dominates the module, it will be clear to modify the 
startup process like using ID to replace Node.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Re: ready for release v0.12.2?

[李萌]

+ 1

On 2021/08/25 02:38:49, Xiangdong Huang  wrote: 
> Hi,
> 
> So I will start a vote. the version is compiled 1 hour ago.
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院
> 
> Dawei Liu  于2021年8月25日周三 上午10:37写道：
> >
> > +1
> >
> >
> > Incidentally, the preview version of 0.12.2 has been running steadily in 
> > our production environment for one month. According to rough statistics, 
> > 800G of raw data is now processed every day, supporting a sequence of 26M+, 
> > and the gain on the query is also very obvious!
> >
> >
> > Dawei Liu
> >
> >  回复的原邮件 
> > | 发件人 | CloudWise-Luke<282583...@qq.com.invalid> |
> > | 发送日期 | 2021年08月25日 09:39 |
> > | 收件人 | dev  |
> > | 主题 | 回复： ready for release v0.12.2? |
> > +1
> >
> >
> >
> > CloudWiseluke.miao
> >
> >
> >  
> >
> >
> >
> >
> > -- 原始邮件 --
> > 发件人:
> > "dev"   
> >  
> > ;
> > 发送时间: 2021年8月25日(星期三) 上午9:26
> > 收件人: "dev";
> >
> > 主题: Re: ready for release v0.12.2?
> >
> >
> >
> > Hi,
> >
> > +1.
> >
> > We have done a lot of work on the stability around the rel/0.12. Looking 
> > forward to the new release!
> >
> > Steve Su
> > Tsinghua University
> >
> > -- Original --
> > From: "dev" ;
> > Date: Tue, Aug 24, 2021 08:22 PM
> > To: "dev";
> > Subject: ready for release v0.12.2?
> >
> > Hi all,
> >
> > I see all PRs for rel/0.12 branch are merged (except pr 3759 while the
> > pr has some argumentation).
> >
> > So, Is it time to release v0.12.2? If so, I can build  RC1 of v0.12.2.
> >
> > Best,
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> 

Re: About enabling Github Discussion

[李萌]

Good idea, but we need to see if it meets the apache regulations first  I think.

On 2021/08/07 02:43:35, Jialin Qiao  wrote: 
> Hi,
> 
> Github Discussion is a forum for the community to collaborate around open
> source projects [1]. You might consider it as a replacement of
> us...@iotdb.apache.org.
> 
> I noticed Apache Couchdb has been already using it [2]
> 
> I'd like to apply a Github discussion for the IoTDB community, it could be
> a replacement for slack, WeChat group, QQ group...
> 
> what do you think?
> 
> [1]
> https://github.blog/2020-05-06-new-from-satellite-2020-github-codespaces-github-discussions-securing-code-in-private-repositories-and-more/
> [2] https://github.com/apache/couchdb/discussions
> 
> Thanks,
> —
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 

Re: Shuanglin Wu's self-introduction

[李萌]

On 2021/08/03 14:42:20, "wushuanglin"  wrote: 
> Hi,
> This is Shuanglin Wu,
> I am a big data programmer. I am very passionate about iotdb. Hope I can 
> contribute to iotdb in the community.
> Best regards,
> Shuanglin Wu
Hi Shuanglin, welcome to the family!

Re: Shuanglin Wu's self-introduction

[李萌]

On 2021/08/03 14:42:20, "wushuanglin"  wrote: 
> Hi,
> This is Shuanglin Wu,
> I am a big data programmer. I am very passionate about iotdb. Hope I can 
> contribute to iotdb in the community.
> Best regards,
> Shuanglin Wu
Hi Shuanglin, 

Welcome to join the family.

Re:[VOTE] Start the graduation process

[李天安]

Hi,

+1 from PPMC

Thanks,


Best Regards,
—
Tianan Li
School of Software, Tsinghua University


李天安
清华大学 软件学院


On 08/30/2020 20:46，Julian Feinauer wrote：
Hi folks,

as we just had a very positive discussion about the maturity of our community 
[1] I want to formally start the vote if we want to start the graduation 
process.

For Information about our maturity assessment please see [2].

Do you agree that the IoTDB project is ready for graduation and that we should 
start the graduation process now?

As usual, this vote will be open for at least 72hrs.

Best
Julian



[1] 
https://lists.apache.org/thread.html/r7607cbf6ff2020dc7e7cc27eb698ce265b768ed2a63d3e353c2dc572%40%3Cdev.iotdb.apache.org%3E
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148645763

Re: [VOTE] Apache IoTDB 0.10.1 (incubating) RC3 release

[李天安]

Hi,
My vote is +1 (PPMC)

I have checked following items:

Source release:
1. Running RAT(mvn apache-rat:check) (Correct)
2. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
extracted source bundle (Correct)
3. Verifying the signature (ASC) (Correct)
4. Verifying the hashes (SHA512) (Correct)

Binary distribution:
1. Verifying the signature (ASC) (Correct)
2. Verifying the hashes (SHA512) (Correct)
3. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files 
(Correct)

Functionality：
1. Run server and client script:  sh start-server.sh, sh start-cli.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
2. REBOOT
3. Execute some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 6
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2020-08-08 12:34:21 (星期六)
> 收件人: dev 
> 抄送: 
> 主题: [VOTE] Apache IoTDB 0.10.1 (incubating) RC3 release
> 
> Hi all,
> 
> Though RC2 gets 12 +1 votes, Jialin found a bug in RC2, which is easily
> reproduced in some scenarios. Therefore, I'd like to call a new vote for
> RC3.
> 
> 
> This is a vote for releasing v0.10.1, which is a bug-fix version of v0.10.0.
> 
> You can get its main changes from [5].
> 
> Comparing with RC1, RC3 (includes RC2) fixes the following issue:
> 
> - add README_ZH.md file into release.
> - upgrade orc-core dependency for license compatibility
> - remove org.json dependency for license compatibility
> - fix a NULL Pointer bug when querying data in some cases
> - fix a bug when users use an incorrect encoding method to define a time
> series.
> 
> Apache IoTDB (Incubating)  0.10.1 has been staged under [2] and it’s time
> to vote
> on accepting it for release.  All Maven artifacts are available under [1].
> If approved we will seek final release approval from the IPMC.
> 
> Voting will be open for 72hr.
> 
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
> 
> tag: release/0.10.1
> Hash for the release tag: 023fb84e0b76c0ac64b7a9ac098c640f26795a25
> 
> According to [3], "Before voting +1 [P]PMC members are required to download
> the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also
> verifying that the package meets the requirements of the ASF policy
> on releases."
> 
> You can achieve the above by following [4].
> 
> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM
> items in [4])
> [ ]  -1 reject (explanation required)
> 
> 
> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1044
> [2] https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.1/rc3
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4]
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> [5]
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.1/rc3/RELEASE_NOTES.md
> [6] https://dist.apache.org/repos/dist/dev/incubator/iotdb/KEYS
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院

Re: [VOTE] Apache IoTDB 0.10.0 (incubating) RC4 release

[李天安]

Hi,
My vote is +1 (PPMC)

I have checked following items:

Source release:
1. Running RAT(mvn apache-rat:check) (Correct)
2. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
extracted source bundle (Correct)
3. Compile and build code with command: mvn clean package. (Correct)
4. Verifying the signature (ASC) (Correct)
5. Verifying the hashes (SHA512) (Correct)

Binary distribution:
1. Verifying the signature (ASC) (Correct)
2. Verifying the hashes (SHA512) (Correct)
3. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files 
(Correct)

Functionality：
1. Run server and client script:  sh start-server.sh, sh start-cli.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
2. REBOOT
3. Execute some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 6
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2020-06-17 20:04:38 (星期三)
> 收件人: dev 
> 抄送: 
> 主题: [VOTE] Apache IoTDB 0.10.0 (incubating) RC4 release
> 
> Hi all,
> 
> 
> Welcome  (well, once again and again) to verify the release of Apache IoTDB
> (incubating) 0.10.0 (RC4).
> 
> 
> Apache IoTDB (Incubating) 0.10.0 has been staged under [1] and it’s time to
> vote
> on accepting it for release.  All Maven artifacts are available under [2].
> 
> 
> RC3 fixes the following issues of RC1, an internal bug in 0.10.0-SNAPSHOT
> and issues in RC2:
> 
> - incorrect version info in CLI.
> 
> - add executable permission on sbin/*, tools/* and conf/iotdb-env.sh files.
> 
> - fix some incorrect links in documents.
> 
> - fix the version of the site module.
> 
> 
> The release note can be get from [5]. The data file of v0.10.0 is
> incompatible with v0.9.x, but v0.10.0 can automatically upgrade the data
> file to the latest format.
> 
> If approved we will seek final release approval from the IPMC.
> 
> 
> Voting will be open for 72hr.
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
> 
> Release tag: release/0.10.0
> Hash for the release tag: f506d9c731231f97d8813528cbf062760d40
> 
> 
> Please follow the steps in [3]:
> 
> "Before voting +1 [P]PMC members are required to download
> the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also
> verifying that the package meets the requirements of the ASF policy
> on releases."
> 
> You can achieve the above by following [4].
> 
> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM
> items in [4])
> [ ]  -1 reject (explanation required)
> 
> 
> [1] https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.0/rc4/
> <https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.0/rc3/>
> [2] https://repository.apache.org/content/repositories/orgapacheiotdb-1038
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4]
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> [5]
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.0/rc4/RELEASE_NOTES.md
> <https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.0/rc3/RELEASE_NOTES.md>
> [6] https://dist.apache.org/repos/dist/dev/incubator/iotdb/KEYS
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院

Re: Re: Using Sync Tool with IoTDB Server

[李天安]

Hi,
 
In the sync tool, the receiver will compare the versions of IoTDB with the 
sender's and rejects the synchronization task if the versions are different.

In the load tool, the engine will check the version number of tsfile and will 
reject the load operation if the version doesn't match with the engine's.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2020-06-17 09:15:03 (星期三)
> 收件人: dev 
> 抄送: 
> 主题: Re: Using Sync Tool with IoTDB Server
> 
> Hi,
> 
> I have a question, is the load or sync function supports communication
> among different versions (e.g., load tsfile of 0.9 to 0.10.0)? If not, is
> there a check?
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院
> 
> 
> 李天安  于2020年6月17日周三 上午1:27写道：
> 
> > Hi Julian,
> >
> > In sync tool, before data synchronization, all schemas will be
> > synchronized first, so the schema of each tsfile transmitted is guaranteed
> > to exist.
> > Synchronous schema adopts incremental method.
> >
> >
> > Best Regards,
> > —
> > Tianan Li
> > School of Software, Tsinghua University
> >
> > 李天安
> > 清华大学 软件学院
> >
> > > -原始邮件-
> > > 发件人: "Julian Feinauer" 
> > > 发送时间: 2020-06-15 18:36:49 (星期一)
> > > 收件人: "dev@iotdb.apache.org" 
> > > 抄送:
> > > 主题: Using Sync Tool with IoTDB Server
> > >
> > > Hi folks,
> > >
> > > just a short question regarding the Sync Tool from IoTDB.
> > > If I have a plain tsfile that I sync via the sync tool to the server are
> > then all names from the timeseries picked automatically and entered in the
> > MTree and all?
> > > Or do I have to configure something upfront?
> > >
> > > Thanks!
> > > Julian
> >

Re: Using Sync Tool with IoTDB Server

[李天安]

Hi Julian,

In sync tool, before data synchronization, all schemas will be synchronized 
first, so the schema of each tsfile transmitted is guaranteed to exist. 
Synchronous schema adopts incremental method.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> -原始邮件-
> 发件人: "Julian Feinauer" 
> 发送时间: 2020-06-15 18:36:49 (星期一)
> 收件人: "dev@iotdb.apache.org" 
> 抄送: 
> 主题: Using Sync Tool with IoTDB Server
> 
> Hi folks,
> 
> just a short question regarding the Sync Tool from IoTDB.
> If I have a plain tsfile that I sync via the sync tool to the server are then 
> all names from the timeseries picked automatically and entered in the MTree 
> and all?
> Or do I have to configure something upfront?
> 
> Thanks!
> Julian

Re: [VOTE] Enable github issue

[李天安]

+1

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2020年1月3日 下午5:33，Jialin Qiao  写道：
> 
> Hi,
> 
> I'd like to call a vote for enabling github issue.
> 
> The github issues could be treated as user mail list because Chinese users
> prefer github issue more than Jira. However, for more convenient project
> management consideration, we also use Jira.
> 
> We could manage github and jira as follows:
> 
> 【Create GitHub issue】Users create github issues when they want to report a
> bug or new feature. Discussion is performed under github issues.
> 
> 【Create Jira issue】 If a github issue is valuable, we create a Jira issue
> for it and link the github issue url. After that, we add a [IOTDB-xxx]
> prefix in the title of github issue to mark it is already related to a Jira
> issue.
> 
> 【Create PR】 Each functional PR or bug-fix PR should relate to a Jira issue.
> PMCs or committers could help link PR with Jira issues. Some minor PRs
> could be merged without a link to Jira issues, such as fix a typo.
> 
> 【Close Jira Issue、Close github Issue、Merge PR】 Before mergine a PR, the
> contributor or the one merges the PR should mark the fixed-version in Jira
> issue and close the github issue if it is related.
> 
> The vote is open for the next 72 hours and passes if at least three +1
> votes and more +1 votes than -1 votes.
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 

Re: [jira] [Created] (IOTDB-375) Bug in parameter adapter

[李天安]

Hi ,

I have noticed this issue.

The incorrect evaluation is a bug in the function of deleting storage group, 
and deleting timeseries doesn’t have this issue.

While deleting sg, it just subtracts the num of sg in 
IoTDBConfigDynamicAdapter, but not subtracts the sum of timeseries in 
IoTDBConfigDynamicAdapter which leads to the incorrect evaluation. This is easy 
to fix.

What's more, removing debug logs while restarting the system doesn't have any 
side-effect.

I have pulled a pr to fix this issue. The link is 
https://github.com/apache/incubator-iotdb/pull/669 
<https://github.com/apache/incubator-iotdb/pull/669>.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年12月19日 下午2:43，Xiangdong Huang  <mailto:saint...@gmail.com>> 写道：
> 
> 1. Is there any side-effect if you remove it?
> 2. How to solve the incorrect evaluation when you delete some time series
> is still not mentioned.
> 
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
> 黄向东
> 清华大学 软件学院
> 
> 
> Jialin Qiao mailto:qiaojia...@apache.org>> 
> 于2019年12月19日周四 下午2:16写道：
> 
>> It is in the restart process, the check could be omitted.
>> 
> 

Re: [jira] [Created] (IOTDB-357) NullPointerException in ActiveTimeSeriesCounter

[李天安]

Hi,

I have found that it’s a hidden bug in method `redoLogs()` of class 
`TsFileRecoverPerformer`.

I have pull a request to fix that. The pr link is 
https://github.com/apache/incubator-iotdb/pull/642 
<https://github.com/apache/incubator-iotdb/pull/642>

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> 2019年12月10日 上午11:41，Jialin Qiao (Jira)  写道：
> 
> Jialin Qiao created IOTDB-357:
> -
> 
> Summary: NullPointerException in ActiveTimeSeriesCounter
> Key: IOTDB-357
> URL: https://issues.apache.org/jira/browse/IOTDB-357
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Jialin Qiao
> 
> 
> Start a empty IoTDB, execute 3 sql:
> {code:java}
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d2.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA;
> insert into root.turbine.d2(timestamp,s0) values(2,25.3);{code}
> Kill the server and restart, then you will see:
> {code:java}
> 11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group 
> root.turbine- registers active time series root.turbine.d2.s0 
> failed11:37:45.157 [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Storage group 
> root.turbine- registers active time series root.turbine.d2.s0 
> failedjava.lang.NullPointerException: null at 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.offer(ActiveTimeSeriesCounter.java:65)
>  at 
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:87)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
>  at 
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)11:37:45.158 
> [pool-1-IoTDB-Recovery-Thread-Pool-thread-1] ERROR 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter - Update 
> root.turbine- active ratio failedjava.lang.NullPointerException: null at 
> org.apache.iotdb.db.conf.adapter.ActiveTimeSeriesCounter.updateActiveRatio(ActiveTimeSeriesCounter.java:79)
>  at 
> org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:91)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:213)
>  at 
> org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:143)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:294)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:223)
>  at 
> org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.(StorageGroupProcessor.java:210)
>  at 
> org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:119) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> {code}
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)

Re: [jira] [Created] (IOTDB-346) StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime

[李天安]

Hi, 

After I replicate your experiments, I found it’s a bug of renaming tsfile when 
it has an out-of-order file name with current tsfiles. I will add a new ut test 
and pull a request to master soon.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> 2019年12月9日 上午12:14，Lei Rui (Jira)  写道：
> 
> Lei Rui created IOTDB-346:
> -
> 
> Summary: StorageGroupProcessor.sequenceFileList is ordered by 
> fileName rather than dataTime
> Key: IOTDB-346
> URL: https://issues.apache.org/jira/browse/IOTDB-346
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Lei Rui
> 
> 
> `StorageGroupProcessor.sequenceFileList` is ordered by fileName rather than 
> by time of data, as reflected in the `StorageGroupProcessor.getAllFiles` 
> method code:
> {code:java}
> tsFiles.sort(this::compareFileName);
> {code}
> 
> I use the following examples to expose the bug when the order of fileName is 
> inconsistent with that of dataTime.
> 
> First, for preparation, I created three tsfiles using the following sql:
> {code:java}
> SET STORAGE GROUP TO root.ln.wf01.wt01
> CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, 
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE, 
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32, 
> ENCODING=PLAIN
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(1, 1.1, false, 11)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(2, 2.2, true, 22)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(3, 3.3, false, 33)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(4, 4.4, false, 44)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(5, 5.5, false, 55)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(100, 100.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(150, 200.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(200, 300.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(250, 400.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(300, 500.5, false, 550)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(10, 10.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(20, 20.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(30, 30.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(40, 40.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(50, 50.5, false, 550)
> flush
> {code}
> The tsfiles created are organized in the following directory structure:
> {code:java}
> |data
> |--sequence
> |root.ln.wf01.wt01
> |--1575813520203-101-0.tsfile
> |--1575813520203-101-0.tsfile.resource
> |--1575813520669-103-0.tsfile
> |--1575813520669-103-0.tsfile.resource
> |--unsequence
> |root.ln.wf01.wt01
> |--1575813521063-105-0.tsfile
> |--1575813521063-105-0.tsfile.resource
> {code}
> ||File Name||Data Time||
> |(a) 1575813520203-101-0.tsfile|1-5|
> |(c) 1575813521063-105-0.tsfile|10-50|
> |(b) 1575813520669-103-0.tsfile|100-300|
> 
> Note how the order of fileName is inconsistent with that of dataTime.
> 
> By the way, if you look into the code, you will know how the file name is 
> generated:
> {code:java}
> System.currentTimeMillis() + IoTDBConstant.TSFILE_NAME_SEPARATOR + 
> versionController.nextVersion() + IoTDBConstant.TSFILE_NAME_SEPARATOR + "0" + 
> TSFILE_SUFFIX
> {code}
> 
> Then, I loaded the three tsfiles into another brand new IoTDB. I did two 
> experiments with different loading orders each.
> 
> In the first experiment, the tsfiles were loaded in their data time order. 
> That is,
> {code:java}
> IoTDB> load 1575813520203-101-0.tsfile // tsfile (a), with data time 1-5
> IoTDB> load 1575813521063-105-0.tsfile // tsfile (c), with data time 10-50
> IoTDB> load 1575813520669-103-0.tsfile // tsfile (b), with data time 
> 100-300{code}
> After loading successfully, I did the following query in the same client

Re: [jira] [Created] (IOTDB-346) StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime

[李天安]

Hi,

The `load` function is my work, so it’s necessary to answer this issue by me.

After your first experiment, it sounds abnormal. I need to replicate your 
results to locate the problem. In normal implementation logic, it wound rename 
the tsfile and load it into engine, which makes sure that the order of 
`fileName` is the same as the order of `dataTime`.

In your second experiment, it’s a normal case. The unsequence file actually has 
no overlap time range with sequences list, the reason why it has become 
`unsequence` is just that the data inserted late. So in `load` function, I will 
judge the time range conflict and based on that it’ll do different actions.

I hope this will help and thanks for your experiments.  


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> 2019年12月9日 上午12:14，Lei Rui (Jira)  写道：
> 
> Lei Rui created IOTDB-346:
> -
> 
> Summary: StorageGroupProcessor.sequenceFileList is ordered by 
> fileName rather than dataTime
> Key: IOTDB-346
> URL: https://issues.apache.org/jira/browse/IOTDB-346
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: Lei Rui
> 
> 
> `StorageGroupProcessor.sequenceFileList` is ordered by fileName rather than 
> by time of data, as reflected in the `StorageGroupProcessor.getAllFiles` 
> method code:
> {code:java}
> tsFiles.sort(this::compareFileName);
> {code}
> 
> I use the following examples to expose the bug when the order of fileName is 
> inconsistent with that of dataTime.
> 
> First, for preparation, I created three tsfiles using the following sql:
> {code:java}
> SET STORAGE GROUP TO root.ln.wf01.wt01
> CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, 
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE, 
> ENCODING=PLAIN
> CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32, 
> ENCODING=PLAIN
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(1, 1.1, false, 11)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(2, 2.2, true, 22)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(3, 3.3, false, 33)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(4, 4.4, false, 44)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(5, 5.5, false, 55)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(100, 100.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(150, 200.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(200, 300.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(250, 400.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(300, 500.5, false, 550)
> flush
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(10, 10.1, false, 110)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(20, 20.2, true, 220)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(30, 30.3, false, 330)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(40, 40.4, false, 440)
> INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) 
> values(50, 50.5, false, 550)
> flush
> {code}
> The tsfiles created are organized in the following directory structure:
> {code:java}
> |data
> |--sequence
> |root.ln.wf01.wt01
> |--1575813520203-101-0.tsfile
> |--1575813520203-101-0.tsfile.resource
> |--1575813520669-103-0.tsfile
> |--1575813520669-103-0.tsfile.resource
> |--unsequence
> |root.ln.wf01.wt01
> |--1575813521063-105-0.tsfile
> |--1575813521063-105-0.tsfile.resource
> {code}
> ||File Name||Data Time||
> |(a) 1575813520203-101-0.tsfile|1-5|
> |(c) 1575813521063-105-0.tsfile|10-50|
> |(b) 1575813520669-103-0.tsfile|100-300|
> 
> Note how the order of fileName is inconsistent with that of dataTime.
> 
> By the way, if you look into the code, you will know how the file name is 
> generated:
> {code:java}
> System.currentTimeMillis() + IoTDBConstant.TSFILE_NAME_SEPARATOR + 
> versionController.nextVersion() + IoTDBConstant.TSFILE_NAME_SEPARATOR + "0" + 
> TSFILE_SUFFIX
> {code}
> 
> Then, I loaded the three tsfiles into another brand new IoTDB. I did two 
> experiments with different loading orders each.
> 
> In the first 

Re: [VOTE] Apache IoTDB (incubating) 0.8.2 RC2 release

[李天安]

Hi,
My vote is +1 (PPMC)

I have checked following items:

Source release:
1. Running RAT(mvn apache-rat:check) (Correct)
2. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
extracted source bundle (Correct)
3. Compile and build code with command: mvn clean package. (Correct)
4. Verifying the signature (ASC) (Correct)
5. Verifying the hashes (SHA512) (Correct)

Binary distribution:
1. Verifying the signature (ASC) (Correct)
2. Verifying the hashes (SHA512) (Correct)
3. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files 
(Correct)

Functionality：
1. Run server and client script:  sh start-server.sh, start-client.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
2. REBOOT
3. Execuet some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 6
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院



>> Hi all,
>> 
>> Now I open a new vote for Apache IoTDB (incubating) 0.8.2 RC2, which fixes
>> issues in RC1.
>> 
>> Apache IoTDB (incubating) 0.8.2 is a bug-fix version from 0.8.1. You can
>> get its mainly changes from [5].
>> 
>> Apache IoTDB (Incubating) 0.8.2 has been staged under [2] and it’s time to
>> vote
>> on accepting it for release.  All Maven artifacts are available under [1].
>> If approved we will seek final release approval from the IPMC.
>> 
>> Voting will be open for 72hr.
>> 
>> A minimum of 3 binding +1 votes and more binding +1 than binding -1
>> are required to pass.
>> 
>> Release tag: release/0.8.2
>> Hash for the release tag: 09a94f2c02dec27f483a2ccb41a8265331be193d
>> 
>> Before voting +1 PMC/PPMC members are required to download
>> the signed source code package, compile it as provided, and test
>> the resulting executable on their own platform, along with also
>> verifying that the package meets the requirements of the ASF policy
>> on releases.
>> 
>> You can achieve the above by following [4].
>> 
>> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM
>> items in [4])
>> 
>> [ ]  -1 reject (explanation required)
>> 
>> 
>> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1021
>> [2] https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.8.2/rc2
>> [3] https://www.apache.org/dev/release.html#approving-a-release
>> [4]
>> 
>> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
>> [5]
>> 
>> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.8.2/rc2/RELEASE_NOTES.md
>> 
>> Best,
>> ---
>> Xiangdong Huang
>> School of Software, Tsinghua University
>> 
>> 黄向东
>> 清华大学 软件学院
>> 
> 

Re: [VOTE] Apache IoTDB 0.9.0 (incubating) RC4 release

[李天安]

Hi,
My vote is +1 (PPMC)

I have checked following items:

Source release:
1. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
extracted source bundle (Correct)
2. Compile and build code with command: mvn clean package. (Correct)
3. Verifying the signature (ASC) (Correct)
4. Verifying the hashes (SHA512) (Correct)
5. Running RAT(mvn apache-rat:check) (Correct)

Binary distribution:
1. Verifying the signature (ASC) (Correct)
2. Verifying the hashes (SHA512) (Correct)
3. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files 
(Correct)

Functionality：
1. Run server and client script:  sh start-server.sh, start-client.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
2. REBOOT
3. Execuet some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 5
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年11月21日 下午12:41，Jialin Qiao  写道：
> 
> Hi all,
> 
> 
> Now I open a new ticket for Apache IoTDB (incubating) 0.9.0 RC4, which fixs
> some issues comparing with RC3. You can find the details at the end of this
> email.
> 
> 
> Apache IoTDB (incubating) 0.9.0 is a major version with many exciting 
> features. 
> You can get the release notes in [5].
> 
> 
> Apache IoTDB (Incubating) 0.9.0 has been staged under [2] and it’s time to
> vote on accepting it for release.  All Maven artifacts are available under 
> [1].
> If approved we will seek final release approval from the IPMC.
> 
> 
> Voting will be open for 72hr.
> 
> 
> A minimum of 3 binding +1 votes and more binding +1 than binding -1 are 
> required to pass.
> 
> 
> Release tag: release/0.9.0
> 
> 
> Hash for the release tag: eedfec84462c0f574f9d6ade5ac53801cb3bc869
> 
> 
> Per [3] "Before voting +1 PMC/PPMC members are required to download the 
> signed source code package, compile it as provided, and test the resulting 
> executable on their own platform, along with also verifying that the package 
> meets the requirements of the ASF policy on releases."
> 
> 
> Difference with RC3:
> 
> 
> Remove the .tsfile
> 
> 
> 
> 
> You can achieve the above by following [4].
> 
> 
> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM items 
> in [4])
> [ ]  -1 reject (explanation required)
> 
> 
> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1016
> [2] https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc4
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4] 
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> [5] 
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc4/RELEASE_NOTES.md
> 
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 
> 乔嘉林
> 清华大学 软件学院

Re: [VOTE] Apache IoTDB 0.9.0 (incubating) RC2 release

[李天安]

Hi,
+1 (PPMC vote)

I have checked following items:

Source release:
1. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
extracted source bundle (Correct)
2. Compile and build code with command: mvn clean package. (Correct)
3. Verifying the signature (ASC) (Correct)
4. Verifying the hashes (SHA512) (Correct)
5. Running RAT(mvn apache-rat:check) (Correct)

Binary distribution:
1. Verifying the signature (ASC) (Correct)
2. Verifying the hashes (SHA512) (Correct)
3. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files 
(Correct)

Functionality：
1. Run server and client script:  sh start-server.sh, start-client.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
2. REBOOT
3. Execuet some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 5
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年11月16日 下午10:43，Jialin Qiao  写道：
> 
> Hi all,
> 
> 
> Now I open a new ticket for Apache IoTDB (incubating) 0.9.0 RC2, which fixs
> some issues comparing with RC1. You can find the details at the end of this
> email.
> 
> 
> Apache IoTDB (incubating) 0.9.0 is a major version with many exciting 
> features. 
> You can get the release notes in [5].
> 
> 
> Apache IoTDB (Incubating) 0.9.0 has been staged under [2] and it’s time to
> vote on accepting it for release.  All Maven artifacts are available under 
> [1].
> If approved we will seek final release approval from the IPMC.
> 
> 
> Voting will be open for 72hr.
> 
> 
> A minimum of 3 binding +1 votes and more binding +1 than binding -1 are 
> required to pass.
> 
> 
> Release tag: release/0.9.0
> 
> 
> Hash for the release tag: 0eb718f4587c824838e39dd53f16ba93a9ea90a8
> 
> 
> Per [3] "Before voting +1 PMC/PPMC members are required to download the 
> signed source code package, compile it as provided, and test the resulting 
> executable on their own platform, along with also verifying that the package 
> meets the requirements of the ASF policy on releases."
> 
> 
> Difference with RC1:
> 
> 
>> - There an unexpected binary file in the release [4]  &&   There is a "jar" 
>> file in the build.
> Remov ./.mvn/wrapper/maven-wrapper.jar in the source release.
>> - LICENSE need to mention license of Maven Wrapper
> Add maven-wrapper in LICENSE
>> - NOTICE is incorrect "Copyright 2019 and onwards” is not valid to say 
> Fix IoTDB NOTICE copyright to: Copyright 2018-2019 The Apache Software 
> Foundation.
>> - NOTICE is missing information from from the common NOTICE file [1][3]
> Add Commons-collections in NOTICE and NOTICE-binary
>> - A large number of files have the incorrect ASF header. There should be no 
>> "Copyright © 2019 Apache IoTDB(incubating) (dev@iotdb.apache.org)” notice in 
>> the files.
> Fix incorrect ASF header in some files, remove : Copyright © 2019 Apache 
> IoTDB(incubating) (dev@iotdb.apache.org)
> 
> 
> 
> 
> You can achieve the above by following [4].
> 
> 
> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM items 
> in [4])
> [ ]  -1 reject (explanation required)
> 
> 
> [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1013
> [2] https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc2
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4] 
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> [5] 
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc2/RELEASE_NOTES.md
> 
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 
> 乔嘉林
> 清华大学 软件学院

Re: [DISCUSS] Release 0.9.0

[李天安]

Hi Jialin,
Maybe the sync module is also worthy to mention in the change list. The pr 
link is https://github.com/apache/incubator-iotdb/pull/372 
<https://github.com/apache/incubator-iotdb/pull/372>.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年10月29日 下午5:11，Jialin Qiao  写道：
> 
> Hi,
> 
> 
> As 0.8.1 has been released, the release of 0.9.0 could start.
> 
> 
> I would like to do the release manager of 0.9.0, and I gathered the change 
> list:
> 
> 
> 
> 
> ## New Features
> 
> 
> * IOTDB-143Compaction of data file
> * IOTDB-205Support storage-group-level Time To Live (TTL)
> * IoTDB 226Hive connect TsFile
> * IOTDB-188Delete storage group
> * IOTDB-253support time expression 
> * IOTDB-239Add interface for showing devices
> * IOTDB-249enable lowercase in create_timeseries sql
> * IOTDB-203add "group by device" function for narrow table display
> * IOTDB-193Create schema automatically when inserting
> * IOTDB-241Add query and non query interface in session
> * IOTDB-223Add a TsFile sketch tool
> * IOTDB-158add metrics web service
> * IOTDB-234Refactor TsFile storage on HDFS
> * IOTDB-199Add a log visulization tool 
> * IoTDB-174Add interfaces for querying device or timeseries number
> * IOTDB-173add batch write interface in session
> * IOTDB-151support number format in timeseries path
> * Spark-iotdb-connector
> * generate cpp, go, and python thrift files under service-rpc
> * display cache hit rate through jconsole
> * support time < 0: Fix initial value of minTimestamp to Long.MIN_VALUE in 
> ChunkBuffer
> * Add interface (Delete timeseries) in session 
> * Add a tool to print tsfileResources (each device's start and end time)
> * Support watermark feature
> * Add micro and nano timestamp precision
> 
> 
> ## Incompatible changes
> 
> 
> * RPC is incompatible, you can not use client-0.8.0 to connect with 
> server-0.9.0 or use client-0.9.8 to connect with server-0.8.0.
> * Server is backward compatible, server-0.9.0 could run on data folder of 
> 0.8.0. The data file will be upgraded background.
> 
> 
> https://github.com/apache/incubator-iotdb/pull/467
> 
> 
> ## Miscellaneous changes
> 
> 
> * IOTDB-258Add documents for Query History Visualization Tool and Shared 
> Storage Architecture
> * IOTDB-233keep metadata plan clear
> * IOTDB-267reduce IO operations in deserializing chunk header
> * IOTDB-265Re-adjust the threshold size of memtable
> * IOTDB-251improve TSQueryDataSet structure in RPC
> * IOTDB-221Add a python client example
> * IOTDB-180Get rid of JSON format in "show timeseries"
> * IOTDB-161Add ErrorCode of different response errors
> * IOTDB-160External sort
> * IOTDB-153further limit fetchSize to speed up LIMIT&OFFSET query
> 
> 
> * reconstruct antlrv3 grammar to improve performance
> * Tooling for release
> * Modified Decoder and SequenceReader to support old version of TsFile 
> * Remove jdk constrain of jdk8 and 11
> 
> 
> ## Known Issues
> 
> 
> * IOTDB-20Need to support UPDATE
> 
> 
> ## Bug Fixes
> 
> 
> * IOTDB-266 NullPoint exception when reading not existed devices using 
> ReadOnlyTsFile
> * IOTDB-264restart failure due to WAL replay error
> * IOTDB-165[TsFile] Fix a example
> * IOTDB-240fix unknown time series in where clause
> * IOTDB-244fix bug when querying with duplicated columns
> * IOTDB-174Fix querying timeseries interface cannot make a query by the 
> specified path prefix
> * IOTDB-195using String.getBytes(utf-9).length to replace string.length() in 
> ChunkGroupMetadata for supporting Chinese
> * IOTDB-211use "%IOTDB_HOME%\lib\*" to refers to all .jar files in the 
> directory in start-server.bat
> * IOTDB-172fix a bug of TsFileResource updateTime
> * fix start-walchecker scripts for leting user define the wal folder
> 
> 
> Is there any missing?
> 
> 
> 
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院

Re: Sync module application scenario

[李天安]

Hi, 

If some users break the limitation，the sync module does not check this 
scenario and will not guarantee the correctness. In that case, the cost is too 
high if we check for that scenario. Under the existing merge policy, after the 
merge process the time range order of the device of the tsfils under the 
sequence folder is broken, only the time range order of the timeseries is 
guaranteed. So when loading a new tsfile, it is hard to determine whether the 
time interval of a tsfile overlaps the data interval of an existing file only 
through TsFileResource. If there has overlap, we need to compare the time range 
of the chunk with that of that file with an overlapping time range of the 
device. In this way, the overhead will increase a lot.
   So I don't recommend checking the scenario under the existing merge policy, 
which will increase a lot of overhead.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年10月23日 下午5:21，Xiangdong Huang  写道：
> 
> Hi,
> 
>> there is no device overlap between the data synchronized by multiple
> senders. Otherwise, the application scenario of the sync module is not
> satisfied.
> 
> So my question is, if some user break the limitation, what will happen? Do
> you have some checks for reject the scenario?  or do we need to do the
> check?
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
> 黄向东
> 清华大学 软件学院
> 
> 
> 李天安  于2019年10月23日周三 上午4:55写道：
> 
>> Hi,
>> 
>>  With the sync module test completed, this feature will be online
>> soon, so I think it is necessary to describe the scene of using the sync
>> function here.
>> 
>>  In the case of a factory application, there are usually multiple
>> sub-factories and multiple general(main) factories. Each sub-factory uses
>> an IoTDB instance to collect data, and then synchronize the data to the
>> general factory for backup or analysis. A general factory can receive data
>> from multiple sub-factories and a sub-factory can also synchronize data to
>> multiple general factories. In this scenario, each IoTDB instance manages
>> different devices.
>> 
>>  In the sync module, each sub-factory is a sender, a general factory
>> is a receiver, and senders periodically synchronizes the data to receivers.
>> In the above application scenario, the data of one device can only be
>> collected by one sender, so there is no device overlap between the data
>> synchronized by multiple senders. Otherwise, the application scenario of
>> the sync module is not satisfied.
>> 
>>  If you have any idea of the sync application scenario, welcome to
>> discuss it here.
>> 
>> 
>> Best Regards,
>> —
>> Tianan Li
>> School of Software, Tsinghua University
>> 
>> 李天安
>> 清华大学 软件学院
>> 
>> 
> 

Sync module application scenario

[李天安]

Hi, 

  With the sync module test completed, this feature will be online soon, so 
I think it is necessary to describe the scene of using the sync function here. 
  
  In the case of a factory application, there are usually multiple 
sub-factories and multiple general(main) factories. Each sub-factory uses an 
IoTDB instance to collect data, and then synchronize the data to the general 
factory for backup or analysis. A general factory can receive data from 
multiple sub-factories and a sub-factory can also synchronize data to multiple 
general factories. In this scenario, each IoTDB instance manages different 
devices. 
  
  In the sync module, each sub-factory is a sender, a general factory is a 
receiver, and senders periodically synchronizes the data to receivers. In the 
above application scenario, the data of one device can only be collected by one 
sender, so there is no device overlap between the data synchronized by multiple 
senders. Otherwise, the application scenario of the sync module is not 
satisfied.

  If you have any idea of the sync application scenario, welcome to discuss 
it here.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC3 release

[李天安]

Hi,
+1 (PPMC vote)

I have checked following items:

Source release:
1. Running RAT(mvn apache-rat:check) (Correct)
2. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
extracted source bundle (Correct)
3. Compile and build code with command: mvn clean package. (Correct)
4. Verifying the signature (ASC) (Correct)
5. Verifying the hashes (SHA512) (Correct)

Binary distribution:
1. Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files 
(Correct)
2. Verifying the signature (ASC) (Correct)
3. Verifying the hashes (SHA512) (Correct)

Functionality：
1. Run server and client script:  sh start-server.sh, start-client.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
2. REBOOT
3. Execuet some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 5
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年10月15日 下午2:18，Jialin Qiao  写道：
> 
> Hi,
> 
> +1 (PPMC vote)
> 
> I checked:
> 
> source release:
> verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the 
> extracted source bundle (ok)
> incubating in name (ok)
> signature and hashes (ok)
> DISCLAIMER exists (ok)
> LICENSE (ok)
> NOTICE (ok)
> ASF headers (ok)
> no jars in source release (ok)
> mvn clean package (ok)
> test all passed on mac os + jdk1.8(ok)
> 
> 
> binary distribution:
> incubating in name (ok)
> UNPACK the binary artefact and check the contents of LICENSE-binary and 
> NOTICE-binary (ok)
> DISCLAIMER exists (ok)
> LICENSE (ok)
> NOTICE (ok)
> Start server (ok)
> Start client (ok)
> The version printed in client is 0.8.1 (ok)
> compatible with 0.8.0, both client-0.8.1 with server-0.8.0 and client-0.8.0 
> with server-0.8.1 (ok)
> Example statements are as follow:
> 
> ```
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=FLOAT, ENCODING=RLE;
> insert into root.turbine.d1(timestamp,s0) values(1,24.5);
> select * from root;
> ```
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 
>> -原始邮件-
>> 发件人: "Xiangdong Huang" 
>> 发送时间: 2019-10-14 17:28:35 (星期一)
>> 收件人: dev@iotdb.apache.org
>> 抄送: 
>> 主题: [VOTE] Apache IoTDB 0.8.1 (incubating) RC3 release
>> 
>> Hi all,
>> 
>> Now I open a new ticket for Apache IoTDB (incubating) 0.8.1 RC3, which fixs
>> some issues comparing with RC2. You can find the details at the end of this
>> email.
>> 
>> Apache IoTDB (incubating) 0.8.1 is a bug-fix version from 0.8.0. You can
>> get its mainly changes from [5].
>> 
>> Apache IoTDB (Incubating) 0.8.1 has been staged under [2] and it’s time to
>> vote
>> on accepting it for release.  All Maven artifacts are available under [1].
>> If approved we will seek final release approval from the IPMC.
>> Voting will be open for 72hr.
>> 
>> A minimum of 3 binding +1 votes and more binding +1 than binding -1
>> are required to pass.
>> 
>> Release tag: release/0.8.1
>> Hash for the release tag: 4b8d46f9861e12bbe4a1d952bede7c9293b90b46
>> 
>> Per [3] "Before voting +1 PMC/PPMC members are required to download
>> the signed source code package, compile it as provided, and test
>> the resulting executable on their own platform, along with also
>> verifying that the package meets the requirements of the ASF policy
>> on releases."
>> 
>> You can achieve the above by following [4].
>> 
>> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM
>> items in [4])
>> [ ]  -1 reject (explanation required)
>> 
>> 
>> Difference with RC1:
>> 
>> - PGP signing: there is no established trust chain to any other Apache ...
>> - Running IoTDB Server according to information in README.md: MINOR I am
>> using OpenJDK 13 and I get an error message that IoTDB only runs on Java 8
>> and 11 ... this sort of contradicts the README which states X >= 1.8
>> - I would suggest to add an Apache Header to the ".checkstyle" XML file
>> - It seems the Hadoop module still contains SNAPSHOT versions: MINOR
>> - Able to run the server and client: MINOR Same finding as with the source
>> distribu

Re: Weird things while using benchmark.

[李天安]

Hi,


[This email should be sent on September 25, 2019, but it has not been sent 
successfully due to Netease Master Mailbox Client.]


This is indeed a problem encountered with the Dynamic Parameter Adapter(DPA) 
module. More work is needed to make the module better.


Currently, the DPA module estimates a load of each storage group(sg) in the 
most pessimistic way, which is the maximum load the sg may have. In most cases, 
it indeed exaggerates the current load on the system. If the module tells you 
that the system’s load is too large, it actually means that if all sgs have 
maximum loads and in this case, you execute the command may crash the system. 


Therefore,  the DPA module considers the maximum possible load rather than the 
real-time load of the system. which ensures that the system will not be out of 
memory, but to some extent sacrifices the availability of the system.  It’s a 
trade-off between these two aspects, it’s better to develop a module that 
monitors the real-time writing load of the system to cooperate with the DPA 
module.




Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


On 09/25/2019 14:38，Xiangdong Huang wrote：
Hi,

The codes of iotdb-benchmark is not in our repository.. but I think it is
fine to discuss the issue about it (if it can help us to improve IoTDB
project).

As for the issue of Dynamic Parameter Adapter, I agree with @Tian jiang.

Can the author (@tianan) give more information about it? and then let's
find a solution if some guys are interested in it..

Besides, when it judges the workload to be unacceptable and refuses to
create time series, it does not leave enough logs or give enough feedback
to the user to indicate the cause.

This should be reported as a new issues on JIRA.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

黄向东
清华大学 软件学院


Tian Jiang  于2019年9月25日周三 上午11:49写道：

Well, it turns out that the Dynamic Parameter Adapter thought the memory
is insufficient according to the workload and refused to create new time
series. But actually when I turned it off, I found that was a misjudge
since it is okay using the default parameters.


I think the definition of "workload too heavy" in the Dynamic Parameter
Adapter is too strict, which dramatically underestimates the ability of the
system. Besides, when it judges the workload to be unacceptable and refuses
to create time series, it does not leave enough logs or give enough
feedback to the user to indicate the cause.


| |
Tian Jiang
|
|
jt2594...@163.com
|
签名由网易邮箱大师定制
On 9/25/2019 11:31，徐毅 wrote：
Hi,


You can contact liurui directly. I guess there are some error
configurations in your file.


Thanks
XuYi
On 09/25/2019 11:04，Tian Jiang wrote：
Hi,


When I used IoTDB-benchmark to perform insertions to IoTDB (both are the
master branch), I got some really weird phenomenons.
1. This is my settings of the benchmark and I expected to have 14 threads
to perform the insertion, while actually I only found 2 threads working.


2. I got a lot of failures when creating new time series, and the error
code did not explain it well. Since I started the experiment with a clean
IoTDB, I do not think this was caused by creating existing timeseries and I
did not find related logs on the server side.




3. This may be related to issue2. In the first attempts of insertions, I
got some errors indicating that the time series are not created
successfully.


These issues have blocked my progress of testing TTL, I hope someone could
fix it ASAP.
| |
Tian Jiang
|
|
jt2594...@163.com
|
签名由网易邮箱大师定制

Re: Failed to send mails to dev@iotdb.apache.org

[李天安]

Hi ZeSong,
Thanks for your suggestion.
By the way, this is also a test email sent by the Mail Client of MacOs.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 2019年10月9日 下午10:37，孙泽嵩  写道：
> 
> Hi Tianan,
> 
> Netease master mailbox client did have problem in sending mails… I had 
> encountered the same situation before.
> 
> Only when RECEIVING your own mail from dev proves that you have successfully 
> sent the e-mail.
> 
> Maybe you have to try another method (Mail client of MacBook is very easy to 
> use, I think).
> 
> Best,
> ---
> Zesong Sun
> School of Software, Tsinghua University
> 
> 孙泽嵩
> 清华大学 软件学院
> 
>> 2019年10月9日 22:11，Kevin A. McGrail  写道：
>> 
>> I've never heard of the netease master mailbox client. I would guess
>> it's a bug with that software.  Can you cc me on an example email using
>> that software and I'll look if I see anything strange with it? 
>> 
>> Maybe it's spoofing some email headers and you are then running afoul of
>> email security and anti-spam constraints is my $0.02 off the cuff.
>> 
>> On 10/9/2019 10:02 AM, 李天安 wrote:
>>> Hi,
>>> Recently, I encountered an awful thing. I found that Netease Master Mailbox 
>>> Client can't send mails to "dev@iotdb.apache.org" successfully. 
>>> Every time after I sent an email, the email would indeed appear in my 
>>> mailing list, which made me think that I sent it successfully, but actually 
>>> I didn't. In this case, I suddenly found that most of my previous emails 
>>> were not sent successfully... That's too bad.  
>>> What's more, I need to retry many times through the web page mailbox before 
>>> I could send an email successfully. 
>>> Can anyone help me with that? Thanks.
>>> 
>>> 
>>> Best Regards,
>>> —
>>> Tianan Li
>>> School of Software, Tsinghua University
>>> 
>>> 李天安
>>> 清华大学 软件学院
>> 
>> -- 
>> Kevin A. McGrail
>> kmcgr...@apache.org
>> 
>> Member, Apache Software Foundation
>> Chair Emeritus Apache SpamAssassin Project
>> https://www.linkedin.com/in/kmcgrail - 703.798.0171
> 
> 

Failed to send mails to dev@iotdb.apache.org

[李天安]

Hi,
Recently, I encountered an awful thing. I found that Netease Master Mailbox 
Client can't send mails to "dev@iotdb.apache.org" successfully. 
Every time after I sent an email, the email would indeed appear in my mailing 
list, which made me think that I sent it successfully, but actually I didn't. 
In this case, I suddenly found that most of my previous emails were not sent 
successfully... That's too bad.  
What's more, I need to retry many times through the web page mailbox before I 
could send an email successfully. 
Can anyone help me with that? Thanks.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

Re: Re: Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release

[李天安]

Hi,
+1 (binding)

I have checked following items:

1. Compile and build the source release code with command: mvn clean package 
-Dmaven.test.skip=true. (Correct)
2. Run server and client script:  sh start-server.sh, start-client.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
3. REBOOT
4. Execuet some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 5
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> -原始邮件-
> 发件人: "刘睿" 
> 发送时间: 2019-10-09 20:26:43 (星期三)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Re:  [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release
> 
> Hi,
> 
> +1 (binding)
> 
> Checked items:
> 
> Source release:
> ✅ downloaded pre-release/iotdb-0.8.1 rc2
> ✅ NOTICE, LICENSE, RELEASE_NOTES and DISCLAIMER
> ✅ mvn clean package
> ✅ Start server
> ✅ Start client
> ✅ Basic statements including metadata manipulation, insertion and query
> 
> 
> Sincerely,
> Rui Liu
> 
> 在2019-10-08 15:35:06,Tian Jiang写道：
> > Hi,
> > 
> > +1 (binding)
> > 
> > Checked items:
> > 
> > Binary release:
> > NOTICE, LICENSE and DISCLAIMER
> > Start server
> > Start client
> > Execute some basic statements (metadata manipulation, insertion, query)
> > Reboot
> > 
> > 
> > Source release:
> > NOTICE, LICENSE and DISCLAIMER
> > mvn clean test install 
> > 
> > Execute some basic statements (metadata manipulation, insertion, query)
> > Reboot
> > 
> > 
> > Best,
> > Tian Jiang 
> > 
> > 
> > 
> > 
> > On 10/8/2019 15:07，Jialin Qiao wrote：
> > Hi,
> > 
> > +1 (binding)
> > 
> > I checked:
> > 
> > Binary release:
> > has NOTICE, LICENSE and DISCLAIMER
> > Start server (ok)
> > Start client (ok)
> > The version printed in client is 0.8.1 (ok)
> > compatible with 0.8.0, both client-0.8.1 with server-0.8.0 and client-0.8.0 
> > with server-0.8.1
> > Example statements are as follow:
> > 
> > ```
> > SET STORAGE GROUP TO root.turbine;
> > CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=FLOAT, ENCODING=RLE;
> > insert into root.turbine.d1(timestamp,s0) values(1,24.5);
> > select * from root;
> > ```
> > 
> > Source release:
> > has NOTICE, LICENSE and DISCLAIMER
> > mvn clean package (ok)
> > test all passed on mac os + jdk1.8(ok)
> > 
> > minor issue:
> > 
> > filters in TsFileRead in the example of tsfile module are not compatible 
> > with the file written by TsFileWrite
> > 
> > Best,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> > 
> > 乔嘉林
> > 清华大学 软件学院
> > 
> > -原始邮件-
> > 发件人: "Xiangdong Huang" 
> > 发送时间: 2019-10-05 19:02:47 (星期六)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release
> > 
> > Hi all,
> > 
> > Now I open a new ticket for Apache IoTDB (incubating) 0.8.1 RC2, which fixs
> > some issues comparing with RC1. You can find the details at the end of this
> > email.
> > 
> > Apache IoTDB (incubating) 0.8.1 is a bug-fix version from 0.8.0. You can
> > get its mainly changes from [5].
> > 
> > Apache IoTDB (Incubating) 0.8.1 has been staged under [2] and it’s time to
> > vote
> > on accepting it for release.  All Maven artifacts are available under [1].
> > If approved we will seek final release approval from the IPMC.
> > Voting will be open for 72hr.
> > 
> > A minimum of 3 binding +1 votes and more binding +1 than binding -1
> > are required to pass.
> > 
> > Release tag: release/0.8.1
> > Hash for the release tag: 37b213b6ab6fc8ca13ab60f21dff099042a0e295
> > 
> > Per [3] "Before voting +1 PMC/PPMC members are required to download
> > the signed source code package, compile it as provided, and test
> > the resulting executable on their own platform, along with also
> > verifying that the package meets the requirements of the ASF policy
> > on releases."
> > 
> > You can achieve the above by following [4].
> > 
> > [ ]  +1 accept (indicate what y

Re: Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release

[李天安]

Hi,
+1 (binding)

I have checked following items:

1. Compile and build the source release code with command: mvn clean package 
-Dmaven.test.skip=true. (Correct)
2. Run server and client script:  sh start-server.sh, start-client.sh -h 
127.0.0.1 -p 6667 -u root -pw root (Correct)
3. REBOOT
4. Execuet some basic operations: (Correct)
1) SET STORAGE GROUP TO root.vehicle
2) CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
3) insert into root.vehicle.d0(timestamp,s0) values(10,100)
4) insert into root.vehicle.d0(timestamp,s0) values(1,101)
5) select * from root.vehicle
6) select * from root.vehicle where time > 5
7) show timeseries
8) show timeseries root.*
9) delete from root.vehicle.d0.s0 where time < 9
10) select * from root.vehicle

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> -原始邮件-
> 发件人: "刘睿" 
> 发送时间: 2019-10-09 20:26:43 (星期三)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Re:  [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release
> 
> Hi,
> 
> +1 (binding)
> 
> Checked items:
> 
> Source release:
> ✅ downloaded pre-release/iotdb-0.8.1 rc2
> ✅ NOTICE, LICENSE, RELEASE_NOTES and DISCLAIMER
> ✅ mvn clean package
> ✅ Start server
> ✅ Start client
> ✅ Basic statements including metadata manipulation, insertion and query
> 
> 
> Sincerely,
> Rui Liu
> 
> 在2019-10-08 15:35:06,Tian Jiang写道：
> > Hi,
> > 
> > +1 (binding)
> > 
> > Checked items:
> > 
> > Binary release:
> > NOTICE, LICENSE and DISCLAIMER
> > Start server
> > Start client
> > Execute some basic statements (metadata manipulation, insertion, query)
> > Reboot
> > 
> > 
> > Source release:
> > NOTICE, LICENSE and DISCLAIMER
> > mvn clean test install 
> > 
> > Execute some basic statements (metadata manipulation, insertion, query)
> > Reboot
> > 
> > 
> > Best,
> > Tian Jiang 
> > 
> > 
> > 
> > 
> > On 10/8/2019 15:07，Jialin Qiao wrote：
> > Hi,
> > 
> > +1 (binding)
> > 
> > I checked:
> > 
> > Binary release:
> > has NOTICE, LICENSE and DISCLAIMER
> > Start server (ok)
> > Start client (ok)
> > The version printed in client is 0.8.1 (ok)
> > compatible with 0.8.0, both client-0.8.1 with server-0.8.0 and client-0.8.0 
> > with server-0.8.1
> > Example statements are as follow:
> > 
> > ```
> > SET STORAGE GROUP TO root.turbine;
> > CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=FLOAT, ENCODING=RLE;
> > insert into root.turbine.d1(timestamp,s0) values(1,24.5);
> > select * from root;
> > ```
> > 
> > Source release:
> > has NOTICE, LICENSE and DISCLAIMER
> > mvn clean package (ok)
> > test all passed on mac os + jdk1.8(ok)
> > 
> > minor issue:
> > 
> > filters in TsFileRead in the example of tsfile module are not compatible 
> > with the file written by TsFileWrite
> > 
> > Best,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> > 
> > 乔嘉林
> > 清华大学 软件学院
> > 
> > -原始邮件-
> > 发件人: "Xiangdong Huang" 
> > 发送时间: 2019-10-05 19:02:47 (星期六)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release
> > 
> > Hi all,
> > 
> > Now I open a new ticket for Apache IoTDB (incubating) 0.8.1 RC2, which fixs
> > some issues comparing with RC1. You can find the details at the end of this
> > email.
> > 
> > Apache IoTDB (incubating) 0.8.1 is a bug-fix version from 0.8.0. You can
> > get its mainly changes from [5].
> > 
> > Apache IoTDB (Incubating) 0.8.1 has been staged under [2] and it’s time to
> > vote
> > on accepting it for release.  All Maven artifacts are available under [1].
> > If approved we will seek final release approval from the IPMC.
> > Voting will be open for 72hr.
> > 
> > A minimum of 3 binding +1 votes and more binding +1 than binding -1
> > are required to pass.
> > 
> > Release tag: release/0.8.1
> > Hash for the release tag: 37b213b6ab6fc8ca13ab60f21dff099042a0e295
> > 
> > Per [3] "Before voting +1 PMC/PPMC members are required to download
> > the signed source code package, compile it as provided, and test
> > the resulting executable on their own platform, along with also
> > verifying that the package meets the requirements of the ASF policy
> > on releases."
> > 
> > You can achieve the above by following [4].
> > 
> > [ ]  +1 accept (indicate what y

A serious hidden but simple bug caused by object abnormal publishing

[李天安]

Hi,
Recently，in developing sync module, I found a very important bug in the 
process of unit testing, which will seriously affect the normal operation of 
the system. During the unit test of sync module, I found that when I retrieved 
the next sequence folder from the multi-file directory manager, I may get an 
unseuqence folder, which made me confused and began to track down the problem.


 In IoTDB, sequence folders and unsequence folders are specified as user 
parameters, and currently the name of the sequence folder is "sequence" and the 
name of unsequence folder is “unsequence”, so there must be no overlap between 
the two parameters. Therefore, I suspected that it was an abnormal publishing 
of the sequence folder list and the unsequence folder list. I tracked all the 
places where it was possible to use the two objects and found that it wil 
returned the object directly when I require them from multi-file directory 
manager. It is further found that merging the two objects acquired in Storage 
Group Processor causes problems.


The problem is very simple and hidden but serious. The reason why it has 
not been found is that this method of incorrectly modifying the contents of the 
list is currently only used in the delete module, which is nearly unused. Now 
that this problem has been solved in sync pr, I just want to share with you 
here. The issue of variable publishing still needs to be paid attention to. In 
addition, people interested in sync module can focus on 
https://issues.apache.org/jira/browse/IOTDB-198.




Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

Re: [VOTE] Apache IoTDB (Incubating) 0.8.0 RC1

[李天安]

Hi,

+1 (binding)


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 在 2019年7月27日，上午11:18，Xiangdong Huang  写道：
> 
> Hi,
> 
> +1 (binding).
> 
> Thanks for pointing out the issues @Justin. Is there anything that we need
> to modify?
> 
> Good work! @Jialin.
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
> 黄向东
> 清华大学 软件学院
> 
> 
> Jialin Qiao  于2019年7月27日周六 上午11:16写道：
> 
>> Hi,
>> 
>> +1 (binding)
>> 
>> 
>> I fixed the problems that Justin mentioned in the PR [1], which is already
>> merged into rel/0.8 and cherry-picked to master. The followings are my
>> modifications:
>> 
>>> - NOTICE may include a little too much text.  I not seen an export
>> control placed in NOTICE before, and you only need to include the bit from
>> Hive notice that are relevant [1]
>> 
>> fixed, and the export control notice is removed.
>> 
>>> - All source files have ASF headers. However this file [4] incorrectly
>> has an ASF header. It’s 3rd party software and should not use the ASF
>> header.
>> 
>> removed the license and ASF header
>> 
>>> In the README.md
>>> - Under "Installation from source code” it point the user to GitHub. The
>> user should be pointed to the projects download page, we don’t want to
>> encourage users to use unreleased software.
>> 
>> I add the projects download url.
>> 
>>> - The text displayed under "If successful, you will see the the
>> following text in the terminal:” is not correct
>> 
>> fixed
>> 
>>> - The instruction for compiling are not correct “mvn XXX -pl iotdb-cli”
>> and “mvn XXX -pl iotdb” don’t work.
>> 
>> fixed
>> 
>>> - The script to start and stop the server have incorrect paths
>> 
>> fixed
>> 
>>> When starting the server I get this error, but I assume that just my
>> setup or environment. Can someone else confirm that once compiled the
>> source release they can start the server?
>>> Error: Could not find or load main class
>> org.apache.iotdb.db.service.IoTDB
>> 
>> not reproduced, may be it is caused by the above incorrect path and
>> commands. Now you can try again.
>> 
>>> For the binary convenience release:
>>> - The binary notice has way too much information in it. For instance you
>> only need to include the bit from Hive notice that are relevant [1],
>> logbook [2] has no notice file so there no need to include anything about
>> it in the NOTICE file [3]. There no need to mention licensing information,
>> that goes in LICENCE
>> 
>> fixed
>> 
>>> - Most permissive licenses say the full text of the license needs to be
>> included, that includes the copyright line for BSD and MIT licensed
>> software, I’m not sure that have generic license test in license covers
>> that.
>> 
>> Actually, we have a "licenses" folder under the root which contains full
>> text of the licenses.
>> 
>> [1] https://github.com/apache/incubator-iotdb/pull/314
>> 
>> 
>> Best,
>> --
>> Jialin Qiao
>> School of Software, Tsinghua University
>> 
>> 乔嘉林
>> 清华大学 软件学院
>> 
>>> -原始邮件-
>>> 发件人: "Justin Mclean" 
>>> 发送时间: 2019-07-27 07:56:31 (星期六)
>>> 收件人: dev@iotdb.apache.org
>>> 抄送:
>>> 主题: Re: [VOTE] Apache IoTDB (Incubating) 0.8.0 RC1
>>> 
>>> Hi,
>>> 
>>> +1 (binding)
>>> 
>>> I checked:
>>> - incubating in artefact names
>>> - DISCLAIMER exists
>>> - LICENSE is good
>>> - NOTICE may include a little too much text.  I not seen an export
>> control placed in NOTICE before, and you only need to include the bit from
>> Hive notice that are relevant [1]
>>> - All source files have ASF headers. However this file [4] incorrectly
>> has an ASF header. It’s 3rd party software and should not use the ASF
>> header.
>>> - No unexpected binary files
>>> - Can compile from source
>>> 
>>> One minor thing in license is this path
>> "./mvn/wrapper/MavenWrapperDownloader.java” should be
>> "./.mvn/wrapper/MavenWrapperDownloader.java"
>>> 
>>> In the README.md
>>> - Under "Installation from source code” it point the user to GitHub. The
>> user should be pointed to the projects download page, we don’t want to
>> encourage us

Re: Binary Release of IoTDB

[李天安]

Hi，
I prefer structure 4 to other structures. 
+ 1 for that.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 在 2019年7月19日，下午2:30，Jialin Qiao  写道：
> 
> Hi,
> 
> Nothing good comes easily :)
> 
> After reorganizing the tools, deduplicating scripts and removing grafana 
> related folders, structure 4 here comes:
> 
> (Structure 4):
> .
> ├──  LICENSE
> ├──  NOTICE
> ├──  RELEASE_NOTES
> │
> ├──  bin
> │  ├──  start-client.bat
> │  ├──  start-client.sh
> │  ├──  start-server.bat
> │  ├──  start-server.sh
> │  ├──  stop-server.bat
> │  └──  stop-server.sh
> │
> ├──  conf
> │  ├──  iotdb-engine.properties
> │  ├──  iotdb-env.bat
> │  ├──  iotdb-env.sh
> │  ├──  iotdb-sync-client.properties
> │  ├──  logback.xml
> │  └──  tsfile-format.properties
> │
> ├──  lib
> │  └──  *.jar
> │
> ├──  licenses
> │  └──  *.license
> │
> └──  tools
>   ├──  export-csv.bat
>   ├──  export-csv.sh
>   ├──  import-csv.bat
>   ├──  import-csv.sh
>   ├──  start-WalChecker.bat
>   ├──  start-WalChecker.sh
>   ├──  memory-tool.bat
>   ├──  memory-tool.sh
>   ├──  start-sync-client.bat
>   ├──  start-sync-client.sh
>   ├──  stop-sync-client.bat
>   └──  stop-sync-client.sh
> 
> 
> Thanks, 
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 
>> -原始邮件-
>> 发件人: "Xiangdong Huang" 
>> 发送时间: 2019-07-18 20:13:52 (星期四)
>> 收件人: dev@iotdb.apache.org
>> 抄送: 
>> 主题: Re: Binary Release of IoTDB
>> 
>> Hi all,
>> 
>> So we have made a consensus :).
>> 
>> Best,
>> ---
>> Xiangdong Huang
>> School of Software, Tsinghua University
>> 
>> 黄向东
>> 清华大学 软件学院
>> 
>> 
>> Julian Feinauer  于2019年7月18日周四 下午4:55写道：
>> 
>>> I also agree with structure 3.
>>> 
>>> Am 18.07.19, 10:39 schrieb "Xiangdong Huang" :
>>> 
>>>+1 for structure 3.
>>> 
>>>But it needs some work to fix current scripts (both the start-*.sh and
>>> the
>>>maven package plugin)...
>>> 
>>>By the way, I find that now when we are using `mvn package` with
>>>"maven-dependency-plugin:copy-dependencies" plugin, the jars whose
>>>lifecycle should be as "test" are copied into the lib folder... (e.g.,
>>>powermock-*.jar)
>>> 
>>>Best,
>>>---
>>>Xiangdong Huang
>>>School of Software, Tsinghua University
>>> 
>>> 黄向东
>>>清华大学 软件学院
>>> 
>>> 
>>>Jialin Qiao  于2019年7月18日周四 上午11:38写道：
>>> 
>>>> Hi,
>>>> 
>>>> I think structure 3 is better than 1 and 2.
>>>> 
>>>> However, what makes the bin folder a mess is some tools script.
>>>> 
>>>> Could we move walcheck and sync related scripts from "server" to a
>>> folder
>>>> named "tools", also csv import/export scripts from client to "tools"?
>>>> 
>>>> Besides, it's better to combine "run-client.bat" with
>>> "start-client.bat".
>>>> 
>>>> Best
>>>> --
>>>> Jialin Qiao
>>>> School of Software, Tsinghua University
>>>> 
>>>> 乔嘉林
>>>> 清华大学 软件学院
>>>> 
>>>>> -原始邮件-
>>>>> 发件人: "RUI, LEI" <1010953...@qq.com>
>>>>> 发送时间: 2019-07-18 11:24:50 (星期四)
>>>>> 收件人: dev 
>>>>> 抄送:
>>>>> 主题: 回复： Binary Release of IoTDB
>>>>> 
>>>>> Hi, I'm here to suggest another structure like this :)
>>>>> 
>>>>> 
>>>>> (Structure 3):
>>>>> .
>>>>> ├── LICENSE
>>>>> ├── NOTICE
>>>>> ├── changes.txt
>>>>> │
>>>>> ├── bin
>>>>> │   ├── client
>>>>> │   │   ├── export-csv.bat
>>>>> │   │   ├── export-csv.sh
>>>>> │   │   ├── import-csv.bat
>>>>> │   │   ├── import-csv.sh
>>>>> │   │   ├── run-client.bat
>>>>> │   │   ├── start-client.bat
>>>>> │   │   └── start-client.sh
>>>>> │   └── server
>>

Re: Memory estimation tool

[李天安]

Hi, Jialin:
Actually, I have written an docu to show how to use this tool, and it has been 
submitted to this pr. https://github.com/apache/incubator-iotdb/pull/256 
<https://github.com/apache/incubator-iotdb/pull/256>

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

> 在 2019年7月17日，下午4:44，Jialin Qiao  写道：
> 
> Hi Tianan,
> 
> Nice work, This is very useful! 
> 
> Could you add an example (command) about how to use this tool?
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 
>> -原始邮件-
>> 发件人: "李天安" 
>> 发送时间: 2019-07-17 14:57:51 (星期三)
>> 收件人: dev@iotdb.apache.org
>> 抄送: 
>> 主题: Memory estimation tool
>> 
>> Hi, 
>> 
>> 
>> Recently, I participated in the development of the dynamic parameter adapter 
>> module[https://github.com/apache/incubator-iotdb/pull/232], which can 
>> dynamically adjust system parameters according to load conditions. When we 
>> have the above module, I think we can develop a cooler thing: the memory 
>> estimation tool. When users provide the number of storage groups and 
>> timeseries according to the workload, the memory estimation tool can give 
>> the minimum memory for writing to meet the workload.
>> 
>> 
>> I think this tool is very useful and I have started to develop it 
>> [https://github.com/apache/incubator-iotdb/pull/256].
>> 
>> 
>> Anyone interested in this tool can discuss it here together.
>> 
>> 
>> Best Regards,
>> —
>> Tianan Li
>> School of Software, Tsinghua University
>> 
>> 
>> 

Memory estimation tool

[李天安]

Hi, 


Recently, I participated in the development of the dynamic parameter adapter 
module[https://github.com/apache/incubator-iotdb/pull/232], which can 
dynamically adjust system parameters according to load conditions. When we have 
the above module, I think we can develop a cooler thing: the memory estimation 
tool. When users provide the number of storage groups and timeseries according 
to the workload, the memory estimation tool can give the minimum memory for 
writing to meet the workload.


I think this tool is very useful and I have started to develop it 
[https://github.com/apache/incubator-iotdb/pull/256].


Anyone interested in this tool can discuss it here together.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

Re: Re: Re: Start the release train

[李天安]

Hi，

Recently I'm working on adjusting parameters dynamically according to different 
workloads. I have the following ideas：

There are three dynamically adjustable parameters in the IoTD: the number of 
MemTables, the size of per MemTable and the threshold size of TsFile.

1. the number of MemTables. This parameter represents the size of the MemTable 
available in the MemTable pool, which is closely related to the number of 
storage groups. In my design, this parameter is set to twice the number of 
storage groups plus a constant for backup. The reason why the ratio is set 
twice is that when the system is running stably, the speed of the flush 
operation is faster than that of data writing, so one is used for the Flush 
process and the other is used for data writing. Otherwise, the system should 
limit the speed of data writing to maintain stability.

2. the size of per MemTable. This parameter determines the threshold value for 
the MemTable in memory to be flushed into disk. When the system load increases, 
the parameter should be set smaller so that the data in memory can be flushed 
into disk as soon as possible.

3. the threshold size of TsFile. This parameter determines the speed of the 
tsfile seal, and then determines the maximum size of metadata information 
maintained in memory. When the system load increases, the parameter should be 
smaller to seal the file as soon as possible, release the memory occupied by 
the corresponding metadata information as soon as possible.

When the system adds or deletes storage groups, adds or deletes timeseries, it 
will evaluate the load of the system and adjust the above three parameters 
dynamically. When the system load is too large, it will reject the 
corresponding operation. At this time, the system needs to increase the memory 
to meet the demand.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University


> -原始邮件-
> 发件人: "Jialin Qiao" 
> 发送时间: 2019-07-08 21:27:18 (星期一)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Re: Start the release train
> 
> 
> Hi, 
> 
> Though the current version is stable, it introduces some new parameters 
> (e.g., the number of memtables in the pool) which users should set carefully.
> 
> I think we can make these parameters dynamically adapting the workload. It 
> does not need to change many codes (but need to design carefully). 
> 
> If it is no hurry, I'd like to finish this function before we start the 
> release process.
> 
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 
> > -原始邮件-
> > 发件人: "Julian Feinauer" 
> > 发送时间: 2019-07-08 21:21:42 (星期一)
> > 收件人: "dev@iotdb.apache.org" 
> > 抄送: 
> > 主题: Re: Start the release train
> > 
> > Just a short notice we should call it `develop` not `dev`.
> > At least my opiniton : )
> > 
> > Am 08.07.19, 15:19 schrieb "Xiangdong Huang" :
> > 
> > Hi all,
> > 
> > Does anyone has opinions for putting the releasing process at the top of
> > the agendas?
> > 
> > If there is no more opinions, I will start the first step: open a vote 
> > for
> > moving the `master` branch into the `dev`.
> > 
> > Best,
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> > 
> >  黄向东
> > 清华大学 软件学院
> > 
> 
> > 
> > 

Re: Re: Avoid long-tail insertion

[李天安]

Hi, 
   Indeed, in this pr of new storage engine, data sync module is not 
included. After the PR of the new storage engine is merged into the master, I 
will add the data sync module and submit a pr of that.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University 

> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2019-07-02 19:11:16 (星期二)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Avoid long-tail insertion
> 
> Hi,
> 
> I have reviewed the codes. The new structure is far better than previous
> version.
> 
> A question is that,  it seems that the merge process and and data sync
> module are not finished in your branch.
> 
> If we merge this PR into the master branch, we need to finish the two
> modules ASAP..
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院
> 
> 
> Jialin Qiao  于2019年7月2日周二 上午8:38写道：
> 
> > Hi,
> >
> > After several days' hard work, the new storage engine and corresponding
> > query engine is finally stable and behaves well...
> >
> > I have submitted a PR[1]. The new engine reduces the long-tail latency
> > from 50+s to under 5s. Detailed performance evaluation result will be
> > presented later.
> >
> > In this version, there are several new modules and strategies:
> >
> > Modules:
> >
> > (1) TVListPool: A cache of TVList, which is designed for reusing the
> > TVList and reducing the GC.
> > (2) PrimitiveArrayPool: A cache of arrays of primitive data type in java,
> > such as int[], long[], double[].
> > (3) TsFileResource: Each data file is denoted by a TsFileResource, which
> > is used for query.
> >
> > Strategies:
> >
> > (1) Asynchronously flushing memtable to disk.
> > (2) Asynchronously appending metadata and closing data file.
> >
> >
> > [1]https://github.com/apache/incubator-iotdb/pull/217
> >
> > Best,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> > > -原始邮件-
> > > 发件人: "Xiangdong Huang" 
> > > 发送时间: 2019-06-28 20:45:44 (星期五)
> > > 收件人: dev@iotdb.apache.org
> > > 抄送:
> > > 主题: Re: Re: Avoid long-tail insertion
> > >
> > > I see. The following sentence is very important to guarantee the
> > > correctness...
> > >
> > > >  At the same time, resubmitting UFP after the end of the task in each
> > > Flush thread ensures that all tasks can be executed.
> > >
> > > I think it should be highlight listed, rather than be drown in the last
> > > paragraph.
> > >
> > > Anyway, now I understand what you want to do.
> > >
> > > Best,
> > > ---
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > Xiangdong Huang  于2019年6月28日周五 下午8:41写道：
> > >
> > > > Hi Tianan,
> > > >
> > > > > the flush tasks in each UFP(There is a flushing queue in UFP) need
> > to be
> > > > executed sequentially.
> > > > > Flush thread polls the first UFP from UFP queue in FlushManager ->
> > polls
> > > > the first flush task in UFP -> completes the flush task -> set
> > > > ‘managedByFlushManager’  of the UFP to false.
> > > >
> > > > It indicates that there are more than one flush tasks in a UFP, but the
> > > > FlushManager  just spends one task from the UFP and then it mark the
> > UFP as
> > > > managedByFlushManager=false and poll it out of the queue? (So, when to
> > > > flush the rest tasks??)
> > > >
> > > > > (1) UFP is not managed by Flush Manager, i.e.'managedByFlushManager'
> > is
> > > > false
> > > >
> > > > If suppose the UFP has one flush task and enqueued the queue. Then the
> > UFP
> > > > has the second flush task. However, it can not be added into the queue
> > > > because managedByFlushManager == true. Using your above logic, you will
> > > > ignore the second flush task
> > > >
> > > >
> > > > > Flush Manager first determines whether UFP meets the criteria for
> > > > submission
> > > >
> > > > Do you want to say both the two criteria should be satisfied? (I

Re: Re: Avoid long-tail insertion

[李天安]

Hi，
 I'm also involved in the design of the new storage engine. Let me complement 
the new design of the flush task.  

 To improve system performance, we changed flush tasks from synchronous to 
asynchronous. We introduced a Flush Manager to manage all flush tasks. The 
tricky problem is that each Unsealed TsFile Processor (UFP) corresponds to a 
data file on a disk, so the flush tasks in each UFP(There is a flushing queue 
in UFP) need to be executed sequentially. However, flush tasks in different 
UFPs have no sequential requirements. How to design them to meet the above 
requirements?

 We introduce a UFP FIFO queue in Flush Manager, and add a boolean attribute 
‘managedByFlushManager’ to each UFP to indicate whether it is managed by Flush 
Manager. Flush Manager maintains a Flush thread pool to perform Flush tasks, so 
the lifecycle of a Flush task is
1. UFP are submitted to FlushManager，FlushManager add UFP to its queue and set 
‘managedByFlushManager’  of the UFP to true.
2. The Flush Pool in FlushManager start a flush thread to execute task.
3. Flush thread polls the first UFP from UFP queue in FlushManager -> polls the 
first flush task in UFP -> completes the flush task -> set 
‘managedByFlushManager’  of the UFP to false. 

There are two ways to submit a UFP to FlushManager:
1. UFP, whenever a MemTable reaches a certain size or forcibly triggers a flush 
task, it submits itself to Flush Manager (because the queue in Flush Manager is 
UFP). Flush Manager first determines whether UFP meets the criteria for 
submission:
(1) UFP is not managed by Flush Manager, i.e.'managedByFlushManager' is false
(2) The Flush task queue in UFP is not empty, that is, there are at least one 
flush task to be executed.

2. When the Flush thread completes the flush task, it sets 
‘managedByFlushManager’ to false and resubmits the UFP of the completed flush 
task to the FlushManager.

Through the above design, we can ensure that at the same time for each UFP, 
Flush Manager will only manage at most once and execute at most one flush task, 
while there is no restriction between different UFPs. At the same time, 
resubmitting UFP after the end of the task in each Flush thread ensures that 
all tasks can be executed. Therefore, we solve the above problem and the design 
meets the requirements of Flush Manager.

Best Regards,
-
Tianan Li
School of Software, Tsinghua University

> -原始邮件-
> 发件人: "Jialin Qiao" 
> 发送时间: 2019-06-27 11:27:24 (星期四)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Avoid long-tail insertion
> 
> Hi,
> 
> The new storage engine is designed to have the following components: 
> 
> (1) MemTable: A memory structure, which stores all inserted data in memory. 
> 
> (2) MemtablePool: Manages all memtables. All memtables are gotten from this 
> pool. The total number of memtables is fixed 
> in the system. Once the pool do not has available memtables, the 
> getMemtable() operation will wait or directly return.
> 
> (3) UnsealedTsFileProcessor (UFP): A writer for one data file. It always has 
> one working memtable that receives writes and a 
> list (flushing list) of memtables that for flush. Once the working memtable 
> reaches a threshold, it will be moved to the 
> flushing list and the working memtable is set null. When a new write arrives, 
> if the working memtable is null, UFP will 
> call getMemtable() of the MemtablePool to get one as the working memtable.
> 
> (4) StorageGroupProcessor (SGP): Each SGP is responsible for all writes and 
> reads in one storage group. It always has one 
> working UFP that receives write and a list (closing list) of UFPs that for 
> close. Once the file size of the working UFP reaches 
> a threshold, the UFP is moved to the closing list and the working UFP is set 
> null. When a new write arrives, if the working UFP 
> is null, a new UFP is generated as working UFP and receives write. 
> 
> (5) StorageGroupManager (SGM): A manager of all SGPs in IoTDB. It is only 
> responsible for routing read and write operations 
> to its corresponding SGP according to the deviceId of the operation.
> 
> (6) Flush thread: The flush thread poll a memtable from the flushing list in 
> UFP and flush a memtable to disk. After flushing, 
> the memtable is returned to the MemtablePool.
> 
> These are only the main components of the new storage engine. Some things may 
> be lost. It would be great if someone could 
> give some advices or supplementations.
> 
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 
> > -原始邮件-
> > 发件人: "Jialin Qiao" 
> > 发送时间: 2019-06-24 20:24:05 (星期一)
> > 收件人: dev@iotdb.apache.org
> > 抄送: 
> > 主题: Re: Re: Re: Avoid long-tail insertion
> > 
> > 
> > Yes, there are many changes. The branch I am working on is 
> > feature_async_close_tsfile. 
> > Anyone interested is welcome to join and discuss.
> > 
> > Best,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University

Re: Re: Synchronization between TsFile and Server

[李天安]

Hi Julian,
According to your issue, I wonder if the IoTDB you use is from the 
official website of IoTDB, which version is v0.7.0. Since the synchronization 
module was reconstructed once in March by myself, and the corresponding 
synchronization module with the document you have read is in the master branch 
of the GitHub repository of IoTDB. This will not synchronize the official 
version in IoTDB website until the new version of IoTDB is released, so you can 
update the IoTDB you use from the master branch of the GitHub of 
IoTDB(https://github.com/apache/incubator-iotdb ) and then use it. I'm sorry 
for the inconvenience. I hope this will help you. What's more, I'm looking 
forward to your feedback.

Best Regards,
-
Tianan Li
School of Software, Tsinghua University

> -原始邮件-
> 发件人: "Julian Feinauer" 
> 发送时间: 2019-06-24 14:58:47 (星期一)
> 收件人: "dev@iotdb.apache.org" 
> 抄送: 
> 主题: Re: Synchronization between TsFile and Server
> 
> Hi Tianan,
> 
> Thanks for the excellent writeup, it really helped me understand how it works 
> (and where to look).
> One minor question… in the current master branch the sh scripts are called 
> “start-postBackClient.sh” is tht correct (in your doc its called 
> “sync-client.sh”).
> 
> For the rest of my questions I’ll dig a bit through the code and come back if 
> I have more questions : )
> 
> Julian
> 
> On 2019/06/15 03:56:41, 李天安 
> mailto:l...@mails.tsinghua.edu.cn>> wrote:
> > Hi,>
> > The documentation for sync tool is in 
> > https://github.com/apache/incubator-iotdb/blob/add_sync_docu/docs/Documentation/UserGuideV0.7.0/7-Tools-Sync.md
> >  
> > <https://github.com/apache/incubator-iotdb/blob/add_sync_docu/docs/Documentation/UserGuideV0.7.0/7-Tools-Sync.md>.
> >  I’m looking forward to receiving your feedback to improve this tool.>
> >
> >
> > Best Regards,>
> > —>
> > Tianan Li>
> > School of Software, Tsinghua University>
> >
> > > 在 2019年6月13日，下午6:17，Julian Feinauer 
> > > mailto:j@pragmaticminds.de>> 写道：>
> > > >
> > > Hi Tianan,>
> > > >
> > > this would be great.>
> > > Thank you very much!>
> > > >
> > > Julian>
> > > >
> > > Am 13.06.19, 12:17 schrieb "李天安" 
> > > mailto:lt...@mails.tsinghua.edu.cn>>:>
> > > >
> > >Hi，>
> > > After consideration, I intend to write a manual for the sync tool 
> > > in these two days. According to the manual, you can use the tool and test 
> > > it. I think this is a solution and I hope this will help you.>
> > > >
> > >Best Regards,>
> > >—>
> > >Tianan Li>
> > >School of Software, Tsinghua University>
> > > >
> > >> 在 2019年6月13日，下午5:38，李天安 
> > >> mailto:lt...@mails.tsinghua.edu.cn>> 写道：>
> > >> >
> > >> Hi，>
> > >>   You’re right. The essence of sync tool is to deal with loading 
> > >> external tsfile in a running IoTDB service, which takes different 
> > >> strategies according to different tsfile types. We will consider your 
> > >> needs and look forward to cooperating with you.>
> > >> >
> > >> Best Regards,>
> > >> —>
> > >> Tianan Li>
> > >> School of Software, Tsinghua University>
> > >> >
> > >>> 在 2019年6月13日，下午5:28，Julian Feinauer 
> > >>> mailto:j@pragmaticminds.de>> 写道：>
> > >>> >
> > >>> okay, this is also the impression I have __>
> > >>> But the general idea still is true, that there will be a way to simply 
> > >>> ship a "local file" over to the server and he does some kind of batch 
> > >>> import for me, right?>
> > >>> If we start to intensivy our tests I can of course help with the 
> > >>> development, if you point me to the right directions.>
> > >> >
> > >> >
> > > >
> > > >
> > > >
> >
> >

Re: Synchronization between TsFile and Server

[李天安]

Hi,
The documentation for sync tool is in 
https://github.com/apache/incubator-iotdb/blob/add_sync_docu/docs/Documentation/UserGuideV0.7.0/7-Tools-Sync.md
 
<https://github.com/apache/incubator-iotdb/blob/add_sync_docu/docs/Documentation/UserGuideV0.7.0/7-Tools-Sync.md>.
 I’m looking forward to receiving your feedback to improve this tool.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年6月13日，下午6:17，Julian Feinauer  写道：
> 
> Hi Tianan,
> 
> this would be great.
> Thank you very much!
> 
> Julian
> 
> Am 13.06.19, 12:17 schrieb "李天安" :
> 
>Hi，
>   After consideration, I intend to write a manual for the sync tool in 
> these two days. According to the manual, you can use the tool and test it. I 
> think this is a solution and I hope this will help you.
> 
>Best Regards,
>—
>Tianan Li
>School of Software, Tsinghua University
> 
>> 在 2019年6月13日，下午5:38，李天安  写道：
>> 
>> Hi，
>>  You’re right. The essence of sync tool is to deal with loading external 
>> tsfile in a running IoTDB service, which takes different strategies 
>> according to different tsfile types. We will consider your needs and look 
>> forward to cooperating with you.
>> 
>> Best Regards,
>> —
>> Tianan Li
>> School of Software, Tsinghua University
>> 
>>> 在 2019年6月13日，下午5:28，Julian Feinauer  写道：
>>> 
>>> okay, this is also the impression I have __
>>> But the general idea still is true, that there will be a way to simply ship 
>>> a "local file" over to the server and he does some kind of batch import for 
>>> me, right?
>>> If we start to intensivy our tests I can of course help with the 
>>> development, if you point me to the right directions.
>> 
>> 
> 
> 
> 

Re: Synchronization between TsFile and Server

[李天安]

Hi，
After consideration, I intend to write a manual for the sync tool in 
these two days. According to the manual, you can use the tool and test it. I 
think this is a solution and I hope this will help you.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年6月13日，下午5:38，李天安  写道：
> 
> Hi，
>   You’re right. The essence of sync tool is to deal with loading external 
> tsfile in a running IoTDB service, which takes different strategies according 
> to different tsfile types. We will consider your needs and look forward to 
> cooperating with you.
> 
> Best Regards,
> —
> Tianan Li
> School of Software, Tsinghua University
> 
>> 在 2019年6月13日，下午5:28，Julian Feinauer  写道：
>> 
>> okay, this is also the impression I have __
>> But the general idea still is true, that there will be a way to simply ship 
>> a "local file" over to the server and he does some kind of batch import for 
>> me, right?
>> If we start to intensivy our tests I can of course help with the 
>> development, if you point me to the right directions.
> 
> 

Re: Synchronization between TsFile and Server

[李天安]

Hi，
You’re right. The essence of sync tool is to deal with loading external 
tsfile in a running IoTDB service, which takes different strategies according 
to different tsfile types. We will consider your needs and look forward to 
cooperating with you.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年6月13日，下午5:28，Julian Feinauer  写道：
> 
> okay, this is also the impression I have __
> But the general idea still is true, that there will be a way to simply ship a 
> "local file" over to the server and he does some kind of batch import for me, 
> right?
> If we start to intensivy our tests I can of course help with the development, 
> if you point me to the right directions.

Re: Synchronization between TsFile and Server

[李天安]

Hi，
Thanks for your attention to IoTDB again. Well,  currently sync tools 
have not yet been fully tested and are temporarily instable,  so there is no 
open documentation. This tool plays an important role in distributed version of 
IoTDB, so it will be perfected with distributed development and will be 
published with distributed manual.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年6月13日，下午4:39，Julian Feinauer  写道：
> 
> Hi lovely IoTDB-community,
> 
> I will soon have some time to play a bit more again with IoTDB.
> And one thing which is crucial for my use cases is the file based storage.
> But, as indicated in some docs there is the possibility (or there was, as 
> least) to sync local TsFiles (e.g. on an edge device) with a running DB 
> Server.
> Can someone help me point at the right place or comment on the current state 
> of that?
> 
> Thanks!
> Julian

Re: [jira] [Created] (IOTDB-101) This is an example of SQL in user manual. Why use sum function and mean function to process Boolean data???

[李天安]

Hi，
Thanks for your report, it’s our carelessness when writing user doc. I 
have pull a pr to fix it. Pr link is 
:https://github.com/apache/incubator-iotdb/pull/182 
 

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年5月24日，下午5:31，zhaoxiuting (JIRA)  写道：
> 
> zhaoxiuting created IOTDB-101:
> -
> 
> Summary: This is an example of SQL in user manual. Why use sum 
> function and mean function to process Boolean data???
> Key: IOTDB-101
> URL: https://issues.apache.org/jira/browse/IOTDB-101
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: zhaoxiuting
> Attachments: image-2019-05-24-17-28-23-691.png
> 
> Step1：The first example of SQL in user manual is SELECT SUM(status), 
> SUM(temperature) FROM root.ln.wf01.wt01 WHERE root.ln.wf01.wt01.temperature < 
> 24.
> 
> The second  example of SQL in user manual is SELECT MEAN(status), 
> MEAN(temperature) FROM root.ln.wf01.wt01 WHERE root.ln.wf01.wt01.temperature 
> < 24
> 
> Step2:
> 
> !image-2019-05-24-17-28-23-691.png!
> 
> Exception:It is suggested to modify the example SQL statement in the user 
> manual.
> 
>  
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

Re: [jira] [Created] (IOTDB-100) Feedback after executing SQL statement, error

[李天安]

Hi，
I have pulled a pr to fix  this problem. The link is 
https://github.com/apache/incubator-iotdb/pull/181 


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年5月24日，下午5:23，zhaoxiuting (JIRA)  写道：
> 
> zhaoxiuting created IOTDB-100:
> -
> 
> Summary: Feedback after executing SQL statement, error
> Key: IOTDB-100
> URL: https://issues.apache.org/jira/browse/IOTDB-100
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: zhaoxiuting
> Attachments: image-2019-05-24-17-18-16-367.png
> 
> Step1：Execute SQL statement  on IoTDB.The SQL statement is SELECT 
> SUM(status), SUM(temperature) FROM root.ln.wf01.wt01 WHERE 
> root.ln.wf01.wt01.temperature < 24
> 
> Step2:Feedback after executing SQL statement, error
> 
> !image-2019-05-24-17-18-16-367.png!
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

Re: [jira] [Created] (IOTDB-100) Feedback after executing SQL statement, error

[李天安]

Hi，
Thanks for your report, I have identified the problem and started to solve it.

Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2019年5月24日，下午5:23，zhaoxiuting (JIRA)  写道：
> 
> zhaoxiuting created IOTDB-100:
> -
> 
> Summary: Feedback after executing SQL statement, error
> Key: IOTDB-100
> URL: https://issues.apache.org/jira/browse/IOTDB-100
> Project: Apache IoTDB
>  Issue Type: Bug
>Reporter: zhaoxiuting
> Attachments: image-2019-05-24-17-18-16-367.png
> 
> Step1：Execute SQL statement  on IoTDB.The SQL statement is SELECT 
> SUM(status), SUM(temperature) FROM root.ln.wf01.wt01 WHERE 
> root.ln.wf01.wt01.temperature < 24
> 
> Step2:Feedback after executing SQL statement, error
> 
> !image-2019-05-24-17-18-16-367.png!
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

Re: [VOTE] vote for IoTDB logo

[李天安]

Hi， 
+1 for the logo

Thanks.
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院

Re: Integrate Calcite in IoTDB

[李天安]

Hi Julian,
Nice to hear from you.
I’m very glad to hear that you’d like to help us with the integration.
We want to proceed the integration  as follows: 
Firstly, we will learn how you integrate calcite with other products from 
official website.
After that, we discuss the structure of Calcite Integration with you.

Tianan Li

> 在 2019年2月26日，上午4:39，Julian Feinauer  写道：
> 
> Hi Tianan,
> 
> nice to hear from you!
> I just recently recognized how far you have come and how less time I commited 
> to IoTDB.
> I would really love to participate in this effort and help you with the 
> Calcite Adapter.
> What do you think is the best way to proceed?
> I can tell you something about the structure of Calcite Integratioons (or 
> make a walkthrough through the Plc4x integration I’ve done).
> 
> What do the others think?
> 
> Julian
> 
> Von: 李天安 
> Datum: Montag, 25. Februar 2019 um 17:01
> An: "jfeina...@apache.org" 
> Betreff: Fwd: Integrate Calcite in IoTDB
> 
> 
> 
> 
> 下面是被转发的邮件：
> 
> 发件人: 李天安 mailto:lt...@mails.tsinghua.edu.cn>>
> 主题: Integrate Calcite in IoTDB
> 日期: 2019年2月25日 GMT+8 下午2:56:56
> 收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org>
> 回复－收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org>
> 
> Hi Julian,
> 
> My name is Tianan Li, one of the initial committers of IoTDB project. Over 
> the past few months, we have optimized the structure and content of the IoTDB 
> project and basically completed it. Now we hope to integrate with Calcite. I 
> wonder if you would like to instruct us to complete the Calcite-IoTDB Adapter.
> 
> Best
> Tianan Li
> 
> 
> 在 2018年12月3日，上午5:25，Julian Feinauer 
> mailto:jfeina...@apache.org>> 写道：
> 
> Hi,
> 
> my name is Julian Feinauer and I come from Apache Calcite, Edgent and PLC4X. 
> My interest is mostly in time series processing at the edge. In my daytime 
> job I work for two companies called pragmatic minds and pragmatic industries 
> which we founded nearly 3 years ago in South Germany.
> I love the idea behind iotDB as it is realy the materialization of many of my 
> thoughts about what we really need for efficient storage of ts data.
> I would love to get my hands dirty on the query and filter api and perhaps 
> even write a small Calcite Adapter, when I find some time in the next weeks.
> 
> We have many "IoT" projects with practical use cases in the industry so I'll 
> try to bring my perspective in as best as possible.
> 
> Best
> Julian
> 
> 

Integrate Calcite in IoTDB

[李天安]

Hi Julian,

My name is Tianan Li, one of the initial committers of IoTDB project. Over the 
past few months, we have optimized the structure and content of the IoTDB 
project and basically completed it. Now we hope to integrate with Calcite. I 
wonder if you would like to instruct us to complete the Calcite-IoTDB Adapter. 

Best
Tianan Li

> 在 2018年12月3日，上午5:25，Julian Feinauer  写道：
> 
> Hi,
> 
> my name is Julian Feinauer and I come from Apache Calcite, Edgent and PLC4X. 
> My interest is mostly in time series processing at the edge. In my daytime 
> job I work for two companies called pragmatic minds and pragmatic industries 
> which we founded nearly 3 years ago in South Germany.
> I love the idea behind iotDB as it is realy the materialization of many of my 
> thoughts about what we really need for efficient storage of ts data.
> I would love to get my hands dirty on the query and filter api and perhaps 
> even write a small Calcite Adapter, when I find some time in the next weeks.
> 
> We have many "IoT" projects with practical use cases in the industry so I'll 
> try to bring my perspective in as best as possible.
> 
> Best
> Julian

Re: Welcome to the IoTDB project

[李天安]

Hello, all,


I'm Tianan Li, it’s my first year of being a master student in School of 
Software at Tsinghua University. I used to take part in the development of 
postback model, which is used to transmit data from end to cloud. 

It's my great honor to participate in such an awesome project and I hope to 
make more contributions for IoTDB.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

> 在 2018年11月30日，上午5:43，Justin Mclean  写道：
> 
> Hi,
> 
> I’m Justin and I volunteered to be a mentor for this project and help out 
> where needed. So if you need any help or have any questions, in particular 
> about how the ASF works, please ask on this list.
> 
> I’ve been involved in the ASF for a while now and mentor and help out on a 
> few projects, several of them in the IoT space. I’m currently the V.P of the 
> Apache incubator and of Apache Mynewt. I’ve been involved in number of 
> commercial IoT projects (and scada systems) and I’m a electronics hobbyist. I 
> run the IoT meetup in Sydney Australia, speak at conferences and run the 
> occasional workshop. I’m a freelance developer and teacher and currently 
> split my time between the two, teaching at an online college / being a casual 
> academic at university and working as a developer or consultant on several 
> projects.
> 
> Thanks,
> Justin