Re: Inaccurate Statistics when write batch

[Jialin Qiao]

Hi,

I have fixed this issue in PR#593

https://github.com/apache/incubator-iotdb/pull/593

This PR is submitted to rel/0.9 and could be released in 0.9.1.

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

-原始邮件-
发件人: "Jialin Qiao (Jira)" 
发送时间: 2019-11-28 14:16:00 (星期四)
收件人: dev@iotdb.apache.org
抄送:
主题: [jira] [Created] (IOTDB-324) Inaccurate Statistics when write batch

Jialin Qiao created IOTDB-324:
-

Summary: Inaccurate Statistics when write batch
Key: IOTDB-324
URL: https://issues.apache.org/jira/browse/IOTDB-324
Project: Apache IoTDB
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Jialin Qiao


When writing a batch in ChunkWriterImpl, the statistics are updated without
considering the batchSize.

When writing a 100 size array, batch size is 90, the last 10 points is also
calculated in the Statistics.

In ChunkWirterImpl:

```
public void write(long[] timestamps, double[] values, int batchSize) {
this.maxTimestamp = timestamps[batchSize - 1];
valueCountInOnePage += batchSize;
if (minTimestamp == Long.MIN_VALUE) {
minTimestamp = timestamps[0];
}
pageWriter.write(timestamps, values, batchSize);
pageStatistics.updateStats(values);   ** this line is not
correct
checkPageSizeAndMayOpenANewPage();
}

```





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IOTDB-325) Refactor Statistics

[Jialin Qiao (Jira)]

Jialin Qiao created IOTDB-325:
-

 Summary: Refactor Statistics
 Key: IOTDB-325
 URL: https://issues.apache.org/jira/browse/IOTDB-325
 Project: Apache IoTDB
  Issue Type: Improvement
Reporter: Jialin Qiao


There are Statistics and StatisticsForFilter in TsFile module, which is in 
chaos.

Also, the ByteBuffer[] in Statistics is useless, as the subclass could 
serialize and deserialize the fields independently. There is no need to 
construct a ByteBuffer[] in the parent class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (IOTDB-279) Merge TsDigest into Statistics

[Jialin Qiao (Jira)]

 [ 
https://issues.apache.org/jira/browse/IOTDB-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jialin Qiao closed IOTDB-279.
-
Resolution: Fixed

> Merge TsDigest into Statistics
> --
>
> Key: IOTDB-279
> URL: https://issues.apache.org/jira/browse/IOTDB-279
> Project: Apache IoTDB
>  Issue Type: Improvement
>Reporter: Jialin Qiao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> As I observe, TsDigest is only the ByteBuffer format of Statistics, so why 
> not merge them, which could make the code clear.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IOTDB-324) Inaccurate Statistics when write batch

[Jialin Qiao (Jira)]

Jialin Qiao created IOTDB-324:
-

 Summary: Inaccurate Statistics when write batch
 Key: IOTDB-324
 URL: https://issues.apache.org/jira/browse/IOTDB-324
 Project: Apache IoTDB
  Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Jialin Qiao


When writing a batch in ChunkWriterImpl, the statistics are updated without 
considering the batchSize.

When writing a 100 size array, batch size is 90, the last 10 points is also 
calculated in the Statistics.

In ChunkWirterImpl:

```
public void write(long[] timestamps, double[] values, int batchSize) {
 this.maxTimestamp = timestamps[batchSize - 1];
 valueCountInOnePage += batchSize;
 if (minTimestamp == Long.MIN_VALUE) {
 minTimestamp = timestamps[0];
 }
 pageWriter.write(timestamps, values, batchSize);
 pageStatistics.updateStats(values);   ** this line is not 
correct
 checkPageSizeAndMayOpenANewPage();
}

```

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[IOTDB-311] Generate anything in test to the target directory

[刘大伟]

Hi all,

I’m working on this issue.

So far, I've modified all of the test-generated files into the target 
directory, and added the TestConstant class for each project test, so that 
anyone who writes unit tests later can use it uniformly to manage the 
parameters in the test.

In addition, when I checked the server unit tests, I found that the indexDir 
field in the configuration file was nowhere to be used, but the test directory 
was generated all the time. 
So I think it's worth removing it from the configuration file and from the unit 
tests.

I have submitted this pr, I hope someone can review it for me.

Pr:
https://github.com/apache/incubator-iotdb/pull/578 


Issue:

https://issues.apache.org/jira/browse/IOTDB-311 



> 在 2019年11月21日，下午7:02，Jialin Qiao (Jira)  写道：
> 
> Jialin Qiao created IOTDB-311:
> -
> 
> Summary: Generate anything in test to the target directory.
> Key: IOTDB-311
> URL: https://issues.apache.org/jira/browse/IOTDB-311
> Project: Apache IoTDB
>  Issue Type: Improvement
>Reporter: Jialin Qiao
> 
> 
> We'd better generate anything in UT/IT to the target directory to avoid 
> including these files in source release.
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
> 

Re: [VOTE] Release Apache IoTDB 0.9.0

[Kevin A. McGrail]

+1 binding

On Wed, Nov 27, 2019, 07:10 Gosling Von  wrote:

> Hi,
>
> +1 (binding)
>
> I’v checked:
> - LICENSE and NOTICE files are good.
> - No binary file in the source kit.
> - DISCLAIMER is included.
> - All files have license headers if necessary.
>
>
>
> Best Regards,
> Von Gosling
>
> > On Nov 27, 2019, at 3:48 PM, Jialin Qiao  wrote:
> >
> > Hello all,
> >
> > This is a call for vote to release Apache IoTDB (Incubating) version
> 0.9.0, a
> > major version with many exciting features.
> >
> > The Apache IoTDB community has voted on and approved a proposal to
> > release Apache
> > IoTDB (Incubating) version 0.9.0.
> >
> > We now kindly request the Incubator PMC members review and vote on
> > this incubator
> > release.
> >
> > Apache IoTDB (incubating) (Database for Internet of Things) is an
> > integrated data management engine designed for time series data. It
> > provides users with services for data collection, storage and analysis.
> >
> > IoTDB community vote and result thread:
> > Result:
> >
> https://lists.apache.org/thread.html/ebf6b2b4f435868a14e89fa3a767f99a0db71f4e22a7acc9a7b5b22b@%3Cdev.iotdb.apache.org%3E
> > Vote:
> >
> https://lists.apache.org/thread.html/ddcdc40e74e7592dd9efe190ef96cc62cf0b07d9e0aa4e7f0d6e7577@%3Cdev.iotdb.apache.org%3E
> >
> > The release candidates (RC5):
> > https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc5
> >
> > Git tag for the release (RC5):
> > https://github.com/apache/incubator-iotdb/releases/tag/release%2F0.9.0
> > 
> >
> > Hash for the release tag:
> > 257df94d57c29c628b1ab0d6a2fdd20ba7df9ca4
> >
> > Release Notes:
> >
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc5/RELEASE_NOTES.md
> >
> > The artifacts have been signed with Key: 0FC7F131CAA00430, which can
> > be found in the keys file:
> > https://dist.apache.org/repos/dist/dev/incubator/iotdb/KEYS
> >
> > Look at here for how to verify this release candidate:
> >
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> >
> > The vote will be open for at least 72 hours.
> >
> > A minimum of 3 IPMC +1 votes and more binding +1 than binding -1 are
> > required to pass the vote.
> >
> > Please vote accordingly:
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> > Jialin Qiao
> > Apache IoTDB
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache IoTDB 0.9.0

[Gosling Von]

Hi,

+1 (binding)

I’v checked:
- LICENSE and NOTICE files are good.
- No binary file in the source kit.
- DISCLAIMER is included.
- All files have license headers if necessary.



Best Regards,
Von Gosling

> On Nov 27, 2019, at 3:48 PM, Jialin Qiao  wrote:
> 
> Hello all,
> 
> This is a call for vote to release Apache IoTDB (Incubating) version 0.9.0, a
> major version with many exciting features.
> 
> The Apache IoTDB community has voted on and approved a proposal to
> release Apache
> IoTDB (Incubating) version 0.9.0.
> 
> We now kindly request the Incubator PMC members review and vote on
> this incubator
> release.
> 
> Apache IoTDB (incubating) (Database for Internet of Things) is an
> integrated data management engine designed for time series data. It
> provides users with services for data collection, storage and analysis.
> 
> IoTDB community vote and result thread:
> Result:
> https://lists.apache.org/thread.html/ebf6b2b4f435868a14e89fa3a767f99a0db71f4e22a7acc9a7b5b22b@%3Cdev.iotdb.apache.org%3E
> Vote:
> https://lists.apache.org/thread.html/ddcdc40e74e7592dd9efe190ef96cc62cf0b07d9e0aa4e7f0d6e7577@%3Cdev.iotdb.apache.org%3E
> 
> The release candidates (RC5):
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc5
> 
> Git tag for the release (RC5):
> https://github.com/apache/incubator-iotdb/releases/tag/release%2F0.9.0
> 
> 
> Hash for the release tag:
> 257df94d57c29c628b1ab0d6a2fdd20ba7df9ca4
> 
> Release Notes:
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc5/RELEASE_NOTES.md
> 
> The artifacts have been signed with Key: 0FC7F131CAA00430, which can
> be found in the keys file:
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/KEYS
> 
> Look at here for how to verify this release candidate:
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> 
> The vote will be open for at least 72 hours.
> 
> A minimum of 3 IPMC +1 votes and more binding +1 than binding -1 are
> required to pass the vote.
> 
> Please vote accordingly:
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Jialin Qiao
> Apache IoTDB

Re: [VOTE] Release Apache IoTDB 0.9.0

[Willem Jiang]

+1 (binding)

I checked below items:
Download links are valid.
There are incubating in the release kit.
Checksums and PGP signatures are valid.
DISCLAIMER is included.
LICENSE and NOTICE files are good.
No binary file in the source kit.
All files have license headers if necessary.


Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem


On Wed, Nov 27, 2019 at 3:48 PM Jialin Qiao  wrote:
>
> Hello all,
>
> This is a call for vote to release Apache IoTDB (Incubating) version 0.9.0, a
> major version with many exciting features.
>
> The Apache IoTDB community has voted on and approved a proposal to
> release Apache
> IoTDB (Incubating) version 0.9.0.
>
> We now kindly request the Incubator PMC members review and vote on
> this incubator
> release.
>
> Apache IoTDB (incubating) (Database for Internet of Things) is an
> integrated data management engine designed for time series data. It
> provides users with services for data collection, storage and analysis.
>
> IoTDB community vote and result thread:
> Result:
> https://lists.apache.org/thread.html/ebf6b2b4f435868a14e89fa3a767f99a0db71f4e22a7acc9a7b5b22b@%3Cdev.iotdb.apache.org%3E
> Vote:
> https://lists.apache.org/thread.html/ddcdc40e74e7592dd9efe190ef96cc62cf0b07d9e0aa4e7f0d6e7577@%3Cdev.iotdb.apache.org%3E
>
> The release candidates (RC5):
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc5
>
> Git tag for the release (RC5):
> https://github.com/apache/incubator-iotdb/releases/tag/release%2F0.9.0
> 
>
> Hash for the release tag:
> 257df94d57c29c628b1ab0d6a2fdd20ba7df9ca4
>
> Release Notes:
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.9.0/rc5/RELEASE_NOTES.md
>
> The artifacts have been signed with Key: 0FC7F131CAA00430, which can
> be found in the keys file:
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/KEYS
>
> Look at here for how to verify this release candidate:
> https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
>
> The vote will be open for at least 72 hours.
>
> A minimum of 3 IPMC +1 votes and more binding +1 than binding -1 are
> required to pass the vote.
>
> Please vote accordingly:
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> Jialin Qiao
> Apache IoTDB