subject:"\\\\\\\[GitHub\\\\\\\] \\\\\\\[iotdb\\\\\\\] qiaojialin commented on a change in pull request #2087\\\\\\\: \\\\\\\[IOTDB\\\\\\\-890\\\\\\\] SDT implementation"

[GitHub] [iotdb] qiaojialin commented on a change in pull request #2087: [IOTDB-890] SDT implementation

2020-11-26 Thread GitBox



qiaojialin commented on a change in pull request #2087:
URL: https://github.com/apache/iotdb/pull/2087#discussion_r530889069



##
File path: 
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/SDTEncoder.java
##
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.encoder;
+
+public class SDTEncoder {
+
+  /**
+   * the last read time and value
+   * if upperDoor >= lowerDoor meaning out of compDeviation range, will store 
lastReadPair
+   */
+  private long lastReadTimestamp;
+  private long lastReadLong;
+  private double lastReadDouble;
+  private int lastReadInt;
+  private float lastReadFloat;
+
+  /**
+   * the last stored time and vlaue
+   * we compare current point against lastStoredPair
+   */
+  private long lastStoredTimestamp;
+  private long lastStoredLong;
+  private double lastStoredDouble;
+  private int lastStoredInt;
+  private float lastStoredFloat;
+
+  /**
+   * the maximum curUpperSlope between the lastStoredPoint to the current point
+   * upperDoor can only open up
+   */
+  private double upperDoor;
+
+  /**
+   * the minimum curLowerSlope between the lastStoredPoint to the current point
+   * lowerDoor can only open downard
+   */
+  private double lowerDoor;
+
+  /**
+   * the maximum absolute difference the user set
+   * if the data's value is within compDeviation, it will be compressed and 
discarded
+   * after compression, it will only store out of range  to 
form the trend
+   */
+  private double compDeviation;
+
+  /**
+   * the minimum time distance between two stored data points
+   * if current point time to the last stored point time distance <= compMin,
+   * current point will NOT be stored regardless of compression deviation
+   */
+  private double compMin;
+
+  /**
+   * the maximum time distance between two stored data points
+   * if current point time to the last stored point time distance >= compMax,
+   * current point will be stored regardless of compression deviation
+   */
+  private double compMax;
+
+  /**
+   * isFirstValue is true when the encoder takes the first point or reset() 
when cur point's
+   * distance to the last stored point's distance exceeds compMax
+   */
+  private boolean isFirstValue;
+
+  public SDTEncoder() {
+upperDoor = Integer.MIN_VALUE;
+lowerDoor = Integer.MAX_VALUE;
+compDeviation = -1;
+compMin = Integer.MIN_VALUE;
+compMax = Integer.MAX_VALUE;
+isFirstValue = true;
+  }
+
+  public boolean encodeFloat(long time, float value) {
+// store the first time and value pair
+if (isFirstValue(time, value)) {
+  return true;
+}
+
+// if current point to the last stored point's time distance is within 
compMin,
+// will not check two doors nor store any point within the compMin time 
range
+if (time - lastStoredTimestamp <= compMin) {
+  return false;
+}
+
+// if current point to the last stored point's time distance is larger 
than compMax,
+// will reset two doors, and store current point;
+if (time - lastStoredTimestamp >= compMax) {
+  reset(time, value);
+  return true;
+}
+
+double curUpperSlope = (value - lastStoredFloat - compDeviation) / (time - 
lastStoredTimestamp);
+if (curUpperSlope > upperDoor) {
+  upperDoor = curUpperSlope;
+  if (upperDoor > lowerDoor) {
+// slope between curr point and last read point
+double slope = (value - lastReadFloat) / (time - lastReadTimestamp);
+// start point of the next segment
+long timestamp = (long) ((lastStoredFloat + compDeviation - 
lastReadFloat + slope * lastReadTimestamp -
+lowerDoor * lastStoredTimestamp) / (slope - lowerDoor));
+lastStoredFloat = (float) (lastStoredFloat + compDeviation + lowerDoor 
* (timestamp - lastStoredTimestamp)
+- compDeviation / 2);
+lastStoredTimestamp = timestamp;
+// recalculate upperDoor and lowerDoor
+upperDoor = (value - lastStoredFloat - compDeviation) / (time - 
lastStoredTimestamp);
+lowerDoor = (value - lastStoredFloat + compDeviation) / (time - 
lastStoredTimestamp);
+// update last read

[GitHub] [iotdb] qiaojialin commented on a change in pull request #2087: [IOTDB-890] SDT implementation

2020-11-26 Thread GitBox



qiaojialin commented on a change in pull request #2087:
URL: https://github.com/apache/iotdb/pull/2087#discussion_r530889069



##
File path: 
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/SDTEncoder.java
##
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.encoder;
+
+public class SDTEncoder {
+
+  /**
+   * the last read time and value
+   * if upperDoor >= lowerDoor meaning out of compDeviation range, will store 
lastReadPair
+   */
+  private long lastReadTimestamp;
+  private long lastReadLong;
+  private double lastReadDouble;
+  private int lastReadInt;
+  private float lastReadFloat;
+
+  /**
+   * the last stored time and vlaue
+   * we compare current point against lastStoredPair
+   */
+  private long lastStoredTimestamp;
+  private long lastStoredLong;
+  private double lastStoredDouble;
+  private int lastStoredInt;
+  private float lastStoredFloat;
+
+  /**
+   * the maximum curUpperSlope between the lastStoredPoint to the current point
+   * upperDoor can only open up
+   */
+  private double upperDoor;
+
+  /**
+   * the minimum curLowerSlope between the lastStoredPoint to the current point
+   * lowerDoor can only open downard
+   */
+  private double lowerDoor;
+
+  /**
+   * the maximum absolute difference the user set
+   * if the data's value is within compDeviation, it will be compressed and 
discarded
+   * after compression, it will only store out of range  to 
form the trend
+   */
+  private double compDeviation;
+
+  /**
+   * the minimum time distance between two stored data points
+   * if current point time to the last stored point time distance <= compMin,
+   * current point will NOT be stored regardless of compression deviation
+   */
+  private double compMin;
+
+  /**
+   * the maximum time distance between two stored data points
+   * if current point time to the last stored point time distance >= compMax,
+   * current point will be stored regardless of compression deviation
+   */
+  private double compMax;
+
+  /**
+   * isFirstValue is true when the encoder takes the first point or reset() 
when cur point's
+   * distance to the last stored point's distance exceeds compMax
+   */
+  private boolean isFirstValue;
+
+  public SDTEncoder() {
+upperDoor = Integer.MIN_VALUE;
+lowerDoor = Integer.MAX_VALUE;
+compDeviation = -1;
+compMin = Integer.MIN_VALUE;
+compMax = Integer.MAX_VALUE;
+isFirstValue = true;
+  }
+
+  public boolean encodeFloat(long time, float value) {
+// store the first time and value pair
+if (isFirstValue(time, value)) {
+  return true;
+}
+
+// if current point to the last stored point's time distance is within 
compMin,
+// will not check two doors nor store any point within the compMin time 
range
+if (time - lastStoredTimestamp <= compMin) {
+  return false;
+}
+
+// if current point to the last stored point's time distance is larger 
than compMax,
+// will reset two doors, and store current point;
+if (time - lastStoredTimestamp >= compMax) {
+  reset(time, value);
+  return true;
+}
+
+double curUpperSlope = (value - lastStoredFloat - compDeviation) / (time - 
lastStoredTimestamp);
+if (curUpperSlope > upperDoor) {
+  upperDoor = curUpperSlope;
+  if (upperDoor > lowerDoor) {
+// slope between curr point and last read point
+double slope = (value - lastReadFloat) / (time - lastReadTimestamp);
+// start point of the next segment
+long timestamp = (long) ((lastStoredFloat + compDeviation - 
lastReadFloat + slope * lastReadTimestamp -
+lowerDoor * lastStoredTimestamp) / (slope - lowerDoor));
+lastStoredFloat = (float) (lastStoredFloat + compDeviation + lowerDoor 
* (timestamp - lastStoredTimestamp)
+- compDeviation / 2);
+lastStoredTimestamp = timestamp;
+// recalculate upperDoor and lowerDoor
+upperDoor = (value - lastStoredFloat - compDeviation) / (time - 
lastStoredTimestamp);
+lowerDoor = (value - lastStoredFloat + compDeviation) / (time - 
lastStoredTimestamp);
+// update last read

[GitHub] [iotdb] qiaojialin commented on a change in pull request #2087: [IOTDB-890] SDT implementation

2020-11-24 Thread GitBox



qiaojialin commented on a change in pull request #2087:
URL: https://github.com/apache/iotdb/pull/2087#discussion_r530055733



##
File path: example/session/src/main/java/org/apache/iotdb/SessionExample.java
##
@@ -48,28 +48,41 @@
 
   public static void main(String[] args)
   throws IoTDBConnectionException, StatementExecutionException {
-session = new Session("127.0.0.1", 6667, "root", "root");
+session = new Session("172.16.48.4", 58890, "root", "root");
 session.open(false);
 
+   // session.executeNonQueryStatement("delete storage group root");
+/*
 try {
   session.setStorageGroup("root.sg1");
 } catch (StatementExecutionException e) {
   if (e.getStatusCode() != 
TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode())
 throw e;
 }
 
-createTimeseries();

Review comment:
   could the SDT be set using Session nosql API?

##
File path: example/session/src/main/java/org/apache/iotdb/SessionExample.java
##
@@ -48,28 +48,41 @@
 
   public static void main(String[] args)
   throws IoTDBConnectionException, StatementExecutionException {
-session = new Session("127.0.0.1", 6667, "root", "root");
+session = new Session("172.16.48.4", 58890, "root", "root");

Review comment:
   Hi, please recover this class.

##
File path: 
tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkWriterImpl.java
##
@@ -119,14 +194,26 @@ public void write(long time, boolean value) {
 
   @Override
   public void write(long time, float value) {
-pageWriter.write(time, value);
-checkPageSizeAndMayOpenANewPage();
+if (!isSdtEncoding || sdtEncoder.encodeFloat(time, value)) {
+  if (isSdtEncoding) {
+time = sdtEncoder.getTime();
+value = (float) sdtEncoder.getValue();
+  }
+  pageWriter.write(time, value);
+  checkPageSizeAndMayOpenANewPage();
+}
   }
 
   @Override
   public void write(long time, double value) {
-pageWriter.write(time, value);
-checkPageSizeAndMayOpenANewPage();
+if (!isSdtEncoding || sdtEncoder.encodeDouble(time, value)) {
+  if (isSdtEncoding) {
+time = sdtEncoder.getTime();
+value = (double) sdtEncoder.getValue();

Review comment:
   could we getDoubleValue()?

##
File path: 
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/encoder/SdtEncoder.java
##
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.encoding.encoder;
+
+import java.nio.FloatBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+
+public class SdtEncoder {

Review comment:
   ```suggestion
   public class SDTEncoder {
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [iotdb] qiaojialin commented on a change in pull request #2087: [IOTDB-890] SDT implementation

[GitHub] [iotdb] qiaojialin commented on a change in pull request #2087: [IOTDB-890] SDT implementation

[GitHub] [iotdb] qiaojialin commented on a change in pull request #2087: [IOTDB-890] SDT implementation

3 matches

Site Navigation

Mail list logo

Footer information