[jira] [Created] (IOTDB-192) Improvement for LRUCache.java
Yuan Tian created IOTDB-192: --- Summary: Improvement for LRUCache.java Key: IOTDB-192 URL: https://issues.apache.org/jira/browse/IOTDB-192 Project: Apache IoTDB Issue Type: Improvement Reporter: Yuan Tian Attachments: LRUCache.java # First, I got a question for the default value of _*CHUNK_METADATA_CACHE_SIZE_ **in MetadataQuerierByFileImpl* which is 10, will the number of sensors in one tsfile really reach that? # Second, if the number can really reach that, why don't we set the *_initialCapacity_* of *LinkedHashMap* to _*CHUNK_METADATA_CACHE_SIZE*_ to avoid rehashing. # Last but not the least, the field *_accessOrder_* and the method *_removeEldestEntry_* in *LinkedHashMap* provide convenience for us to implement *LRUCache**. The method _moveObjectToTail_**, _removeFirstObjectIfCacheIsFull_**, _removeFirstObject_* in *LRUCache.java* are unnecessary and these methods may also lower the efficiency. So, I simplify the code of *LRUCache.java* which you can see in the attachment. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-194) Redundant field 'measurementSchemaList' in Schema.java
Yuan Tian created IOTDB-194:
---
Summary: Redundant field 'measurementSchemaList' in Schema.java
Key: IOTDB-194
URL: https://issues.apache.org/jira/browse/IOTDB-194
Project: Apache IoTDB
Issue Type: Improvement
Reporter: Yuan Tian
There are two fields in Schema.java, 'measurementSchemaMap' and
'measurementSchemaList'. I don't understand the meaning of the existence of the
'measurementSchemaList' in Schema.java. If we use the following constructor and
there are repetitive measurement_id in the list, this will cause the
inconsistency of the above two fields.
{code:java}
private Map measurementSchemaMap;
private List measurementSchemaList;
public Schema(List measurements) {
this();
this.registerMeasurements(measurements);
}
public void registerMeasurements(List measurements) {
measurements.forEach((md) -> registerMeasurement(md));
}
public void registerMeasurement(MeasurementSchema descriptor) {
this.measurementSchemaMap.put(descriptor.getMeasurementId(), descriptor);
this.measurementSchemaList.add(descriptor);
}
{code}
If the purpose of 'measurementSchemaList' is to maintain the insertion order of
the measurements, why not use LinkedHashMap, then the order of map.values() is
certain.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (IOTDB-195) Bug about 'serializedSize' in ChunkGroupMetaData.java
Yuan Tian created IOTDB-195:
---
Summary: Bug about 'serializedSize' in ChunkGroupMetaData.java
Key: IOTDB-195
URL: https://issues.apache.org/jira/browse/IOTDB-195
Project: Apache IoTDB
Issue Type: Bug
Reporter: Yuan Tian
According to the comments, the field 'serializedSize' in ChunkGroupMetaData
class means 'Byte size of this metadata'.
However, when you calculate the value in the method 'deserializeFrom' as we can
see in the following, chunkGroupMetaData.deviceID.length() may not equal to the
byte size of the string.
{code:java}
/**
* Byte size of this metadata. this field is not serialized.
*/
private int serializedSize;
chunkGroupMetaData.serializedSize = Integer.BYTES +
chunkGroupMetaData.deviceID.length() + Integer.BYTES + Long.BYTES + Long.BYTES
+ Long.BYTES;
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (IOTDB-196) One question for the explanation about the structure of DeviceIndexMetadata in the document
Yuan Tian created IOTDB-196: --- Summary: One question for the explanation about the structure of DeviceIndexMetadata in the document Key: IOTDB-196 URL: https://issues.apache.org/jira/browse/IOTDB-196 Project: Apache IoTDB Issue Type: Bug Reporter: Yuan Tian Attachments: WechatIMG109.png After reading the corresponding code of reading data from TsFile, I think the second field of DeviceIndexMetadata must be start offset of TsDeviceMetaData, not as described in the documentation. !WechatIMG109.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-202) Bug in example module
Yuan Tian created IOTDB-202: --- Summary: Bug in example module Key: IOTDB-202 URL: https://issues.apache.org/jira/browse/IOTDB-202 Project: Apache IoTDB Issue Type: Bug Reporter: Yuan Tian Attachments: WechatIMG111.png When I use TsFileWriteWithRowBatch class to write into test.tsfile, both of TsFileRead and TsFileSequenceRead work fine. However, while using TsFileWriteWithTSRecord to write, TsFileSequenceRead still work well, but TsFileRead will crash. After printing some record, it will throw an exception as following: !WechatIMG111.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-207) Bug about the method query(QueryExpression queryExpression, long partitionStartOffset, long partitionEndOffset) in ReadOnlyTsFile
Yuan Tian created IOTDB-207: --- Summary: Bug about the method query(QueryExpression queryExpression, long partitionStartOffset, long partitionEndOffset) in ReadOnlyTsFile Key: IOTDB-207 URL: https://issues.apache.org/jira/browse/IOTDB-207 Project: Apache IoTDB Issue Type: Bug Reporter: Yuan Tian While using the method query(QueryExpression queryExpression, long partitionStartOffset, long partitionEndOffset) in ReadOnlyTsFile, I found that if the TsFiles only contains one chunkGroup and I set the parameter ‘partitionStartOffset’ to the startOffset of the chunkGroup and the parameter ‘partitionEndOffset’ to the endOffset of the chunkGroup, the method should return a queryDataset that contains all data. However, it lost some data. More specifically, in my case, I use TsFileWriteWithRowBatch in example module to create a TsFile consisting of only one device, ten sensors and 1 million row data totally. While using the method as mentioned before to read, I only got 999424 row data. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-209) Improvement on the Hadoop module
Yuan Tian created IOTDB-209: --- Summary: Improvement on the Hadoop module Key: IOTDB-209 URL: https://issues.apache.org/jira/browse/IOTDB-209 Project: Apache IoTDB Issue Type: Improvement Reporter: Yuan Tian Hadoop module isn't updated with the TsFile module, so there are lots of history's legacy in that module. Hoping that the Hadoop module can be remaintained, thus we can do some Hadoop map-reduce jobs on the TsFiles. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-210) One else if branch will never be reached in the method optimize of ExpressionOptimizer
Yuan Tian created IOTDB-210:
---
Summary: One else if branch will never be reached in the method
optimize of ExpressionOptimizer
Key: IOTDB-210
URL: https://issues.apache.org/jira/browse/IOTDB-210
Project: Apache IoTDB
Issue Type: Bug
Reporter: Yuan Tian
The code has been shown in the following. As we can see, the condition in the
last else if, if 'left.getType() == ExpressionType.SERIES && right.getType() ==
ExpressionType.SERIES' is true, the condition in the previous else if which is
'left.getType() != ExpressionType.GLOBAL_TIME&& right.getType() !=
ExpressionType.GLOBAL_TIME' must be true. Thus, the last else if code block
will never be executed.
{code:java}
else if (left.getType() != ExpressionType.GLOBAL_TIME
&& right.getType() != ExpressionType.GLOBAL_TIME) {
} else if (left.getType() == ExpressionType.SERIES
&& right.getType() == ExpressionType.SERIES) {
...
}{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (IOTDB-245) Different types of measurements that have same name in one storage group should be supported.
Yuan Tian created IOTDB-245: --- Summary: Different types of measurements that have same name in one storage group should be supported. Key: IOTDB-245 URL: https://issues.apache.org/jira/browse/IOTDB-245 Project: Apache IoTDB Issue Type: Improvement Reporter: Yuan Tian If we have two devices named d1 and d2, and both of them have a sensor named s1, d1.s1 and d2.s1 must have the same type in current version. So if you really have d1.s1 which has float type and d2.s1 which has int32 type, you have to separate them into two different storage groups which is unreasonable. So we want to remove the restriction about measurements' type of same name in one storage group. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-246) No docs for Hadoop-Connector in IoTDB websites.
Yuan Tian created IOTDB-246: --- Summary: No docs for Hadoop-Connector in IoTDB websites. Key: IOTDB-246 URL: https://issues.apache.org/jira/browse/IOTDB-246 Project: Apache IoTDB Issue Type: Improvement Reporter: Yuan Tian There are no docs for Hadoop-Connector in IoTDB websites. Users may not know how to use the function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-285) Duplicate fields in EngineDataSetWithoutValueFilter.java
Yuan Tian created IOTDB-285:
---
Summary: Duplicate fields in EngineDataSetWithoutValueFilter.java
Key: IOTDB-285
URL: https://issues.apache.org/jira/browse/IOTDB-285
Project: Apache IoTDB
Issue Type: Improvement
Reporter: Yuan Tian
There two fields in EngineDataSetWithoutValueFilter.java used to fetch the
minimum time.
{code:java}
// Some comments here
private PriorityQueue timeHeap;
private Set timeSet;
{code}
the Set is used to keep heap from storing duplicate time.
However, a TreeSet field can do both things. No duplicate time and ensure the
time order. There is no need to use these two.
Especially, when we want to change to multiThread version, to keep the
timeHeapPut thread safe, we have to add a synchronized onto the method, like
this:
{code:java}
private synchronized void timeHeapPut(long time) {
if (!timeSet.contains(time)) {
timeSet.add(time);
timeHeap.add(time);
}
}
{code}
But, if we only use TreeSet, we can simply use the corresponding version,
ConcurrentSkipListSet, to replace it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-290) Bug about threadlocal field in TSServiceImpl.java
Yuan Tian created IOTDB-290:
---
Summary: Bug about threadlocal field in TSServiceImpl.java
Key: IOTDB-290
URL: https://issues.apache.org/jira/browse/IOTDB-290
Project: Apache IoTDB
Issue Type: Bug
Reporter: Yuan Tian
There are two fields named queryStatus and queryRet:
{code:java}
private ThreadLocal> queryStatus = new
ThreadLocal<>();
private ThreadLocal> queryRet = new
ThreadLocal<>();
{code}
They are used to save the PhysicalPlan and QueryDataSet of one query. However,
the keys of both are sql string to be executed, so if the client execute the
same sql in one connection, no matter whether the first execution has finished,
the second sql execution will clear the first one's QueryDataSet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-291) Statement close operation may cause the whole connection's resource to be released
Yuan Tian created IOTDB-291:
---
Summary: Statement close operation may cause the whole
connection's resource to be released
Key: IOTDB-291
URL: https://issues.apache.org/jira/browse/IOTDB-291
Project: Apache IoTDB
Issue Type: Bug
Reporter: Yuan Tian
The function closeClientOperation() in IoTDBStatement.java invokes the function
closeOperation(TSCloseOperationReq) in TSServiceImpl.
Because the queryId field in TSCloseOperationReq is set to be -1L, the
releaseQueryResource function will release all the resources in that
connection.
{code:java}
private void closeClientOperation() throws SQLException {
try {
if (operationHandle != null) {
TSCloseOperationReq closeReq = new TSCloseOperationReq(operationHandle,
-1);
closeReq.setStmtId(stmtId);
TSStatus closeResp = client.closeOperation(closeReq);
RpcUtils.verifySuccess(closeResp);
}
} catch (Exception e) {
throw new SQLException("Error occurs when closing statement.", e);
}
}
{code}
{code:java}
private void releaseQueryResource(TSCloseOperationReq req) throws
StorageEngineException {
Map contextMap = contextMapLocal.get();
if (contextMap == null) {
return;
}
if (req == null || req.queryId == -1) {
// end query for all the query tokens created by current thread
for (QueryContext context : contextMap.values()) {
QueryResourceManager.getInstance().endQueryForGivenJob(context.getJobId());
}
contextMapLocal.set(new HashMap<>());
} else {
QueryResourceManager.getInstance()
.endQueryForGivenJob(contextMap.remove(req.queryId).getJobId());
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-347) Bug in limit clause
Yuan Tian created IOTDB-347:
---
Summary: Bug in limit clause
Key: IOTDB-347
URL: https://issues.apache.org/jira/browse/IOTDB-347
Project: Apache IoTDB
Issue Type: Bug
Reporter: Yuan Tian
The test function `testQuery` in `IoTDBQueryResultSetTest.java` construct data
as following
{code:java}
Object[][] input = {
{2L, 2.22F, 4L, null,},
{3L, 3.33F, null, null,},
{4L, 4.44F, null, null,},
{50L, null, 5L, null,},
{100L, null, 199L, null,},
{101L, null, 199L, null,},
{103L, null, 199L, null,},
{105L, 11.11F, 199L, 3,},
{1000L, 1000.11F, 5L, 2,}};
{code}
and query sql is
{code:java}
String testSql = "select *,s1,s0,s2 from root.vehicle.d0 where s1 > 190 or s2 <
10.0 "
+ "limit 20 offset 1 slimit 4 soffset 2";
{code}
with the offset 1, we should skip the `\{2L, 2.22F, 4L, null,}`, but the
expected answer doesn't do that:
{code:java}
String standard =
"Time,root.vehicle.d0.s2,root.vehicle.d0.s1,root.vehicle.d0.s0,root.vehicle.d0.s2,\n"
+ "2,2.22,4,null,2.22,\n"
+ "3,3.33,null,null,3.33,\n"
+ "4,4.44,null,null,4.44,\n"
+ "50,null,5,null,null,\n"
+ "100,null,199,null,null,\n"
+ "101,null,199,null,null,\n"
+ "103,null,199,null,null,\n"
+ "105,11.11,199,3,11.11,\n"
+ "1000,1000.11,5,2,1000.11,\n";
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IOTDB-414) Create Time series and show storage group Statements can not execute successfully
[
https://issues.apache.org/jira/browse/IOTDB-414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013630#comment-17013630
]
Yuan Tian commented on IOTDB-414:
-
I do the same thing on the master branch, the create timeseries statement is
actually executed successfully.
{code:java}
create timeseries root.sg1.d2.s1 with datatype=INT64, encoding=RLE,
compression=SNAPPY;{code}
and I use
compression=SNAPPY not compressor=SNAPPY, I don't know what's the difference
between them
> Create Time series and show storage group Statements can not execute
> successfully
> --
>
> Key: IOTDB-414
> URL: https://issues.apache.org/jira/browse/IOTDB-414
> Project: Apache IoTDB
> Issue Type: Bug
>Affects Versions: master branch
>Reporter: Xiangdong Huang
>Assignee: Boris Zhu
>Priority: Major
>
> Hi, current master branch has some issues for executing some sqls. e.g.,:
>
> IoTDB> show timeseries;
> show timeseries;
> +-+-+-+-+-+
> | timeseries|storage group| dataType| encoding| compression|
> +-+-+-+-+-+
> |root.sg1.d1.s1| root.sg1| INT64| RLE| SNAPPY|
> |root.sg1.d1.s2| root.sg1| INT64| RLE| SNAPPY|
> |root.sg1.d1.s3| root.sg1| INT64| RLE| SNAPPY|
> +-+-+-+-+-+
> Total line number = 3
> It costs 0.006s
> IoTDB> create timeseries root.sg1.d2.s1 with datatype=INT64, encoding=RLE,
> compression=SNAPPY;
> create timeseries root.sg1.d2.s1 with datatype=INT64, encoding=RLE,
> compression=SNAPPY;
> Msg: The resultDataType or encoding or compression of the last node s1 is
> conflicting in the storage group root.sg1
> It costs 0.003s
> IoTDB> show storage group;
> show storage group;
> Msg: column 2 does not exist
> It costs 0.001s
> # The create timeseries statement should be executed successfully.
> # what is the problem of "show storage group"?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IOTDB-478) Redesign the cache in TsFile/Engine
[ https://issues.apache.org/jira/browse/IOTDB-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038713#comment-17038713 ] Yuan Tian commented on IOTDB-478: - Yes, we keep the cache in ChunkLoaderImpl in TsFile module and just rename that class. In Server module, we stop using the ChunkLoaderImpl in TsFile. We implement another two class named 'DiskChunkLoader' and 'MemChunkLoader' to manage cached RAM in server. > Redesign the cache in TsFile/Engine > --- > > Key: IOTDB-478 > URL: https://issues.apache.org/jira/browse/IOTDB-478 > Project: Apache IoTDB > Issue Type: Improvement > Components: Core/Engine, Core/TsFile >Reporter: Jialin Qiao >Assignee: Yuan Tian >Priority: Major > > Currently, the ChunkLoaderImpl has an internal cache, which is not used in > the server. This may cause much gc overhead. > I suggest removing the cache in ChunkLoaderImpl, leaving ChunkLoader only a > tool, and redesign the cache in TsFile and server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
