gortiz opened a new issue, #10633:
URL: https://github.com/apache/pinot/issues/10633

   Timestamp indexes are explained as indexes to users (see the 
[documentation](https://docs.pinot.apache.org/basics/indexing/timestamp-index)) 
but they are not implemented as actual indexes. Instead, they are a syntactic 
sugar that:
   - Modifies (see 
[TimestampIndexUtils](https://github.com/apache/pinot/blob/44fe6940e19cbf29d2f5f606124c2f6b77e85360/pinot-spi/src/main/java/org/apache/pinot/spi/utils/TimestampIndexUtils.java#L128))the
 TableConfig and Schema in order to:
      - Add one new column per granularity.
      - Add a range index on each column.
    - Applies some query override in order to use the new columns when 
available. I didn't find the code where this is applied, but given that 
`FieldConfig.IndexType.TIMESTAMP` is not read, I guess that the query engine 
looks for columns with specific pattern names as a general optimization.
   
   This is not ideal and we should correctly define Timestamp _indexes_. 
Options are:
   1. We can either accept that they are not actual indexes and pull them 
somewhere else. They would still be configured in 
`tableConfig.fieldConfigList.timestampConfig` but we will not call them 
_indexes_. We would need to classify them as a new concept.
   2. Or we make them actual indexes.
   
   The first approach mainly implies to change documentation and keep the 
current implementation.
   This will make things less confusing, but the current implementation is 
still difficult to maintain and/or not very efficient:
   - Usually implementations do not actually modify the `TableConfig` and 
`Schema`.
   - Adding more columns may be problematic in terms of storage. If the table 
is stored by the timestamp, new columns forward indexes should be small in 
size, but if it is not, the these new columns can be quite large in bytes.
   
   Therefore we propose to make them actual indexes.
   This implies to create their own IndexType, IndexCreator, IndexReader, 
IndexHandler... but also configure them in the 
`tableConfig.fieldConfigList.indexes` section and store them in a single buffer 
instead of being an alias on top of artificial columns.
   
   ## Configuration
   
   Like other indexes, we will add a new way to configure Timestamp indexes. 
   To keep backward compatibility, timestamp indexes defined in 
`tableConfig.fieldConfigList.timestampConfig` will continue to be supported.
   But the new and recommended way to define them would be something like:
   
   ```js
   //tableconfig
   {
     fieldConfigList: [{
       name: "columnName"
       indexes: {
         "timestamp": { 
             ...timestampConfig  // same object we had in timestampConfig but 
adding the standard "disabled" property
          }
       }
     }]
   }
   ```
   
   Since `index-spi` was merged, timestamp _indexes_ are the only one that 
cannot be defined in the `indexes` section.
   To let them be configured in the `indexes` section seems to be something 
easy to implement.
   
   ## Index implementation
   The idea here would be to implement this kind of index as a normal index, 
which means to store all information in a single buffer and do not create new 
columns.
   
   At the end of the day a Timestamp index contains N different range indexes. 
Right now they are implemented as N different columns with 1 range index each. 
The idea here would be to store the N range indexes in 1 single buffer. 
   Each range index should have the same size of the one used in the current 
solution, but we won't need to add one forward index per granularity.
   
   At query time, we need to add rules like we have in other indexes and apply 
it if the operation is compatible. The current optimizations applied for 
timestamp indexes are:
   1. GROUP BY: functions like dateTrunc('DAY', ts) will be translated to use 
the underly column $ts$DAY to fetch data.
   2. PREDICATE: range index is auto-built for all granularity columns.
   
   The predicate optimization can be applied in the same way with the proposed 
actual index type.
   The group by may be tricky and we may need to think more about them, but at 
worst, users can decide to create their own virtual columns like they do to 
optimize similar patterns with other types.
   
   This change may be more difficult to implement and require to invest more 
time than the change in the configuration. If we decide that this is the 
solution we would like to have at long term, we can start implementing the 
changes in the configuration in order to provide an uniform way to configure 
all indexes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to