nibinqtl opened a new issue, #18780:
URL: https://github.com/apache/druid/issues/18780
### Affected Version
35.0.0
### Description
Please include as much detailed information about the problem as possible.
- Cluster size: Single server small
- Configurations in use: small
- Steps to reproduce the problem
- The error message or stack traces encountered. Providing more context,
such as nearby log messages or even entire logs, can be helpful.
- Any debugging that you have already done
I'm ingesting data from using local input source, JSON format, with
segmentGranularity=hour
each batch results in a segment partition for the hour. About 10-30 in each
hour, about 60k rows in each partition.
I created a auto-compaction to consolidate into segments with 4H interval.
Here is the config JSON:
{
"dataSource": "data",
"skipOffsetFromLatest": "PT4H",
"tuningConfig": {
"partitionsSpec": {
"type": "single_dim",
"partitionDimension": "hostHeader",
"targetRowsPerSegment": 3000000,
"assumeGrouped": false
},
"type": "index_parallel"
},
"granularitySpec": {
"segmentGranularity": {
"type": "period",
"period": "PT4H"
},
"rollup": false
},
"ioConfig": {
"dropExisting": true
}
}
What I found is the skipOffsetFromLatest seems to be ignored. The compact
task always starts right after each 4 hour boundary for the previous 4-hour
interval. For example, the new segment 04:00:00/08:00:00 is generated at
08:06:00. There is no wait time. I tried to set skipOffsetFromLatest at
different values but made no difference.
However, the datasource tab of the console correctly shows: "Fully compacted
(except the last PT4H of data, 12 segments skipped)"
Any idea why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
