estebanz01 opened a new issue, #12547: URL: https://github.com/apache/pinot/issues/12547
Hola! 👋 I'm having some trouble with a pinot cluster deployed into kubernetes with minion enabled. I want to move data from real time table to offline table but it's failing with the following information: ```java ^T[16:18:00.383 [TaskStateModelFactory-task_thread-7] ERROR org.apache.pinot.minion.taskfactory.TaskFactoryRegistry - Caught exception while executing task: Task_RealtimeToOfflineSegmentsTask_8961b037-3c41-47d7-b56f-375ef16dc2fc_1709569080105_0 org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 1 attempts at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.fetchSegmentToLocal(HttpSegmentFetcher.java:62) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:158) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:152) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:202) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:190) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:201) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:77) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:157) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:118) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] ``` That's the only error I see in the minion pods and there's nothing else on the other pods for pinot. Any ideas on how to debug this further? here's the schema and table config for my hybrid table: <details> <summary>Schema definition</summary> ```json { "schemaName": "data_counting", "dimensionFieldSpecs": [ { "name": "device_name", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "bytes_sent", "dataType": "LONG" } ], "dateTimeFieldSpecs": [ { "name": "__key", "dataType": "TIMESTAMP", "format": "1:MICROSECONDS:EPOCH", "granularity": "1:MICROSECONDS" }, { "name": "__metadata$eventTime", "dataType": "TIMESTAMP", "format": "1:MICROSECONDS:EPOCH", "granularity": "1:MICROSECONDS" } ] } ``` </details> <details> <summary>Table configuration (REALTIME)</summary> ```json { "REALTIME": { "tableName": "data_counting_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "data_counting", "replication": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "15", "replicasPerPartition": "1", "minimizeDataMovement": false, "timeColumnName": "__key" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "tableIndexConfig": { "invertedIndexColumns": [], "noDictionaryColumns": [], "streamConfigs": { "streamType": "pulsar", "stream.pulsar.topic.name": "persistent://client/devices/all", "stream.pulsar.bootstrap.servers": "pulsar://pulsar-proxy.pulsar.svc.cluster.local:6650", "stream.pulsar.prop.auto.offset.reset": "smallest", "stream.pulsar.consumer.type": "lowlevel", "stream.pulsar.fetch.timeout.millis": "20000", "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.pulsar.consumer.factory.class.name": "org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory", "realtime.segment.flush.threshold.rows": "10000", "realtime.segment.flush.threshold.time": "1h", "stream.pulsar.metada.populate": "true", "stream.pulsar.metadata.fields": "eventTime" }, "loadMode": "MMAP", "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": false, "rangeIndexColumns": [], "rangeIndexVersion": 2, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.85, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "sortedColumn": [], "bloomFilterColumns": [] }, "metadata": {}, "quota": {}, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "1h", "bufferTimePeriod": "2h", "mergeType": "concat", "maxNumRecordsPerSegment": "100000", "schedule": "0 * * * * ?" } } }, "routing": {}, "query": { "timeoutMs": 60000 }, "ingestionConfig": { "continueOnError": false, "rowTimeValueCheck": false, "segmentTimeValueCheck": true }, "isDimTable": false } } ``` </details> <details> <summary>Table configuration (OFFLINE)></summary> ```json { "OFFLINE": { "tableName": "data_counting_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "schemaName": "data_counting", "replication": "1", "replicasPerPartition": "1", "timeColumnName": "__key", "minimizeDataMovement": false, "segmentPushType": "APPEND", "segmentPushFrequency": "HOURLY" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "invertedIndexColumns": [], "noDictionaryColumns": [], "rangeIndexColumns": [], "rangeIndexVersion": 2, "createInvertedIndexDuringSegmentGeneration": false, "autoGeneratedInvertedIndex": false, "sortedColumn": [], "bloomFilterColumns": [], "loadMode": "MMAP", "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": false, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0.85 }, "metadata": {}, "quota": {}, "routing": {}, "query": {}, "ingestionConfig": { "continueOnError": false, "rowTimeValueCheck": false, "segmentTimeValueCheck": true }, "isDimTable": false } } ``` </details> I'm using apache pulsar 3.2.0 and apache pinot version 1.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org