#general
@hljpzz1982: @hljpzz1982 has joined the channel
#random
@hljpzz1982: @hljpzz1982 has joined the channel
#troubleshooting
@deemish2: while executing batch ingestion job using command - bin/pinot-ingestion-job.sh -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml under pinot-0.9.0 . It gives error - Error: Main method not found in class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand, please define the main method as: public static void main(String[] args) or a JavaFX application class must extend javafx.application.Application
@xiangfu0: can you try this one: ```bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml ```
@xiangfu0: I will take a look this script
@xiangfu0: this method is actually deprecated and should be replaced with ```bin/pinot-admin.sh LaunchDataIngestionJob ```
@xiangfu0: I will update the doc
@xiangfu0: Thanks for bring this up! @deemish2
@deemish2: it is working . Thanks
@lrhadoop143: Hi Team, I'm trying to load data from Minio to pinot but facing issues while running yaml files. ERROR:expected '<document start>', but found BlockMappingStart in 'string', line 6, column 1: jobType: SegmentCreationAndMetad ...
@xiangfu0: seems yaml issue
@xiangfu0: should be no space before jobType
@xiangfu0: check indentation
@lrhadoop143: Now i'm getting this error
@lrhadoop143: java.lang.IllegalStateException: PinotFS for scheme: http has not been initialized
@lrhadoop143: i'm trying this is in docker trying to read data from minio bucket used s3 as the scheme pinotFSSpecs: - scheme: s3 className: org.apache.pinot.plugin.filesystem.S3PinotFS
@lrhadoop143: Could not instantiate file system for class org.apache.pinot.plugin.filesystem.S3PinotFS with scheme s3 java.lang.IllegalArgumentException: null
@xiangfu0: can you add a region into the config?
@xiangfu0: ```- scheme: s3 className: org.apache.pinot.plugin.filesystem.S3PinotFS configs: region: 'us-west-2'```
@lrhadoop143: no use still i'm facing java.lang.IllegalStateException: PinotFS for scheme: https has not been initialized
@lrhadoop143: Facing {"code":500, "error":"org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata"}
@lrhadoop143: when creating realtime table in pinot ,i'm using{ "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9876", "realtime.segment.flush.threshold.size": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.desired.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } }
@xiangfu0: Hmm, how do you start Pinot ?
@xiangfu0: Docker or k8s or script?
@xiangfu0: I guess your Kafka is local but in docker?
@hljpzz1982: @hljpzz1982 has joined the channel
@ayush.jha: Hi everyone , recently I tried to upgrade pinot version from 0.7.1 and I am doing ingestion using files stored in azure blob but I am getting this error ```java.lang.IllegalStateException: Unable to extract out the relative path for input file file path "file path"``` in 0.8.0 and 0.9.0 but it is working fine in 0.7.1
@mayanks: Can you share the stack trace and your ingestion job spec file?
@ayush.jha: this is general blue print of ingestion.yaml ```executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '
@ayush.jha: and stacktrace is ```Failed to generate Pinot segment for file - file:/TABLE_NAME/FILE1 java.lang.IllegalStateException: Unable to extract out the relative path for input file 'file:/TABLE_NAME/FILE1', based on base i nput path:
@ayush.jha: p.s same ingestion.yaml is working fine in 0.7.1
@mayanks: What do you see in the log for this message: ```
@mayanks: Also, your spec says `dataFormat: 'avro'`, but you are using `'org.apache.pinot.plugin.inputformat.orc.ORCRecordReader'`?
@ayush.jha: No luck changing dataFormat: 'avro' to dataFormat: 'orc' ```Initializing PinotFS for scheme adl, classname org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS Authenticating using the access key to the account. ADLSGen2PinotFS is initialized (accountName=accountName, fileSystemName=fileSystemName, dfsServiceEndpointUrl=dfsServiceEndpointUrl, blobServiceEndpointUrl=blobServiceEndpointUrl, enableChecksum=false)```
@mayanks: @snlee ^^
@snlee: @mayanks It looks that the exception is thrown during relative path computation. I can easily reproduce the issue ``` public static void main(String[] args) { getRelativeOutputPath(URI.create("
@xiangfu0: right, the scheme should be `adl` not `file`
@snlee: Somehow the `inputFile` is passed as `file:/TABLE_NAME/FILE1`
@snlee: not `
@xiangfu0: hmm, that’s weird, it means the inputFileURI scheme is not adl but file
@mayanks: Yes I suspected that too, that’s why I asked for log but is shows above adls gen2 was initialized
@xiangfu0: it means this `String[] files = _inputDirFS.listFiles(_inputDirURI, true);` is giving all the uris with `file` scheme?
@xiangfu0: is it because adl PinotFs listFiles doesn’t give scheme?
@snlee: oh yeah probably we don’t put the prefix
@mayanks: Is this a new issue then?
@xiangfu0: I think once we saw this for hdfs but not checked adls
#pinot-dev
@matteobovetti: @matteobovetti has joined the channel
@ken: I’m working on a PR for
@ken: Oh, right - and the problem with writing a test for a protected method is that the `SegmentPushUtilsTest` file is in the `pinot-core` subproject, in the `org.apache.pinot.core.util` package.
@xiangfu0: maybe pinot-segment-spi
@xiangfu0: or pinot-common I think as it has reference to that
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
