#general
@sirsh: @sirsh has joined the channel
@mattk: @mattk has joined the channel
#random
@sirsh: @sirsh has joined the channel
@mattk: @mattk has joined the channel
#troubleshooting
@mrpringle: I have the kafka low level consumer setup to move completed segments to offline tenant servers. However looking in the logs for these offline servers I can see a lot of exceptions `021/09/10 04:43:10.103 error [groupbyorderbycombineoperator] [pqr-5] timed out while combining group-by order-by results after 9920ms, querycontext = querycontext` do we know why this happens/how to fix it?
@mrpringle: Also a lot of `Query scheduling took 12916ms (longer than query timeout of 9999ms)`
@sirsh: @sirsh has joined the channel
@mattk: @mattk has joined the channel
#getting-started
@sirsh: @sirsh has joined the channel
@sirsh: Hello - im just getting setup and would like to get some advice... I have added Pinot via Helm on EKS and i will want to (a) ingest parquet from S3 and (b) stream data from MSK(Kafka 2.2.1 on the same VPC) and probably requiring SSL 1. I added a simple schema and table spec - they look ok 2. I (think) i configure deep storage for S3 3. Batch ingestion - Really i am interested in any recommended (standalone) way to ingest data from S3 (watching folders on some interval) and im not sure if the docs are providing what i need? Is there an example of posting directly to the controller? 4. I would like to try with Kafka too in this case im just wondering about the configs (in thread). I am feeling my way though and in this case if anyone has a sample config that would be nice to see for this MSK setup but i expect ill get there with some trial and error a. I was a little put off but the mention of needing to update the
@xiangfu0:
@xiangfu0: also are you building the docker image from source code ? if not then latest image should just pack kafka 2.0
@sirsh: Thanks for this - in fact i read through this documentation but given that i already had MSK configured i got a little stuck with my particular setup. My setup is to have Pinot installed on EKS managed with ArgoCD/Heml on EKS and MKS on the same VPC. So i am using the local Pinot CLI to submit configs etc. Currently I can create/submit the table spec and see the table created but the table has a "Bad" status. I was little unsure if i was able to connect to Kafka at all and wondered if SSL or something like that was causing issues. Here is my config ``` "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "security.protocol": "SSL", "stream.kafka.topic.name": "TOPIC "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "largest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M", "stream.kafka.zk.broker.url": "
@xiangfu0: I think the issue is pinot cannot connect to msk
@xiangfu0: can you try to enter pinot container and see if you can ping to that host
@xiangfu0: `kubectl exec -it pod/pinot-controller-0 -n pinot -- bash`
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
