Apache Pinot Daily Email Digest (2022-05-20)

Pinot Slack Email Digest Fri, 20 May 2022 19:24:05 -0700

#general

@diana.arnos: Hey there :) Do someone have a helm chart and config example for running more than 1 zookeeper running? For instace: I want to run 2 brokers, 2 controllers, 3 zookeepers and 4 servers Is it possible?
@mayanks: You want to run 3 or 5 ZK. With 2 ZK quorum won’t be achieved.
@diana.arnos: Yes, 3 zk I mistyped
@mayanks: @dlavoie
@mitchellh: Which helm chart are you working off of now?
@diana.arnos: The one available in the oficial repo
@mitchellh: As noted in the helm chart, it's best to use a separate Zookeeper operator. Is this a tenable solution?
@ebrucucen: @ebrucucen has joined the channel
@matthias.wesolowski: Is there any terraform provider to manage apache pinot (e.g. Setting up tables and streams etc)?
@g.kishore: No. there is a helm chart to deploy on k8s. is this for on-prem?
@matthias.wesolowski: No, it's the startree service... But also to configure this one it would be nice to not build something own
@bill: @bill has joined the channel
@bob: @bob has joined the channel
@aadil: @aadil has joined the channel

#random

@ebrucucen: @ebrucucen has joined the channel
@bill: @bill has joined the channel
@bob: @bob has joined the channel
@aadil: @aadil has joined the channel

#troubleshooting

@ebrucucen: @ebrucucen has joined the channel
@bill: @bill has joined the channel
@bob: @bob has joined the channel
@aadil: @aadil has joined the channel
@nrajendra434: :wave: Attempting to debug an error running pinot spark batch ingestion job. Using pinot 0.10.0 release with jdk 8 built via `mvn clean install -DskipTests -Pbin-dist -T 4 -Djdk.version=8` getting this error ```Caused by: shaded.com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.10.0 at shaded.com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at shaded.com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at shaded.com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:808) at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59) at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala) ... 32 more``` job spec is here ```executionFrameworkSpec: name: 'spark' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner' extraConfigs: stagingDir: dependencyJarDir: '' jobType: SegmentCreation inputDirURI: '' includeFileNamePattern: 'glob:**/*.parquet' outputDirURI: '' overwriteOutput: true pinotFSSpecs: - className: org.apache.pinot.plugin.filesystem.S3PinotFS scheme: s3 configs: region: us-east-1 recordReaderSpec: dataFormat: 'parquet' className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader' tableSpec: tableName: 'students' schemaURI: '' tableConfigURI: ''```
@mayanks: @kharekartik ^^

#pinot-dev

@kathuriakritihp: In Pinot, segments belonging to a particular dataset may have different indexes. Is this correct? If so, what is the benefit, aside from not having to reload all segments to apply index changes?
@richard892: you'd do that to accelerate queries, new data will have the new indexing which will (hopefully) make the queries faster. Any reloaded segments will have the new indexes too.

#getting-started