#general
@babyzhou163: @babyzhou163 has joined the channel
@minimizer: @minimizer has joined the channel
@renjith.varma: @renjith.varma has joined the channel
@hjj8645561: @hjj8645561 has joined the channel
@tejaswini.iiitn: Hi Team, Anyone aware of dbt integration with pinot. I am trying use dbt to transformation data and push the data to pinot
@g.kishore: No, not yet. Can you file a GitHub issue
@tejaswini.iiitn:
@g.kishore: thank you
@tejaswini.iiitn: @g.kishore Here is my use case I get the raw data from kafka stream . I perform transformation using dbt tool. Transformed data pushed to pinot.
@tejaswini.iiitn: Can you suggest alternative for transformation engine in pinot
@g.kishore: What kind of transformation.. complex joins or single event transform
@tejaswini.iiitn: single event transform for now. logiaal name mapping and some simple derivation and date conversions
@g.kishore: You can do that in Pinot during ingestion
@g.kishore: No need of another tool
@g.kishore:
@tejaswini.iiitn: I will check
@diana.arnos: Hello, everyone :wave: Does Pinot have (or plan to have) something available inside AWS marketplace?
@mayanks: Not that I am aware of.
@shreesh.mansotra: @shreesh.mansotra has joined the channel
#random
@babyzhou163: @babyzhou163 has joined the channel
@minimizer: @minimizer has joined the channel
@renjith.varma: @renjith.varma has joined the channel
@hjj8645561: @hjj8645561 has joined the channel
@shreesh.mansotra: @shreesh.mansotra has joined the channel
#troubleshooting
@babyzhou163: @babyzhou163 has joined the channel
@minimizer: @minimizer has joined the channel
@renjith.varma: @renjith.varma has joined the channel
@hjj8645561: @hjj8645561 has joined the channel
@saumya2700: I am setting up prometheus to monitor pinot, some metrices are not coming up, guessing it might be issue with setting up scrape configs , please tell where we need to add this config in which file : ```controller: ... service: annotations: "
@richard892: which metrics are not showing up?
@richard892: the way we configure this in managed deployments is to pass the name of the rules file to the prometheus agent
@saumya2700: These are some which we need to see if segments are going offline, and really wanted to know if there some OOM or resource related issues.
@richard892: yes, clearly, prometheus isn't configured properly
@saumya2700: We followed this document (
@richard892: we set ```-javaagent:${PINOT_HOME}/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=${PORT}:${PINOT_HOME}/etc/jmx_prometheus_javaagent/configs/{server|broker|controller|minion|pinot}.yml```
@richard892: it looks to me like you haven't added the javaagent
@saumya2700: We are running it in EKS, and we are giving this javaagent in controller statefullsets : ```env: - name: JAVA_OPTS value: >- -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log -Dlog4j2.configurationFile=/opt/pinot/conf/log4j2.xml -Dplugins.dir=/opt/pinot/plugins -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml```
@saumya2700: Thanks @richard892 for your quick response, we got to manage to resolve this in EKS pod so we need to add that javaagent for every pod and also annotations. But not sure why pinot server taking so much memory. @
@richard892: how much memory are you expecting it to take?
@saumya2700:
@saumya2700:
@lars-kristian_svenoy: Hello everyone :wave: For upsert tables its specified that you need to partition your input stream (kafka) by the primary key. How is this achieved when working with composite keys? I'm also curious how this works together with the column partition map, I thought it had to correspond to the partition in kafka?
@lars-kristian_svenoy: For example..
@lars-kristian_svenoy: This is the segment partitioning ``` "tableIndexConfig": { "loadMode": "MMAP", "segmentPartitionConfig": { "columnPartitionMap": { "myPartitionKey": { "functionName": "Murmur", "numPartitions": 40 } } },``` Messages in the kafka topic are partitioned on "myPartitionKey" but the schema defines the composite primary key ```"primaryKeyColumns": ["myPartitionKey", "anotherKey"]```
@lars-kristian_svenoy: Is this correct?
@mayanks: @jackie.jxt @yupeng for comments.
@jackie.jxt: @lars-kristian_svenoy Your config should work because as long as `myPartitionKey` is partitioned, the composite keys are also partitioned
@jackie.jxt: The prerequisite for using upsert is that all the records of the same primary key located in the same streaming partition
@lars-kristian_svenoy: Great thank you. That is what I expected, and means it will work the way I have it now :+1:
@shreesh.mansotra: @shreesh.mansotra has joined the channel
#getting-started
@babyzhou163: @babyzhou163 has joined the channel
@minimizer: @minimizer has joined the channel
@renjith.varma: @renjith.varma has joined the channel
@hjj8645561: @hjj8645561 has joined the channel
@shreesh.mansotra: @shreesh.mansotra has joined the channel
#introductions
@babyzhou163: @babyzhou163 has joined the channel
@greetbot: @babyzhou163 I think this is the beginning of a beautiful friendship :relaxed:
@minimizer: @minimizer has joined the channel
@greetbot: Live long and prosper @minimizer :spock-hand:
@renjith.varma: @renjith.varma has joined the channel
@greetbot: Woof! Woof! @renjith.varma is here!
@hjj8645561: @hjj8645561 has joined the channel
@greetbot: Don’t stop, make it pop, @hjj8645561 blow my speakers up :boom:
@tlberglund: Hello!
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
