#general
@juan: @juan has joined the channel
@praveen82: @praveen82 has joined the channel
@trustokoroego: Hello! I need to connect presto to Pinot with basic Auth. Could anyone point point me to how I can set this in the pinot.properties Presto catalog configuration.
@dadelcas: This has been added to Trino recently. I'm not sure if presto supports that, last time I checked the connector was a bit behind the latest releases of pinot
@mayanks: I think that can only be done via HTTP headers in Presto. @xiangfu0?
@trustokoroego: @mayanks I am not sure how to configure the HTTP header, since I am using the Pinot connector config. How do I do it via HTTP header in presto?
@mayanks: Ah I see. Will need to find, unless @xiangfu0 comes back sooner
@mingleifly: Hi, Why do we use DISTINCTCOUNT instead of using COUNTDISTINCT here ?
@mark.needham: `distinctcount` is the name of the function that does exact distinct counting?
@mingleifly: Yes. That’s right. Does it name should be COUNTDISTINCT ?
@mark.needham: Oh I see. yeh I'm not sure why the name is that way around, @mayanks will probably know
@mayanks: It is there from PQL days. Now that we moved to SQL a while back, you can also use the sql syntax of count(distinct, …).
@dadelcas: Hello, I've got a question about realtime tables. If I'm correct the kafka consumer group ID is built in the code using the table name and replica ID, however I'm not able to find a consumer group for the table in my kafka cluster. Is there a way to list all the consumer groups that a realtime table is using? I would look like those IDs are stored in ZK under ideal states but I can't find them. Thanks
@apte.kaivalya: IIUC pinot doesn’t use consumer groups to consume from kafka and it is `null`. This is done mainly to let multiple pinot consumers consume from the same partition (for replication).
@apte.kaivalya: context:
@dadelcas: Thanks for forwarding that answer! I think that may relate to an issue I see in one of our tables where we set `group.id`
@dadelcas: I'd still like to track offsets on each partition if this possible at all
@mark.needham: Pinot does keep track of the offsets in individual segments (which are also mapped to partitions, although the partition value is embedded in the segment name, so it's maybe not exactly what you 're after
@apte.kaivalya: Is there a metric that Pinot exposes for tracking lag in consuming?
@dadelcas: I'd like to know which offset it's been consumed at any time. The information in the segment metadata only provides start and end offsets for the segment. Alternatively I'd like to be able to store the offset and partition with every ingested row
@mark.needham: how about the consuming segment info REST endpoint?
@mark.needham: e.g. ```curl -X GET "
@dadelcas: Thanks Mark, that may do the trick. Just one more question, if I want to include consumer record metadata in my table (let's say headers, offset, etc) is there an easy way to extract this? Any examples you can point me at?
@mark.needham: not that I know of, but lemme have a quick look at the code
@mark.needham: If we can't find anything, you could always write it up as a GH issue -->
@dadelcas: I've seen issue 7004 opens the discussion to store those values as table metadata but the ticket has been opened for 8 months now
@mark.needham: @mayanks I guess
@mayanks: @dadelcas Can you comment on the issue describing your need and +1, it will greatly help.
@dadelcas: Yup, done
@naga.b: @naga.b has joined the channel
@jatink.5251: @jatink.5251 has joined the channel
@jatink.5251: Hello Everyone, Can we ingest data from presto to pinot(offline table). ? Is there any work around on this?
@manish.jaiswal: @manish.jaiswal has joined the channel
#random
@juan: @juan has joined the channel
@praveen82: @praveen82 has joined the channel
@naga.b: @naga.b has joined the channel
@jatink.5251: @jatink.5251 has joined the channel
@manish.jaiswal: @manish.jaiswal has joined the channel
#troubleshooting
@4shivamsajwan: Hi all, I am trying to run Pinot locally by providing zookeeper url as it is recommended. But I am getting this: Exception while reading External view from zookeeper
@4shivamsajwan:
@mark.needham: can you share what commands you've run that lead you to that state? Did you start a controller/broker/server?
@4shivamsajwan: @mark.needham I have setup using these instructions
@mark.needham: oh, that's weird. It seems like the external view is empty for some reason. Are you able to navigate to
@4shivamsajwan: Yes i am able to navigate to brokerResource. here is how it looks
@juan: @juan has joined the channel
@praveen82: @praveen82 has joined the channel
@apte.kaivalya: Hey :wave: , I am facing an issue where after some time (usually a few mins) servers start dying and segments go BAD. These servers are able to join back on restarting, but are DEAD again. In the controller logs I see some errors like: ```Caught 'java.net.SocketTimeoutException: Read timed out' while executing GET on URL:
@richard892: hi I'll take a look and get back to you
@apte.kaivalya: Thanks @richard892
@mayanks: ZK disconnects usually happen if the servers are GC’ing. Can you check on that @apte.kaivalya?
@richard892: @apte.kaivalya you can capture gc pause times without restarting by running `jcmd <controller pid> JFR.start duration=60s filename=controller.jfr settings=profile`
@apte.kaivalya: Thanks @mayanks and @richard892 let me use this and see what I get.
@walterddr: also the error log comes from an external facing API `/table/{tableName}/size` which is usually called by pinot UI if you open the pinot controller link. this doesn't seem to be the root cause but rather an indicator that the server has already gone dead.
@walterddr: were you able to check the server log?
@aditya.patwal: Hi Folks, I am experimenting with pinot merge and rollup task It failed to download the segments from s3. Couldn't find and config reference for minions similar to server `pinot.server.segment.fetcher.protocols` Is it possible to use s3 deep store with minions? What is the config for this?
@richard892: what's your configuration?
@npawar: It's on this page, scroll all the way to the bottom
@aditya.patwal: Thank you! The task was able to download from s3
@aditya.patwal: The merge and rollup worked but got another err ```2022/02/21 16:40:26.018 ERROR [PinotSegmentUploadDownloadRestletResource] [grizzly-http-server-0] Any segments from 'segmentsTo' should not be available in the table at this point``` data returned by brokers is as expected, rows reduced after rollup but older segments still showed up in ideal state Tried deleting all segments and re pushing old segments from s3, but deleting all segments didn't cleared the segment lineage 2 of the old segments are effectively hidden from brokers point of view I'll now try deleting the segment lineage from zk
@naga.b: @naga.b has joined the channel
@jatink.5251: @jatink.5251 has joined the channel
@manish.jaiswal: @manish.jaiswal has joined the channel
@mrpringle: Think I have noticed a bug with how the controller (maybe other components) allocates threads to http requests; it appears to be unbounded and 1 thread per request? Found an issue in pen testing that we can crash the controller by sending in thousands of http get requests to random resources.
#pinot-dev
@jatink.5251: @jatink.5251 has joined the channel
@jatink.5251: @g.kishore can you please suggest? Can we ingest data from presto to pinot(offline table). ? Is there any work around on this?
@g.kishore: @elon.azoulay I know you have built this for trino-pinot, can that be extended to presto-pinot?
#community
@juan: @juan has joined the channel
#announcements
@juan: @juan has joined the channel
#presto-pinot-connector
@juan: @juan has joined the channel
#pinot-perf-tuning
@juan: @juan has joined the channel
#getting-started
@juan: @juan has joined the channel
@trustokoroego: Hello! I need to connect presto to Pinot with basic Auth. Could anyone point point me to how I can set this in the pinot.properties Presto catalog configuration.
@xiangfu0: you can set extra http headers
@trustokoroego: Any reference ? I see in Trino documentation, the pinot connect has an explicit configuration. ```pinot.controller.authentication.type pinot.controller.authentication.user pinot.controller.authentication.password pinot.broker.authentication.user pinot.broker.authentication.password``` Just to confirm if this apply to presto
@trustokoroego: @xiangfu0 could you please give more details on how to apply your suggestion?
@mayanks: If I am guessing right, the HTTP headers to presto will be passed to Pinot @xiangfu0? Or is there a different mechanism here?
@xiangfu0: you can use below config in presto to set any http headers ```pinot.extra-http-headers=k1:v1,k2:v2 ```
@francois: Hi :slightly_smiling_face: Even with the doc I cannot find a proper answer ... Can a dimension table be an hybrid table ?
@mark.needham: a dimension table can only be an offline table. So I think no to hybrid
@francois: Ok . thanks for you feedback
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
