#general
@whatatrip888: hi Recently I was working with pinot group by command . I have loaded 3 million data in data in the table. I executed group by query with limit. If the limit is above 1 lakh, the query returns only 1 lakh records. As per documentations I have changed query limit over riding property to fix this limit issue but it didn't work. The issue is attached as screenshot below. Thanks in advance for the help!
@g.kishore: Looks like it did not reach the group by limit
@g.kishore: What is the cardinality of MSISDN
@whatatrip888:
@whatatrip888: column.MSISDN.cardinality = 1000000
@whatatrip888: select MSISDN from instant_offline LIMIT 1000000 This query works fine. The only issue with group by queries
@g.kishore: Why do you say there is issue with group by? It will return only 1million groups since the cardinality is 1M
@g.kishore: Oh sorry, your limit is 100k
@g.kishore: What’s the variable you set?
@whatatrip888: pinot-broker.conf pinot.broker.enable.query.limit.override=true pinot.broker.query.response.limit=2147483647
@whatatrip888: hi any solution regarding this issue
@g.kishore: This is not the right setting
@npawar: Which documentation did you follow?
@npawar: the setting is `pinot.server.query.executor.num.groups.limit`
@npawar: @jackie.jxt please confirm that this setting is sufficient ^^ ?
@jackie.jxt: Yes, please set `pinot.server.query.executor.num.groups.limit` to a higher value
@jackie.jxt: Default is 100K, if you have only 1 segment, then it will only return 100K groups
@tymm: Hi, can pinot aggregate all the data periodically?
@mohammedgalalen056: You can use "Pinot Minion" which can handle any background task, you can schedule periodical aggregation with it.
@tymm: hi there, how to use this pinot minion? i couldnt find any documentations for it
@jakob.edding: @jakob.edding has joined the channel
@mats.poerschke: @mats.poerschke has joined the channel
@jakob.edding: Hi, does Pinot offer an implementation of Change Data Capture (CDC) for OLAP queries so that updated query results can be obtained automatically? Or an alternative way to be notified of updated indexes so that we can manually retrigger the query?
@mayanks: Not at the moment. Moreover, with real-time ingestion/indexing unsure if that makes sense.
@g.kishore: What are you trying to accomplish?
@jakob.edding: We’re trying to stream Pinot query results as input to a Flink stream processor. We would like to have a new message in that stream either periodically or when the results of the query differ from the last result (in our understanding, that is equivalent to when the underlying indices change that are relevant for the query). As we understand it, we could implement much of the logic for that ourselves using the Pinot Java client but were just wondering if there are certain things in Pinot that could ease this task. :slightly_smiling_face:
@g.kishore: Got it. we dont have that but can be a useful contribution
@g.kishore: happy to help/guide if you are interested in contributing
@farshad: @farshad has joined the channel
@amitchopra: @amitchopra has joined the channel
@divya.sudhakar429: @divya.sudhakar429 has joined the channel
@amitchopra: I am new to Pinot and was trying out some of the examples. One document feedback - i saw that the name of controller created during
@g.kishore: Thanks for the feedback Amit, we will fix it.
@chinmay.cerebro: @amitchopra this is fixed. Thanks so much for reporting
@amitchopra: sure, no problem :slightly_smiling_face:
@ami.oneworld: @ami.oneworld has joined the channel
@ken: Hi all - what’s the difference between <#C01822DR7UP|pinot-docs> and <#C012V7EDCG3|docs>? I’ve found some issues, wondering where to discuss, thanks.
@chinmay.cerebro: Hmm... not sure @g.kishore @mayanks ^^ ?
@mayanks: Not sure either. Let's pick one and stick with it :slightly_smiling_face:?
@mayanks: Seems like we are going with <#C01822DR7UP|pinot-docs>
@chinmay.cerebro: lets deprecate the other one ? @mayanks ?
@chinmay.cerebro: as in just remove
#random
@jakob.edding: @jakob.edding has joined the channel
@mats.poerschke: @mats.poerschke has joined the channel
@farshad: @farshad has joined the channel
@amitchopra: @amitchopra has joined the channel
@divya.sudhakar429: @divya.sudhakar429 has joined the channel
@ami.oneworld: @ami.oneworld has joined the channel
#docs
@chinmay.cerebro: @chinmay.cerebro has joined the channel
@mayanks: @mayanks has joined the channel
@chinmay.cerebro: Folks in this channel - we're most likely going to delete this channel. Please join <#C01822DR7UP|pinot-docs> instead.
@g.kishore: @g.kishore has left the channel
#pinot-docs
@ken: @ken has joined the channel
@ken: Just made a small change to
@g.kishore: @g.kishore set the channel topic: Channel to Pinot docs suggestions/issues/reviews
@chinmay.cerebro: @chinmay.cerebro has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
