#general
@me1189: @me1189 has joined the channel
@jose.roca: @jose.roca has joined the channel
@graham: @graham has joined the channel
@dovydas: @dovydas has joined the channel
@pabraham.usa: Hello, I tried incubator-pinot helm chart and set replica count for server as 2 . I noticed that all the segments are going to one server and it is used heavily and the other statefulset is kind of doing nothing. Is this some sort of active passive behaviour? Also I scheduled 26G ram with jmx 4G for the server and sometimes the memory reaches max as well. How can i spread the segments and distribute the load? Also which metric can we use for autoscaling?
@g.kishore: do you have both the servers up and running?
@fx19880617: is this realtime or offline table? Can you check current online instances and the table idealstate? If it's realtime, what's the kafka topic partition you have.
@pabraham.usa: @g.kishore I have both servers up and running
@pabraham.usa: @fx19880617 it is a realtime table. I went to pinot controller UI and there it says both server alive. I added one more server to make it 3 and can see some segments going to the third one. But the server don't have much logs though.
@fx19880617: How many Kafka partitions in the topic? Which will translate to the number of consuming segments
@pabraham.usa: Topic only have 1 partition
@fx19880617: I suspect there is only one Kafka partition
@pabraham.usa: I want to maintain ordering
@pabraham.usa: Thats why created single partition
@pabraham.usa: @fx19880617 so if there is single partition then single server will consume it?
@pabraham.usa: What will happen if I create another topic with single partition?
@pabraham.usa: Also I assume the search queries will be served by all the servers?
@fx19880617: I feel the issue is the real-time segment assignment always go to the first server. Let me check the code.
@pabraham.usa: Thanks @fx19880617. if thats the case I can create another topic and stream some of the data there. So another Pinot server will pick it up?
@fx19880617: Yes
@fx19880617: Or add replicas ;)
@fx19880617: If you want redundant
@fx19880617: I will check the segment assignment logic
@pabraham.usa: you mean replication and replicasperpartition?
@pabraham.usa: even if I set that to 2 for instance, still the topic will be consumed by single server right?
@fx19880617: Btw, for ordering, if you have primary key, then you can still have a partitioned Kafka topic
@fx19880617: It will be two servers
@fx19880617: Which means one segment should go to two Pinot servers
@pabraham.usa: ohh ok, make sense. I have primarykey will check kafka side on that.
@fx19880617: I think the issue is that segments are not evenly distributed for one Kafka topic partition table. We will look into this issue .
@pabraham.usa: I am trying to avoid query order by desc
@fx19880617: ic
@fx19880617: But in Pinot query, you still need to do order by
@pabraham.usa: ohh..!! I thought Pinot will maintain order
@fx19880617: Pinot query across multiple segments are not ordering preserved
@pabraham.usa: In my test Pinot created around 100 segments. however seems like ordering is maintained however like you said it could be because the results are pulled from a particular segment
@pabraham.usa: I will try with diff queries and see, also any tips to improve performance when using order by ?
@g.kishore: whats the query type
@g.kishore: query pattern
@pabraham.usa: @g.kishore it is just order by timemillis
@pabraham.usa: i just tried a few and seems it is performing ok
@tymm: hello, where can I find guides/ documentations for pinot-minion?
#random
@me1189: @me1189 has joined the channel
@jose.roca: @jose.roca has joined the channel
@graham: @graham has joined the channel
@dovydas: @dovydas has joined the channel
#troubleshooting
@ken: I was trying out the steps on
@ken: After another 15 minutes or so, the cluster status changed to DELETING, and it’s still that way after almost 3 hours.
@ken: Has anyone else run into this?
@g.kishore: is it happening consistently?
@ken: Don’t know yet, I’ll try again. I was concerned that my client paid for servers that we never got to use (was setting up a cluster of big boxes), but looks like it didn’t get to the point of EC2 instances starting.
@dlavoie: Are you testing on us-east-1? AWS had massive outage today
@ken: We were, yes :disappointed:
@ken: I hope I didn’t cause the outage :slightly_smiling_face:
@dlavoie: ```8:05 AM PST: Kinesis is experiencing increased API errors in the US-EAST-1 Region. This is also causing issues with ACM, Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, CloudWatch, Cognito, Connect, DynamoDB, EventBridge, IoT Services, Lambda, LEX, Managed Blockchain, Resource Groups, S3, SageMaker, Support Console, and Workspaces. This issue has also affected our ability to post updates to the Service Health Dashboard.```
@dlavoie: Best part being : `This issue has also affected our ability to post updates to the Service Health Dashboard`.
@ken: How meta
@dlavoie: I feel you may have been impacted by this :stuck_out_tongue:
@ken: Well, I once did take down the main timeshare system at MIT, by accidentally printing a specific sequence of control characters. So there’s some chance my eksctl command was to blame.
@ken: And thanks for the heads-up, looks like our big Pinot demo will be getting rescheduled for next week.
#docs
@dlavoie: @dlavoie has left the channel
@mayanks: @mayanks has left the channel
#community
@mohammedgalalen056: @mohammedgalalen056 has joined the channel
#pinot-docs
@amitchopra: @amitchopra has joined the channel
@mohammedgalalen056: @mohammedgalalen056 has joined the channel
@vananth22: @vananth22 has joined the channel
@ken: I don’t see any documentation for the new “segmentCreationJobParallelism” setting, other than a reference in the job spec file on
@fx19880617: This is newly added config. I’ll complete the docs. This should be in the ingestion part. Tutorials may give concrete examples there.
@ken: Is there one page that talks about all of the possible values for the job file
@npawar: this is the page where all the spec should be:
@g.kishore: @fx19880617 ^^
@amitchopra: Few doc feedback for Streaming ingestion example 1. For
@chinmay.cerebro: @amitchopra thanks for reporting. will take a look
#config-tuner
@chinmay.cerebro: @chinmay.cerebro has joined the channel
@ssubrama: @ssubrama has joined the channel
@steotia: @steotia has joined the channel
@g.kishore: @g.kishore has joined the channel
@chinmay.cerebro: lets discuss the issue here
@chinmay.cerebro: Current issues with the approach: • Subbu: Once we add a public interface/config, it gets very hard to change/move it around, and hence this discussion. • Sidd: Need to look at current work on RecommendationEngine to figure out what more params do we need for tuner
@chinmay.cerebro: @ssubrama to your question - we're not defining a fixed interface
@chinmay.cerebro: we're defining an annotation type
@chinmay.cerebro: similar to ScalarFunction
@chinmay.cerebro: which is easy to evolve over time
@chinmay.cerebro: you could have multiple tuners with different kinds of parameters over time
@chinmay.cerebro: and it will not break the existing ones
@g.kishore: I thought the public interface is take tableconfig and return tableconfig
@chinmay.cerebro: As pointed out by Subbu and Sidd, we might need other params (eg: query pattern, url to sample segment)
@chinmay.cerebro: depending on type of tuning
@steotia: right
@chinmay.cerebro: My point is we can still do that with the annotation type
@chinmay.cerebro: @ssubrama we don't have to lock in on any one interface
@chinmay.cerebro: for instance, the very basic tuner doesn't care about query patterns
@g.kishore: why is that needed as part of interfaces
@chinmay.cerebro: I can make that change to the TableConfigTunerRegistry to allow that
@steotia: IIUC, there are two high level issues here right? • Injection of the tuner implementation itself (using a factory or annotation) • The interaction of the tuner with a high level interface that eventually results in a tuned config What is the potential flow here? 1. *Input* - table config into a high level interface (something like TableConfigTuner) 2. Custom inputs for the tuner/recommender (the current RecommendationEngine one takes query pattern, schema etc.. can optionally also take prod logs, segment info for the case you want to optimize an existing running use case) 3. Recommender generates output - series of recommendations (output is json format for the current recommender RecommendationEngine) 4. TableConfigTuner consumes the recommendations and applies them to table config 5. *Output* - tuned table config At step 4, do we need the tuners to generate output in a particular format for the applier to consume in step 4. This along with step 2 is the one that I was thinking that might require a set interface and it should be fleshed out now. If this can be flexible, then that is better
@g.kishore: its really input and output
@g.kishore: 2 ,3, 4 are not relevant and we dont want to complicate this
@g.kishore: we can let the tuner derive information from where ever it wants
@steotia: how will the tuner be invoked then?
@steotia: We are not adding a mere placeholder right.. it has to be integrated with the tuner
@g.kishore: it just invokes it every time a table is added
@g.kishore: and then periodically
@steotia: I understand that input and output is table config. I am only suggesting that the code that takes this table config input and generates table config output needs to be able to talk to the tuner. If every tuner has its own way of taking input and generating output then this code will keep changing forever or we will have to go back and change the tuner
@g.kishore: thats not the case right
@g.kishore: when you say taking input which input are you referring to?
@steotia: the input of the tuner in _If every tuner has its own way of taking input and generating output_
@g.kishore: thats just table config right
@steotia: no
@steotia: much more than that
@g.kishore: what is that?
@g.kishore: logs, metrics etc cannot be part of interface
@steotia: the recommendation engine that was recently implemented takes schema, query pattern at the minimum .. can optionally also take prod logs, segment info for the case you want to optimize an existing running use case. So it is meant for both optimal recommendation for a table for the first time and then continuous optimization in production
@g.kishore: its up to the tuner implementation to use what ever it needs
@g.kishore: thats not the right design for interface
@g.kishore: you can take those things as init config params for the tuner
@g.kishore: and do what ever is needed
@g.kishore: its just a map
@g.kishore: linkedin might take ingraph end point for e.g
@g.kishore: open source might take prometheus or something else
@g.kishore: there is no way to design an interface that can cater to all these complex cases
@steotia: wait.. whatever I mentioned is taken in an init file (input json file)
@steotia: so yes it is taken as init config params
@steotia: ```/** * This is the runner class for the rule engine, it parses the input json and maps it to a input manager, * Then according to the _recommend* flags set in the RulesToExecute, the engine will call the corresponding rules * constructed by RuleFactory */ public class RecommenderDriver { private static final Logger LOGGER = LoggerFactory.getLogger(RecommenderDriver.class); private static final String RULE_EXECUTION_PREFIX = "isRecommend"; private static final String RULE_EXECUTION_SUFFIX = "Rule"; public static String run(String inputJson) throws InvalidInputException, IOException {```
@steotia: run() method is called with the input json
@g.kishore: so Tuner interface is really just that right ```Tuner { init(config) Tableconfig apply(Tableconfig) }```
@g.kishore: you can use recommenderDriver in your TunerImplementation
@steotia: yeah this one currently doesn't implement apply method.. we can certainly add The reason why we don't have one is that it is invoked from LinkedIn from nuage UI
@steotia: but yes it can be added
@g.kishore: add apply to recommenderDriver?
@steotia: ok taking a step back here.. when you refer to Tuner interface, is it the engine that is actually doing the tuning or the high level pluggable interface. The current RecommenderDriver doesn't implement an interface as of now
@g.kishore: zoom call?
@g.kishore: its just an interface that will be called periodically and what ever is the output will be set in Helix
@g.kishore: its really that simple
@g.kishore: now you can make recommenderDriver implement Tuner interface and have a init method and apply method
@steotia: yes.. I just wanted to see how recommenderDriver can fit into this
@g.kishore: ok, is it clear now?
@steotia:
@steotia: Sorry for the confusion
@steotia: we can implement any custom tuner and register with TableConfigTunerRegistry
@steotia: implement init and apply
@g.kishore: exactly
@g.kishore: i am sure this we will enhance the annotations
@g.kishore: like add priority etc
@g.kishore: or how often it should be called etc
@g.kishore: but for now it should be really simple
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
