#general
@diogo.baeder: Hi folks! Let me ask a question about SQL queries in Pinot: is it possible to either truncate a table, or at least delete all of its rows? I'm asking this because I want to be able to do that for integration-testing my application. Thanks!
@g.kishore: Yes. You can delete all the segments using rest api call
@mayanks: Hi, yes, you can delete all segments from a Pinot table using rest-api (Swagger).
@diogo.baeder: Ah, got it, thanks! I'll try that :slightly_smiling_face:
@diogo.baeder: It worked just fine, thanks!
@mayanks: Cool, thanks for confirming.
@diogo.baeder: How do I immediately create a new segment after that, though? If I delete all segments, I'm left with no segment where I can put my data as part of my tests...
@g.kishore: See base cluster integration test.. @ken had some ideas of creating a miniPinot cluster for test purpose
@diogo.baeder: Where can I see that? Was that a discussion here in the channel?
@g.kishore: Yes but we use free slack and it has no history
@diogo.baeder: Oh... that's why I couldn't find it then :disappointed:
@diogo.baeder: Anyway, it's already late here and I'll continue this tomorrow. Thanks a lot for the help, guys!
@jain.arpit6: Hi, I am seeing 2 segments in consuming state for my table and rest are online. Is that expected behavior? Also the consumption is very slow after a certain number of records 3M. What could be the reason for this?
@mayanks: Number of segments in `CONSUMING` state should be the same as number of partitions.
@jain.arpit6: Ok
@jain.arpit6: Than its fine... thanks
@ssubrama: If consumption is slow, there can be multiple reasons. Given enough cpu and memory pinot will consume as fast as the producer produces data. So, if you see a mismatch, it is either cpu or memory on the consumer side. Too many partitions in a single host? Too much paging going on? A burst of queries stole cpu from consuming threads?
@mayanks: I think I also remember seeing a discussion recently where there was an issue upstream in the kafka broker(s). So you may want to check on that as well.
@sasha: @sasha has joined the channel
@sandeep.hadoopadmn: Hi Team, We are planning to build Pinot cluster, and the data from Kafka will be 1000 TPS, So what would be the best cluster spec will look like? like how many nodes we required, memory, cores?
@mayanks: Would need a bit more info on this such as read qps, data size, etc.
@sandeep.hadoopadmn: Ack
@yawei.li: @yawei.li has joined the channel
@abhishek: Hi @all can any one please add me to the Thirdeye slack workspace, I am trying to do the setup and the docs look bit old. Facing issues with installation
@g.kishore: @pyne.suvodeep ^^
@pyne.suvodeep: Hi @abhishek Did you find this:
@ruicong.xie: @ruicong.xie has joined the channel
#random
@sasha: @sasha has joined the channel
@yawei.li: @yawei.li has joined the channel
@ruicong.xie: @ruicong.xie has joined the channel
#feat-upsert
@kavin.kuppusamy: @kavin.kuppusamy has joined the channel
#troubleshooting
@girishpatel.techbiz: Hi Team, I have a SaaS service which will keep data globally with data owners across regions. I want to keep their data in respective regions to comply with Data Privacy Laws & Regulations. But I have to run queries globally to do analytics across regions. According to me, there are two ways to do it. 1. Have multi-region Pinot setup with real-time/offline servers localized to specific region. And brokers from across regions will call servers to run queries. 2. Have region wise multiple Pinot setups. Do querying and aggregating result at application level. Any other approach possible ? Which approach looks more feasible and easier to implement ? Thanks !
@richard892: I think you will end up paying for a lot of egress if you do scatter gather queries across regions, so I would lean towards option 2 because it will probably lead to smaller amounts of data crossing regions
@g.kishore: Will you have any requirements to aggregate across the two regions?
@g.kishore: Or will the query have a filter on customerid?
@girishpatel.techbiz: We have requirement of aggregation also across regions.
@g.kishore: how does privacy work in that case?
@girishpatel.techbiz: Various compliance forbid to store data in their geographic boundary but not while reading data.
@girishpatel.techbiz: We can do read time aggregation and serve in any other region.
@g.kishore: got it
@g.kishore: I dont know about your setup, but if cost is not a big factor and if your queries are mostly aggregation then 1 is better
@g.kishore: 2 is a slippery slope and you might have to rewrite the broker code in Pinot
@girishpatel.techbiz: @richard892, option 2 will require designing and development at application level. We will not be able to leverage broker for aggregation. Can prestodb be another option ?
@girishpatel.techbiz: @g.kishore agreed.
@girishpatel.techbiz: For option 1, is there any mechanism to do segment assignment based on partition ?
@g.kishore: yes, you can control the assignment based on partition
@girishpatel.techbiz: Any documentation referring how to do it ?
@g.kishore: prestodb is a good workaround for option 2 but that will still incur the cost assuming prestodb spans across the regions
@g.kishore: so, if you are ok with prestodb, then you should be theoretically ok with option 1 as well
@g.kishore: @jackie.jxt ^^ where are the docs for partition based assigment
@girishpatel.techbiz: For option 1, costing is due to network only or any other factor for costing ?
@jackie.jxt:
@girishpatel.techbiz: My requirement is to keep a partition and it's replica limited to specific region's servers. I couldn't find how to do it in above documentation.
@girishpatel.techbiz: I went through instance assignment also
@jackie.jxt: I believe the requirement is to do partition assignment across multiple clusters. That has to be controlled on the client side by pushing different segments to different cluster
@jackie.jxt: Segment assignment can only manage the assignment within a single cluster. I don't know if it is possible to maintain a multi-region cluster, and all of them sharing the same controller
@g.kishore: @jackie.jxt this is for option 1 - Have multi-region Pinot setup with real-time/offline servers localized to specific region. And brokers from across regions will call servers to run queries.
@jackie.jxt: Understood, but the segment assignment can only be applied to single cluster. It cannot assign segments across multiple clusters
@girishpatel.techbiz: Can we setup single multi-region cluster ?
@mayanks: @jackie.jxt I think there's just one Pinot cluster, but servers are located in different regions. Also, the requirement (if I am reading correctly) is that servers on a specific region host specific segments
@jackie.jxt: The potential problem of having one cross region cluster is the latency between controllers and instances in other region, also the ZK
@jackie.jxt: If the multi-region cluster setup works, then segment assignment can assign segments based on the partition id to different regions
@mayanks: Yeah, there will be some latency and egress cost. but barring that, if we use partition based segment assignment such that data for a region goes to servers for that region, isn't that what we need here?
@girishpatel.techbiz: Yes, that is the requirement.
@girishpatel.techbiz: @jackie.jxt how can we assign segment based on partition id ? Is there any documentation for that ?
@jackie.jxt: The segment and instance assignment doc you just posted
@jackie.jxt: You might need to manually set the `InstancePartitions` (mapping from (partition id & replica id) to servers)). Pinot won't know which server is in which region automatically
@girishpatel.techbiz: Need little direction here, how to set `InstancePartitions` ? Can I set any configuration or need to do something else ?
@mayanks: @jackie.jxt ^^
@jackie.jxt: Here is the rest api signature: ``` @PUT @Produces(MediaType.APPLICATION_JSON) @Path("/tables/{tableName}/instancePartitions") @Authenticate(AccessType.UPDATE) @ApiOperation(value = "Create/update the instance partitions") public Map<InstancePartitionsType, InstancePartitions> setInstancePartitions( @ApiParam(value = "Name of the table") @PathParam("tableName") String tableName, String instancePartitionsStr)``` You may check the `InstancePartitions.java` for details
@tamas.nadudvari: Hello, we’re trying to use pool based assignment. We have three availability zones, a server group in each with different pool tags. The table replication setting is 2. Our issue is that we either have a three sized replica group and the segments will be replicated to each pool (unnecessary increasing the table replication) or a 2 sized replica group but then one of the server groups won’t have any segments to serve. Any idea for a configuration where each server groups are used and the replication of the table remains 2?
@tamas.nadudvari: this is the configuration when we end up increased replication: ```"instanceAssignmentConfigMap": { "OFFLINE": { "tagPoolConfig": { "tag": "DefaultTenant_OFFLINE", "poolBased": true, "numPools": 0 }, "replicaGroupPartitionConfig": { "replicaGroupBased": true, "numInstances": 0, "numReplicaGroups": 3, "numInstancesPerReplicaGroup": 0, "numPartitions": 1, "numInstancesPerPartition": 2 } } },```
@g.kishore: Would you mind describing the problem you are trying to solve
@g.kishore: Pool based is a bit advanced feature
@tamas.nadudvari: We’re running Pinot in Kubernetes, and we’d like to achieve zero downtime upgrade. Thus we created independent pinot server Kubernetes deployments for each availability zone, all of them are members of the same Pinot cluster. We’d like to create a table configuration where a segment and its replica are in a different availability zone(/Kubernetes deployment) in any case. We’re trying to use pool based assignments for this. Servers in the same zone are in the same pool. But so far we were only able to create configs where either a segments are available in every zone which increases our desired replica size for the table or using only two of the server groups leaving one server group completely unused.
@diogo.baeder: Hi folks, I'd like to ask for a bit of help here: I'm still struggling to be able to run integration tests between my app and Pinot. What I need is to be able to start each test with a clean slate, while also making sure I have the tables I need created. I tried deleting the tables and recreating them, tried deleting all segments then creating the tables afterwards (hoping a new segment would automatically be created), and none of these methods are working. How can I accomplish what I need?
@diogo.baeder: Also another problem I'm facing is, if I try to delete the table and then recreate it, and then add data to it, it seems like the table was never deleted, because in the "Incubator" (the Pinot admin UI) I can see all the previously created rows.
@g.kishore: Delete happens asynchronously.. you will have to wait in your code for query to return zero records
@diogo.baeder: So, how would you put all of this together, in steps? Would this make sense?: 1. Delete all tables 2. Wait a few secs 3. Recreate all tables 4. Wait a few secs 5. Start running the test (publishing rows and querying them afterwards)
@g.kishore: I would not make it wait based on time
@g.kishore: Instead wait for count * queries to return 0
@g.kishore: See baseclusterintegrationtest in Pinot code
@g.kishore: It has everything you need
@diogo.baeder: Nice, I'll look for that. Thanks man!
@mayanks: Also, if your tests are table specific, then you can create different tables for different tests in the same cluster. That way one test doesn't have to wait for the previous test to complete and delete the table.
@diogo.baeder: Got it. It's going to be 2 tables total, actually, we're going to use it for 2 parts of our internal analytics, so I'll work on having them cleaned up before each test. Thanks!
@diogo.baeder: Alright, so, with this:
@g.kishore: you can use beforeSuite for setup then
@g.kishore: in each setup for testcase you can delete the segments if table exists and wait for the count to go to zero
@diogo.baeder: Hmmm... not sure I understand that last part ("delete the table if it exists and wait for it go to zero"), but how about I: 1. Delete the table 2. Keep listing tables until my table doesn't exist 3. Create the table 4. Keep counting until I get result zero
@g.kishore: edited - in each setup for testcase you can delete the segments if table exists and wait for the count to go to zero (edited)
@g.kishore: I think your flow may not work because the segments might still be on the servers
@g.kishore: delete segments is asynchronous
@diogo.baeder: Hmmm... got it... but after I delete all the segments, how do new ones get created? Last time I deleted all the segments in the table I got no new ones created...
@sasha: @sasha has joined the channel
@yawei.li: @yawei.li has joined the channel
@ruicong.xie: @ruicong.xie has joined the channel
#pinot-dev
@girishpatel.techbiz: @girishpatel.techbiz has joined the channel
#thirdeye-pinot
@kavin.kuppusamy: @kavin.kuppusamy has joined the channel
#getting-started
@girishpatel.techbiz: @girishpatel.techbiz has joined the channel
#pinot-rack-awareness
@lars-kristian_svenoy: @lars-kristian_svenoy has joined the channel
@lars-kristian_svenoy: :eyes: Found this channel in a random mailing list when trying to figure out how to enable rack awareness for pinot + kafka.. Is this feature available?
@g.kishore: Hi Lars, you can ask this in troubleshooting
#fix-numerical-predicate
@kavin.kuppusamy: @kavin.kuppusamy has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
