#general
@wenlong.xiong: @wenlong.xiong has joined the channel
@erjcan: @erjcan has joined the channel
@erjcan: hi guys, what is the difference btw Apache Kudu and Pinot? use cases?
@erjcan: both seem to be about fast scalable olap analytics, but what is the difference?
@sosyalmedya.oguzhan: Kudu is just a storage engine for fast analytics on fast data. That means you can write data to Kudu in realtime, then query it using the Spark, Impala, Presto etc. Pinot on the other hand is a realtime OLAP datastore includes storage and query engines with low latency. You can compare Kudu with parquet, ORC or another formats but pinot is very different than others.
@mayanks: Thanks @sosyalmedya.oguzhan. Also, to add Pinot can consume data directly from stream systems (eg Kafka) as well as offline, and provide near-realtime analytics for user facing analytics at high throughput (10's of thousands of reads at p99 read latency within tens of millis).
@erjcan: hmm i wonder why then we have apache druid, impala, and apache drill.. i understand it's slack is for Pinot strictly, but still - why create so many seemingly similar analytics platforms?
@mayanks: You can refer to this thread:
@jmeyer: Hello Is it possible to benefit from Pinot's Star-Tree index when performing aggregation queries on known time ranges ? For example, if I know in advance that I will get queries like this : `SELECT SUM(value) FROM values WHERE timeString BETWEEN '2021-01-01' AND '2021-01-08'` (ex: rolling week) Can some configuration of StarTree index precompute this sort of query ? (or even part of it) [I know this looks like a TSDB use case, but still, I'm hopeful :smile:]
@mayanks: You could do that, I'd recommend first checking the perf of out-of-the-box Pinot.
@mayanks: Reason being you'll do a better storage vs latency cost balance.
@jmeyer: Yep, actually I've tried it out of the box, and performance was already very good, but I was thinking ahead in case we need even more (or have greater volume of data) Good to know it's possible ! Thanks @mayanks :slightly_smiling_face:
@mayanks: Is this a realtime only use case?
@jmeyer: Should be mostly batch in that case
@mayanks: I asked because you can also do some pre-aggregation in your batch job. In short, multiple options to explore
@jmeyer: That would be outside Pinot, right ? Or is there some ingestion option for this ?
@mayanks: Good question - There's a pre-aggregate metrics option for RT tables to do it within Pinot. For batch, you could do it more efficiently outside.
@jackie.jxt: You can simply add `timeString` into the star-tree split order and star-tree is able to handle the query
@jackie.jxt: What is the granularity of the time column? Based on the query, may I assume it is in DAYS?
@jmeyer: @jackie.jxt you're right, daily data
@jackie.jxt: Then including `timeString` in the star-tree dimension split order should work well
@jmeyer: I'll keep that in mind, thanks !
@mayanks: Yeah, the out-of-box should already give you a lot of mileage. But as Jackie said, if you need more you have this option.
@erjcan: frankly, i found about Pinot only today, i asked my friend if they use Cassandra, but he told me they use pinot.. :slightly_smiling_face: my question: cassandra is OLTP, but Pinot is olap. did anyone make connection btw pinot and cassandra? my question - how to stress test , load test reading, analyzing from cassandra into pinot?
@mayanks: Nice, that your friends are using Pinot :+1:
@mayanks: What's your source of data for Cassandra? Perhaps it is better to directly consume from there into Pinot?
@erjcan: i wanna do some benchmarking with this combination: cassandra as storage + analytics on pinot
@mayanks: One of the powers of Pinot is its indexing techniques (its own storage component).
@mayanks: If you can provide more details around your use case, perhaps we can suggest better alternatives?
@erjcan: hmm in fact i m only learning cassandra and looking for ways to integrate it with any OLAP system. i dont have any use case now, just wondering about easy of integration btw cassandra and pinot. by the way we are based in Kazakstan lol :slightly_smiling_face:
@mayanks: Nice that Pinot is also reaching Kazakstan :slightly_smiling_face:. So typically, for Pinot, you ingest data either from Realtime streams (eg Kafka) and/or batch (ie offline data push).
@mayanks: As I mentioned, Pinot is fast partly because of its own storage + execution engines. Using another storage system is probably not going to get what you want. I am not sure if I see the value in using Cassandra as storage and Pinot as query execution.
@mayanks: If your end goal is analytics, then better to use just Pinot by itself.
@erjcan: @mayanks u mean integration of casandra and pinot is not a good idea? but where does pinot get data from? or pinot just connects to whatever place where the data is stored?(s3, hdfs)
@mayanks: I am saying I am unsure what the value add is for connecting the two (unless you have strong business case).
@erjcan: to me Casandra is storage, so i assumed that any other apache tool (druid, pinot) can simply connect to it and i can run my olap queries directly from casandra... wrong?
@mayanks: Pinot is also storage.
@mayanks: The question is do you need OLTP storage, or are you good with OLAP storage
@erjcan: @mayanks oh ok, i was just trying to integrate them for the sake of seeing how well they glue together.
@mayanks: If you need OLTP, then many folks use flows like OLTP -> debezium (or similar systems) to push changes to messaging system (kafka) -> Pinot
@erjcan: yeah i should probalby focus on actual biz case thx mayank
@mayanks: yeah, that would define your requirements, and help us suggest better solutions / usage of Pinot
@g.kishore: @erjcan
@g.kishore: you can replace MySQL in that diagram with any other storage - Cassandra, MongoDB etc
@erjcan: @g.kishore hmm thx , where can the kafka connect be placed? in cloud?
@narayanan.nachiappan: @narayanan.nachiappan has joined the channel
#random
@wenlong.xiong: @wenlong.xiong has joined the channel
@erjcan: @erjcan has joined the channel
@narayanan.nachiappan: @narayanan.nachiappan has joined the channel
#feat-text-search
@pingkarthickselvaraj: @pingkarthickselvaraj has joined the channel
#feat-presto-connector
@pingkarthickselvaraj: @pingkarthickselvaraj has joined the channel
#troubleshooting
@wenlong.xiong: @wenlong.xiong has joined the channel
@erjcan: @erjcan has joined the channel
@humengyuk18: I’m getting `Caught exception while transforming the record:` when ingesting realtime stream, it’s a `java.lang.ClassCastException: null` exception, looks like it’s casting a null value to some type, is this a bug?
@mayanks: Without asking more details could be a bug. Are you using your own custom transform? If so, then the bug may be in your custom code. Else, if you can file an issue with more details that would be really helpful.
@narayanan.nachiappan: @narayanan.nachiappan has joined the channel
#pinot-dev
@pingkarthickselvaraj: @pingkarthickselvaraj has joined the channel
@amrish.k.lal: I have been seeing these errors on github Integration Test runs with different test cases which don't seem related to PR changes. I am wondering if this an integration test issue and if there is a workaround? ```Error: Tests run: 9, Failures: 1, Errors: 0, Skipped: 8, Time elapsed: 698.523 s <<< FAILURE! - in org.apache.pinot.integration.tests.FlakyConsumerRealtimeClusterIntegrationTest Error: org.apache.pinot.integration.tests.FlakyConsumerRealtimeClusterIntegrationTest.setUp Time elapsed: 698.359 s <<< FAILURE! java.lang.AssertionError: Failed to meet condition in 600000ms, error message: Failed to load 115545 documents``` ```Error: Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 739.3 s <<< FAILURE! - in org.apache.pinot.integration.tests.BasicAuthRealtimeIntegrationTest Error: org.apache.pinot.integration.tests.BasicAuthRealtimeIntegrationTest.setUp Time elapsed: 725.035 s <<< FAILURE! java.lang.AssertionError: Failed to meet condition in 600000ms, error message: Failed to load 115545 documents ```
@dlavoie: the `BasicAuthRealtimeIntegrationTest` is definately a random flacky test. Haven seen the `FlakyConsumerRealtimeClusterIntegrationTest` fail before though
#pinot-rack-awareness
@jaydesai.jd: @jackie.jxt @g.kishore @ssubrama Can u guys help review the PR :
#minion-improvements
@laxman: adding @singalravi
@singalravi: @singalravi has joined the channel
@laxman: @jackie.jxt /@npawar: Have few questions on tuning/scaling minion. • After schedule REALTIME to OFFLINE conversion job for some table my Minion is going heap OOM. How do I calculate the heap memory requirement for Minion? • Does minion run multiple tasks of different tables at the same time? If yes, is there a way to control the parallelism? • If I have multiple minion instances, does multiple tasks of a single table run simultaneously on those instances? If yes, how the checkpoint in zookeeper for the task is updated safely for REALTIME to OFFLINE conversion job? Can you please shed some light on this area.
@npawar: have you configured that job for many tables?
@laxman: Yes. I configured for 3 tables
@laxman: Right now I have only one instance of minion
@npawar: regarding your 3rd question, for RealtimeToOffline task, only 1 task can run at a time in the cluster per table.
@npawar: since you have 3 tables, 3 tasks could be running in parallel, each for the differet table
@npawar: regarding controlling this parallelism across tables, i’m not sure, will let Jackie answer
@npawar: we have an open issue to address some memory optimization on the realtiemToOffline task:
@npawar: it is a little memory intensive right now
@npawar: in the meantime, you could create smaller segments, by setting the max rows per segment, or by decreasing the bucket time period, or increase the number of minions
@laxman: > regarding your 3rd question, for RealtimeToOffline task, only 1 task can run at a time in the cluster per table. Where is this configured/controlled? I remember seeing this hardcoded in code, but lost track of that. Just curious to know. By design, this should be configurable per task type.
@npawar: for RealtimeToOffline, it is not configurable. By design we want only 1 of them to run at a time for a table
@npawar: we did that to ensure we can process the time ranges in order
@laxman: Yes. I agree, for RealtimeToOffline we need to ensure the processing order.
@laxman: But, is this configurable for other task types where there may not be any order dependency?
@npawar: i believe so, i think the config for that differs task by task. But i may be wrong
@jackie.jxt: There is a method `getNumConcurrentTasksPerInstance()` in `PinotTaskGenerator` to config the concurrency of the task (default 1)
@jackie.jxt: But currently there is no control cross multiple tasks
@jackie.jxt: If you only have `RealtimeToOfflineSegmentsTask` configured, then each minion should only run one task at a time
@npawar: Is that concurrency config across tables? or is it per table?
@npawar: btw Laxman, I will try to pick that open issue today
@jackie.jxt: That is cross table, but per task type
@jackie.jxt: So different task types can run in parallel
@laxman: Thanks so much @jackie.jxt and @npawar.
#complex-type-support
@yupeng: hey, folks, i have a PR to implement the transformer proposed in the design.
@yupeng: plz take a look
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
