#general
@juraj.pohanka: @juraj.pohanka has joined the channel
@sriramdas.sivasai: @sriramdas.sivasai has joined the channel
@neilteng233: Hey, if I want to see the execution plan of a query. Where can I see that? Is the trace info the execution plan? I have a query like this ```select * from table where DATETRUNC('day',updatedAt,'MILLISECONDS')>=1624492800000``` and I want to know whether it use the range index or not on the updatedAt column. ```"traceInfo": { "pinot-server-1.pinot-server-headless.analytics-prod.svc.cluster.local": "[{\"0\":[{\"SelectionOnlyCombineOperator Time\":1},{\"InstanceResponseOperator Time\":1}]},{\"0_0\":[{\"ExpressionFilterOperator Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator Time\":0},{\"ExpressionFilterOperator Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator Time\":1}]}]", "pinot-server-0.pinot-server-headless.analytics-prod.svc.cluster.local": "[{\"0\":[{\"SelectionOnlyCombineOperator Time\":3},{\"InstanceResponseOperator Time\":3}]},{\"0_0\":[{\"ExpressionFilterOperator Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator Time\":1},{\"ExpressionFilterOperator Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator Time\":0},{\"ExpressionFilterOperator Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator Time\":0}]}]" ```
@gqian3: Hi, we have a table planned to migrate to Pinot, it has only dimension fields and one datetimefield, the data volume is nearly constant, we mainly use it as a look up table, is this a good candidate to migrate to Pinot to gain any query latency benefit from Pinot for this table?
@karinwolok1: Big hello and welcome to all our newest community members! :wine_glass: @specsek @hsaini @vlum @juraj.pohanka @sriramdas.sivasai @srikanth @evan.galpin @jiangok2006 @dougdeu @jkim0110610 @leeon2013 @s.azimigehraz @bharath.sbk @huzongxing0826 @hemanga.borah @b.barisercan @gmr.sql @anindabhattacharjee12 @qun0929 @atri.sharma @thiagopsnfg @ravinder2021.kr @baluvignesh @aaronlevin @jadami @dang @ojasmulay @mail765 @nithinbharadwajd661 @qianbo.wang @nishanth @hamoop @prasanna.gsl @hashhar @azri Would love to. know who you are and what brought you here! :smiley: Also, don't miss our event TODAY with presentations and panels from experts from companies like LinkedIn, Microsoft, DoorDash, Stripe, Confluent, Decodable and more! *Starts in 2 hours!* There will also be a great demo of StarTree Cloud! :tada: :partying_face: We'd also love your support, please share this event with any of your friends who are also *obsessed* with *SPEED* and *REAL-TIME-ANALYTICS!* :zap:
@juraj.pohanka: Thank you @karinwolok1. Glad to be a part of a very vibrant community of a project with huge potential! :slightly_smiling_face:
@karinwolok1: We're so glad to have you!!!!! :smile:
@karinwolok1: See you today?
@juraj.pohanka: Definitely :)
@karthick.nitt: @karthick.nitt has joined the channel
@mark: @mark has joined the channel
@jai.patel856: I have some questions about Pinot realtime/upsert tables: 1. Retention. We have an upsert table whose retention is set to 10 days. However, I’m seeing “latest” rows where the value is 14 days old. Is the cleanup process “lazy”? 2. Although not required are there any advantages/disadvantages to the time column being the same as the sorted column? Am I correct to understand that the recommendation is to only have one sorted column?
@jai.patel856: @jackie.jxt
@jackie.jxt: 1. The retention management is on segment level, so the segment won't be removed if any record within the segment is not expired 2. Having a column sorted can accelerate the filtering and benefit from the data locality. Pinot does not support secondary sorted column yet, so usually it is hard to sort on multiple columns
@jai.patel856: okay, thanks. we’ll keep the sorted index same as the time column.
@jai.patel856: thanks.
@jackie.jxt: Based on your use case, you might want to choose sorted column differently. E.g. if you always filter on a key column (e.g. `memberId`), you might want to sort on it to benefit from fast filtering and data locality
@jackie.jxt: If you always have a range filter on time, then sorting on time column is also a good option
@neilteng233: Hi, if I do a hot-update on the table config of a real time table. What will happen to the old index will they be cleaned up automatically?
@mayanks: What kinde of update?
@neilteng233: I change one col in the dimensionsSplitOrder of the star-tree index.
@mayanks: Hmm, I am not sure if reload will rebuild the star tree in this case. @jackie.jxt?
@neilteng233: I guess hot update on the forward index and sorted is not allowed. But I am not sure about the bitmap inversed index, star-tree and others who are not directly related to the how the data sit on the disk.
@jackie.jxt: Yes, the reload should drop the old star-tree and create the new ones
@neilteng233: awesome. Thank you
@neilteng233: btw, can I see the trace info as the execution plan of pinot?
@jackie.jxt: You may enable the trace on the query console, and the trace can be found in the query response metadata
@jackie.jxt: It tracks the time spent within each operator
@ajithreddy118: @ajithreddy118 has joined the channel
@canerbalci: Curious if there are folks from Slack here. Maybe Slack would be kind enough to donate a Pro tier for Pinot Slack workspace so the message history could be kept for a bit longer.
@kennybastani: Thanks for the recommendation! @srini gave us some extra details for making this happen. Will check back in the next week.
#random
@juraj.pohanka: @juraj.pohanka has joined the channel
@sriramdas.sivasai: @sriramdas.sivasai has joined the channel
@karthick.nitt: @karthick.nitt has joined the channel
@mark: @mark has joined the channel
@ajithreddy118: @ajithreddy118 has joined the channel
#feat-presto-connector
@juraj.pohanka: @juraj.pohanka has joined the channel
#pinot-helix
@juraj.pohanka: @juraj.pohanka has joined the channel
#troubleshooting
@juraj.pohanka: @juraj.pohanka has joined the channel
@patidar.rahul8392: How realtime backward-Compatible(new column) works in Pinot. I have some events in Kafka and created one pinot realtime table .next time I have again pushed some data in Kafka with 1 new field and again executed AddTable command with updated schema(new column added in schema) and then on pinot ui did reload all segments.on ui in schema I am able to see that new column but when I am doing select * from tablename the newly added column is not showing there. Here I have added city as new column. @jmeyer
@jackie.jxt: Please refer to this doc:
@jackie.jxt: In order to reload consuming segment, you need to set a flag in the server config `pinot.server.instance.reload.consumingSegment`
@jackie.jxt: It is disabled by default in version `0.7.1`
@patidar.rahul8392: I'm able to see other column data only for newly added column it's not showing any data.this is my configuration file. Kindly suggest.
@xiangfu0: old data are not refilled as pinot won’t consume old data again.
@sriramdas.sivasai: @sriramdas.sivasai has joined the channel
@syedakram93: [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.094 s [INFO] Finished at: 2021-06-24T19:02:06+05:30 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project pinot-spi: Compilation failure [ERROR] javac: invalid target release: 11 [ERROR] Usage: javac <options> <source files> [ERROR] use -help for a list of possible options [ERROR] [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1]
@jackie.jxt: Please use java 11 to compile the project
@syedakram93: ok, but i cant use java 11 to run this pinot
@syedakram93: BTW, earlier using jdk8, it was success
@jackie.jxt: Yeah, we recently change the default jdk to 11. You can also use jdk 8 to build adding this option `-Djdk.version=8`
@syedakram93: while running maven, shall i use `-Djdk.version=8` ?
@jackie.jxt: Yes. `mvn clean install ... -Djdk.version=8`
@jackie.jxt: More context here:
@syedakram93: ya working , thanks
@syedakram93:
@jackie.jxt: Yeah, thanks for pointing that out
@syedakram93: Exception while creating a build using source
@syedakram93:
@syedakram93: Apache Maven 3.6.3 Maven home: /usr/share/maven Java version: 1.8.0_292, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre Default locale: en_IN, platform encoding: UTF-8 OS name: "linux", version: "5.8.0-55-generic", arch: "amd64", family: "unix"
@syedakram93: can someone help?
@syedakram93: stack trace: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project pinot-spi: Compilation failure javac: invalid target release: 11 Usage: javac <options> <source files> use -help for a list of possible options at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
@karthick.nitt: @karthick.nitt has joined the channel
@mark: @mark has joined the channel
@ajithreddy118: @ajithreddy118 has joined the channel
#onboarding
@karthick.nitt: @karthick.nitt has joined the channel
#custom-aggregators
@juraj.pohanka: @juraj.pohanka has joined the channel
#docs
@juraj.pohanka: @juraj.pohanka has joined the channel
#aggregators
@juraj.pohanka: @juraj.pohanka has joined the channel
#pinot-dev
@sriramdas.sivasai: @sriramdas.sivasai has joined the channel
@mark: @mark has joined the channel
@srikanth: @srikanth has joined the channel
@santosh.reddy: Can we have multiple zookeeper for pinot?
@ssubrama: This is more a question for zookeeper, but yes. Zookeeper by itself support multiple servers, managing quorum. They suggest an odd number of them.
#announcements
@mark: @mark has joined the channel
#pinot-docs
@karthick.nitt: @karthick.nitt has joined the channel
#aggregate-metrics-change
@juraj.pohanka: @juraj.pohanka has joined the channel
#pinot-perf-tuning
@juraj.pohanka: @juraj.pohanka has joined the channel
#getting-started
@sriramdas.sivasai: @sriramdas.sivasai has joined the channel
@mark: @mark has joined the channel
@srikanth: @srikanth has joined the channel
#pinot-docsrus
@karthick.nitt: @karthick.nitt has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
