#general
@humengyuk18: Hi All, anyone know what’s the buildQueryPlan_Count metric mean in pinot server?
@xiangfu0: This is a timer to measure the latency of query phases, `buildQueryPlan` is just one phase. `buildQueryPlan_Count` is published as the count of the metrics, you can ignore it as you already have query count metrics
@xiangfu0: See `org.apache.pinot.common.metrics.ServerQueryPhase.java` for server query phase references
@humengyuk18: I’m seeing thousands of buildQueryPlan_Count from server but only very few query count from broker, is this normal?
@mayanks: Hmm, for a single server, the count should match the broker. However, if there are multiple servers the aggregation across servers would be large. How many servers do you have?
@grvmakkar: @grvmakkar has joined the channel
@gregsimons84: Hi all is this still the right place for doc updates
@kulbir.nijjer: Yes that should be correct place:
@gregsimons84: Thanks I will join. Issue I saw was content here
@kulbir.nijjer: Thanks @gregsimons84. cc @mayanks
@mrpringle: How are people handling timeseries analytics with pinot? Looking for things like zero filing with last value (step interpolation), summing across interpolated series to get a total by time bucket, get last and first value in a time bucket, moving averages etc. Guess we can go over to python etc and implement but it would be nice for some native analytic functions.
@gregsimons84: I’m not well placed on this one but i was experimenting a little with Apache third eye
@g.kishore: Most end up implementing udf to achieve that.. if you think some of these are useful beyond your use case, please file issue
@mrpringle: Looks like presto has a pretty broad set of aggregation and window functions.
@g.kishore: Yes
@luisfernandez: @luisfernandez has joined the channel
@s.swaminathan: @s.swaminathan has joined the channel
@art.artik: @art.artik has joined the channel
#random
@grvmakkar: @grvmakkar has joined the channel
@luisfernandez: @luisfernandez has joined the channel
@s.swaminathan: @s.swaminathan has joined the channel
@art.artik: @art.artik has joined the channel
#troubleshooting
@anu110195: Hey , was trying to ingest s3 hosted json file using rest api, but getting some exception in controller node
@anu110195: ```2021/08/23 05:26:07.465 ERROR [PinotIngestionRestletResource] [jersey-server-managed-async-executor-0] Caught exception when ingesting file into table: networks3_REALTIME. Cannot ingest file into REALTIME table: networks3_REALTIME java.lang.IllegalStateException: Cannot ingest file into REALTIME table: networks3_REALTIME at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestData(PinotIngestionRestletResource.java:184) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestFromURI(PinotIngestionRestletResource.java:170) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:381) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:819) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:814) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]```
@mayanks: ```.checkState(tableType != null, "Must provide table name with type suffix for table: %s", tableNameWithType);```
@anu110195: I'm sending type as well, (networks3_REALTIME)
@anu110195: is there any way to import data in realtime table apart from realtime ingestion via kafka
@mayanks: You can use other steam systems like Kinesis. Could you share your requirement a bit?
@nadeemsadim: what should be the disk allotment for pinot dev installation.. means for controller, broker , minion .. the disk space allotted for server is 200 GB * 3 pods = 600 GB total .. so what should be the controller , broker , minion disk requirement in proportion to that for 3 pods each on k8s .. @mayanks @xiangfu0 @g.kishore @ssubrama @jackie.jxt
@kangren.chia: +1 would love to see more info on server/broker/controller sizing and tuning
@nadeemsadim: also is ssd recommended only for pinot server or for everything .. also if we are not loading offline segments .. do we really need 3 pods of minion .. and if ram is 8gb * 3 = 24 gb for pinot server.. for broker, controller , minion . .what should be the ram allotment .. same goes for cpu cores if cores for server is 2.5 cores per pod * 3 = 7.5 cores .. how much core allotment will be prudent for controller , broker ,minion
@mayanks: Controller only needs storage to store the pushed segments. Deepstore is recommended for this, not local storage. broker does not need any local storage either
@mayanks: SSD is recommended for Pinot server only
@nadeemsadim: what about minion
@nadeemsadim: and RAM allotment for broker/controller/minion?
@mayanks: Minion as well. CPU/Mem depends on the workload you want to throw at the cluster
@nadeemsadim: is the workload real time data ingested or offline data loaded .. I mean minion is involved in real time ingestion also or only while loading data into offline tables
@grvmakkar: @grvmakkar has joined the channel
@luisfernandez: @luisfernandez has joined the channel
@s.swaminathan: @s.swaminathan has joined the channel
@art.artik: @art.artik has joined the channel
#pinot-dev
@xiangfu0: Hi Pinot devs, I’ve finished this issue
#getting-started
@luisfernandez: @luisfernandez has joined the channel
#pinot-docsrus
@snlee: @snlee has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
