#general
@rsibanez89: @rsibanez89 has joined the channel
@erik.bergsten: @erik.bergsten has joined the channel
@erik.bergsten: Question about tiered storage+deep storage. We want to use pinot to query mostly fresh data (last week or so) and then store data long term on s3. We have deployed pinot in our k8s cluster using the helmchart and configuring the controller to backup to s3 seems to work. If I understand it correctly this is only to backup data which is also stored on local disks however and we want to keep the local disk usage low! Can we use tiered storage and configure one server to use s3 and another server to use local disks?
@g.kishore: Hi Erik, this feature is only available in StarTree version of Pinot.
@erik.bergsten: Oh! So these pages in the docs
@g.kishore: The concept is the same.. open source supports hdfs ssd, ebs etc as tiers
@g.kishore: StarTree version adds S3 as a tier
@nizar.hejazi: @g.kishore What about using S3 for deep storage (not tiered storage), is it also only supported in StarTree version?
@g.kishore: no, that is available in OSS version as well
@ysuo: when I run an explain plan for query multiply times, I got different query results. And the *totalDocs* number is different. A very simple query like explain plan for select * from table_name. Any idea why is that?
@richard892: hi can you share the plans?
@mayanks: also cc: @amrish.k.lal
@mathew.pallan: @mathew.pallan has joined the channel
@dh258: @dh258 has joined the channel
@ralph.debusmann967: @ralph.debusmann967 has joined the channel
#random
@rsibanez89: @rsibanez89 has joined the channel
@erik.bergsten: @erik.bergsten has joined the channel
@mathew.pallan: @mathew.pallan has joined the channel
@dh258: @dh258 has joined the channel
@ralph.debusmann967: @ralph.debusmann967 has joined the channel
#troubleshooting
@rsibanez89: @rsibanez89 has joined the channel
@erik.bergsten: @erik.bergsten has joined the channel
@deemish2: Hi Team , I am using pinot-0.10.0 to execute pinot ingestion job with avro file and it gives error like :- Caused by: java.lang.UnsupportedOperationException: Unsupported Avro type: NULL It is working fine with json file. can anyone please help into this issue?
@tisantos: Hi Deepak, would you happen to have the avro schema of your input data? Seems like there might be an invalid field with a type `NULL`.
@deemish2: Schema is same for json and avro
@deemish2: It is getting problem on this field { “name”: “countryInfo”, “dataType”: “STRING” },
@deemish2: In data there is no value , It should come with default value - NULL after populating data in pinot table
@tisantos: The exception is thrown for the `viceusage` field. Can you confirm what the avro type is for that field?
@deemish2: { “name”: “viceusage”, “dataType”: “STRING” }
@deemish2: error is coming where the datatype - STRING and if there is no data of this field
@tisantos: This exception is happening in the part of the code the is purely parsing your Avro schema. If the `viceusage` field can be null, then it should be an avro UNION type of `["null", "long"]` . I don't think that is currently the case.
@mathew.pallan: @mathew.pallan has joined the channel
@mathew.pallan: Hi, I am evaluating Apache Pinot and wanted to understand the deep storage options while deploying on Azure. From the docs, it seems like Azure blob is not supported and Azure Data Lake Storage has to be used as the deep storage. Can you please confirm on the same. Also is PinotFS the abstraction used for deep storage as well. The docs mention PinotFS in the context of importing data and hence this query.
@g.kishore: That’s right.. only adls implementation is available as of now.. PinotFs is used for deep storage abstraction as well.. you can look at adsl implementation and add/contribute azure blob storage FS implementation
@mark.needham:
@mark.needham: This is the Azure data lake implementation -
@mathew.pallan: Thanks @g.kishore @mark.needham for the quick response
@mathew.pallan: @g.kishore I got a bit confused with the comment above that says - PinotFs is used for deep storage abstraction as well. Does it imply that there are other abstractions as well and we PinotFS is the new approach(or one of the approaches) to abstract deep storage?
@g.kishore: my bad, pinotFS is the abstraction to interact with deep storage.
@mathew.pallan: Thanks, clear now :thumbsup:
@mayanks: @mathew.pallan To add more context, ADLS gen2 is built on top of ABS (if I am not wrong). And it has better abstraction in terms of what PinotFS needs, so we went with ADLS instead of ABS.
@elon.azoulay: Hi, if we make a change to the controller app, how do we build it? i.e. `npm install` and `npm run build`?
@elon.azoulay: Want to do a pr, but it seems that some additional steps need to be done first. If you have a chance lmk. Thanks!
@dh258: @dh258 has joined the channel
@dh258: Hi All, While deploying Pinot
@diana.arnos: You need to overwrite the `image` property in the values file: ```image: repository:
@dlavoie: ```--set zookeeper.image.registry=<your-registry> \ --set zookeeper.image.repository=<your-repository>``` to be specific about zookeeper. The dependend chart has the following default values: ```## Bitnami Zookeeper image version ## ref:
@dlavoie: @dh258 ^^
@dh258: With these changes do I need to put all images used in pinot install in my registry or just zookeeper?
@dlavoie: Pinot is using a a child chart for zookeeper. You can configure the nested helm chart with `zookeeper.x` values
@dh258: Can you be more specific what changes I need to make where?
@ralph.debusmann967: @ralph.debusmann967 has joined the channel
@luisfernandez: does anyone know why i may get the following error from the broker `"message": "2 servers [pinot-server-1_O, pinot-server-0_O] not responded"` I just changed the version of my broker trying to downgrade the entire system to `0.9.3`
@luisfernandez: anyone familiar with this error? ```"message": "QueryExecutionError:\njava.lang.RuntimeException: Caught exception while running CombinePlanNode.\n\tat org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:146)\n\tat org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:41)\n\tat org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:296)\n...\nCaused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException\n\tat java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)\n\tat org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:135)\n\t... 15 more\n...\nCaused by: java.lang.NullPointerException\n\tat org.apache.pinot.core.util.trace.TraceContext.registerThreadToRequest(TraceContext.java:140)\n\tat org.apache.pinot.core.util.trace.TraceCallable.call(TraceCallable.java:41)\n\t... 8 more",```
@luisfernandez: btw a restart of servers made this go away but i’m curious why it happened on the first place lol
#getting-started
@rsibanez89: @rsibanez89 has joined the channel
@erik.bergsten: @erik.bergsten has joined the channel
@mathew.pallan: @mathew.pallan has joined the channel
@dh258: @dh258 has joined the channel
@ralph.debusmann967: @ralph.debusmann967 has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
