#general
@phanstasmal: @phanstasmal has joined the channel
@mrpringle: is there any example controller.conf file or a generator, all I really want to do is change the port number but now I have a conf file it keeps asking for more and more properties e.g zookeeper address
@fx19880617:
@mrpringle: thanks
@amitbsuryavanshi: @amitbsuryavanshi has joined the channel
@josefarf: @josefarf has joined the channel
@yohan: @yohan has joined the channel
@david.fachini: @david.fachini has joined the channel
@karinwolok1: Here's the recording from yesterday's meetup if anyone is interested! :slightly_smiling_face:
@rkanumul: @rkanumul has joined the channel
@kseniase: @kseniase has joined the channel
#random
@phanstasmal: @phanstasmal has joined the channel
@amitbsuryavanshi: @amitbsuryavanshi has joined the channel
@josefarf: @josefarf has joined the channel
@yohan: @yohan has joined the channel
@david.fachini: @david.fachini has joined the channel
@rkanumul: @rkanumul has joined the channel
@kseniase: @kseniase has joined the channel
#feat-presto-connector
@josefarf: @josefarf has joined the channel
#pql-2-calcite
@josefarf: @josefarf has joined the channel
#feat-upsert
@josefarf: @josefarf has joined the channel
#troubleshooting
@phanstasmal: @phanstasmal has joined the channel
@amitbsuryavanshi: @amitbsuryavanshi has joined the channel
@josefarf: @josefarf has joined the channel
@yohan: @yohan has joined the channel
@david.fachini: @david.fachini has joined the channel
@laxman: I got some time to do more analysis on this today. Found the root cause. Posting my analysis here as several others may hit into the same problem in k8s environment where you try to control the total memory of a pod. *Background (Direct buffers and their garbage collection)* • P*inot Broker* has no explicit requirement (Like off-heap cache) for huge Direct memory. In Broker, direct memory is used only in netty layer (socket nio) • Direct buffers are collected generally on Full GC. When it reaches the max limit, JVM triggers a full GC even if heap is not full • To avoid this full GC, different JVM vendors has different proprietary and vendor-specific mechanisms to release/collect the direct buffers. • Several open source projects tend to implement (including hadoop, netty and Pinot) hacks to use these internal implementations (Example: sun.misc.Unsafe) to clean up the direct buffers. (io.netty.buffer.PooledByteBufAllocator, io.netty.util.internal.CleanerJava9, org.apache.pinot.core.util.CleanerUtil) *Root cause* Direct buffers not getting collected as JVM flag DisableExplicitGC is set. • When direct memory is full, JVM triggers a full gc System.gc() • On other hand, we disabled explicit gc via JVM flags (-XX:+DisableExplicitGC) • Which means System.gc() is ignored completely and equivalent to a NOOP • Throws OOM as there is no free direct memory. cc: @g.kishore @suraj
@g.kishore: Can you please post this in an issue?
@laxman: I don’t think this is a pinot issue at all. This flag is being set explicitly on prod. I just posted this analysis here as most of the k8s users may fall into this trap.
@g.kishore: This is amazing!
@g.kishore: Are we setting that JVM flag in our helm chart or it’s something you had on your side?
@laxman: This is set in our setup only. I double checked pinot codebase. I don’t see this flag being set in pinot helm charts.
@g.kishore: thanks. We suggest keeping the jvm flags minimal G1GC and maxgcpause to 200ms, these are the only two things we have at LinkedIn as well
@dlavoie: I don’t recall seing this flag on the default args of the helm chart.
@rkanumul: @rkanumul has joined the channel
@rkanumul: Hi Folks, I’m a newbie on pinot.. trying to add a new dependency to support service principal based access to ADL gen2 ``` <dependency> <groupId>com.azure</groupId> <artifactId>azure-identity</artifactId> <version>1.2.2</version> </dependency>``` But It’s giving convergence errors ```[WARNING] Dependency convergence error for org.codehaus.woodstox:stax2-api:3.1.4 paths to dependency are: +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.azure:azure-core:1.12.0 +-com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.9.8 +-org.codehaus.woodstox:stax2-api:3.1.4 and +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.azure:azure-core:1.12.0 +-com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.9.8 +-com.fasterxml.woodstox:woodstox-core:5.0.3 +-org.codehaus.woodstox:stax2-api:3.1.4 and +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-org.linguafranca.pwdb:KeePassJava2:2.1.4 +-org.linguafranca.pwdb:KeePassJava2-simple:2.1.4 +-com.fasterxml:aalto-xml:1.0.0 +-org.codehaus.woodstox:stax2-api:4.0.0 [WARNING] Dependency convergence error for com.nimbusds:oauth2-oidc-sdk:7.4 paths to dependency are: +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.microsoft.azure:msal4j:1.8.0 +-com.nimbusds:oauth2-oidc-sdk:7.4 and +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.nimbusds:oauth2-oidc-sdk:7.1.1 [WARNING] Dependency convergence error for com.microsoft.azure:msal4j:1.8.0 paths to dependency are: +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.microsoft.azure:msal4j:1.8.0 and +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.microsoft.azure:msal4j-persistence-extension:1.0.0 +-com.microsoft.azure:msal4j:1.4.0 [WARNING] Dependency convergence error for net.java.dev.jna:jna-platform:5.5.0 paths to dependency are: +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-com.microsoft.azure:msal4j-persistence-extension:1.0.0 +-net.java.dev.jna:jna-platform:5.5.0 and +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-net.java.dev.jna:jna-platform:5.6.0 [WARNING] Rule 1: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message: Failed while enforcing releasability. See above detailed error message.``` I tried few different things but none works What do we do in this case ? Thank you
@fx19880617: I think you need to exclude the conflict dependency and declare it explicitly in pom.xml
@rkanumul: I tried to declare the conflict explicitly.. that didnt help.. can you tell more about the exclude ? appreciate if u can help me try that out
@wrbriggs: Another option is to use a `dependencyManagement` block in the pom, to override the various differing transitive dependency versions. This can be less invasive than trying to individually exclude + then explicitly re-add the necessary dependencies.
@jlli: e.g. exclude `org.codehaus.woodstox:stax2-api` from pinot-adls, since it uses a lower version
@rkanumul: Thank you Will.. Ill try that out and update here..
@fx19880617: ``` <dependency> <groupId>com.azure</groupId> <artifactId>azure-storage-file-datalake</artifactId> <version>12.0.0-beta.12</version> <exclusions> <exclusion> <groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>com.azure</groupId> <artifactId>azure-identity</artifactId> <version>1.2.2</version> <exclusions> <exclusion> <groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId> <version>4.0.0</version> </dependency>```
@fx19880617: just one example
@wrbriggs: Or this, if the enforcer will let you (it’s been a while since I dealt with the Maven enforcer plugin, as I use sbt or mill for most of my builds now): ```<dependencyManagement> <dependencies> <dependency> <groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId> <version>4.0.0</version> </dependency> </dependencies> </dependencyManagement>```
@rkanumul: Will try out one after the other until it works for me :smile:
@kseniase: @kseniase has joined the channel
#pinot-docs
@josefarf: @josefarf has joined the channel
#pql-sql-regression
@josefarf: @josefarf has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
