Apache Pinot Daily Email Digest (2021-09-06)

[Pinot Slack Email Digest]

#general

@diwa414: @diwa414 has joined the channel
@shish: *# Kubernetes deployment* 1. git clone 2. cd pinot/kubernetes/helm 3. helm install -n pinot-test pinot . `Error: Chart.yaml file is missing` Should I run `helm install -n pinot-test pinot ./pinot` ? Or I am making any mistake in above steps which are mentioned in documentation
  @xiangfu0: you can go to the pinot directory then run this
  @shish: Okay thanks
  @xiangfu0: where you saw this doc?
  @shish:
  @xiangfu0: thanks, will fix the doc
  @shish: Okay cool
@itdev: @itdev has joined the channel
@samsarode89: @samsarode89 has joined the channel
@yogesh.dandawate: @yogesh.dandawate has joined the channel

#random

@diwa414: @diwa414 has joined the channel
@itdev: @itdev has joined the channel
@samsarode89: @samsarode89 has joined the channel
@yogesh.dandawate: @yogesh.dandawate has joined the channel

#troubleshooting

@diwa414: @diwa414 has joined the channel
@wentjin: Hi team, there is a performance issue when using ‘JSONPATH’ transformation functions, this is may be caused by the jayway Inefficient LRUCache design. When encountering high concurrency in data ingestion, there will be fierce competition for CPU and all ingestion threads are waiting for the lock, and data consumption will be delay.
  @richard892: Hi @wentjin, good find :+1: In order to confirm your suspicion that the problem is due to contention, it would help to disable the cache entirely. I'm new to pinot, but I took a look at the jayway library and it looks like you can override the cache implementation by calling ```com.jayway.jsonpath.spi.cache.CacheProvider.setCache(new NOOPCache());``` but you need to do it before anything else in the same `ClassLoader` calls `CacheProvider.getCache` to make sure the registration works. To really confirm this is related to contention, it would be great to get a contention profile from async-profiler _without overriding the cache provider_. You can run it as a native agent as outlined , and use this JVM setting: ```-agentpath:/path/to/libasyncProfiler.so=start,event=lock,file=contention.html``` (Or set ```-agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=cpu.html``` to get a cpu profile which should identify the bottleneck without proving the issue is contention) Getting a look at the flamegraph will really help to understand whether this is contention or just slow JSON path evaluation.
@itdev: @itdev has joined the channel
@samsarode89: @samsarode89 has joined the channel
@yogesh.dandawate: @yogesh.dandawate has joined the channel

#pinot-dev

@itdev: @itdev has joined the channel
@ken: I created a PR (), but two of the tests fail. When I look at the logs, the failures seem unrelated to my change (one seems to be a ZkClient connection issue, the other is a timeout during an operation). Is this common for PR builds? What’s the right way to confirm my change didn’t cause the problem(s)?
@ken: I’ve been working on a logging issue when building segments using Hadoop - the expected logging output from the map task isn’t showing up. While poking around the jars on the classpath, I noticed that many of the plugins include logging implementations, which can mess with the Log4J-based Hadoop logging configuration. It seems odd that plugins would including logging classes in their builds, as the pinot-all jar has Log4J logging classes inside it. Thoughts on this?
@g.kishore: yes, that has been a pain
@g.kishore: please let us know if you can come up with a fix
@ken: Well, normally you’d have all code just use the `slf4j-api`, and that would be the only compile-time dependency. Then you’d have test dependencies on `log4j-slf4j-impl` everywhere, and only have a compile time dependency on `log4j-slf4j-impl` in stand-alone components like the server, broker, controller, etc.
@g.kishore: thats how it is right?
  @ken: You do use slf4j, but (for example) the pinot-spi pom has: ``` <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> </dependency>``` So this pulls in the log4j implementation.
  @ken: You could also remove a number of log4j entries from the dependency management section of the top-level pom.xml, as once you have a dependency on `log4j-slf4j-impl`, that pulls in log4j-core and log4j-api, etc.
  @ken: Other than that, the following project pom.xml files should probably be modified to remove log4j dependencies: ```pinot-spi pinot-common```
  @ken: I think these are what’s causing the plugins to pull in log4j classes
  @ken: You’d have to add in log4j dependencies to the server/broker/controller projects
  @g.kishore: makes sense, I dont see any reason for not doing that
  @g.kishore: may be tools as well
  @ken: Right, tools would want to include the `log4j-slf4j-impl` dependency
@g.kishore: we use slf4j everywhere

#getting-started

@itdev: @itdev has joined the channel

#releases

@itdev: @itdev has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org