#general
@sambitdixit: @sambitdixit has joined the channel
@ho0001vu: hi Pinot-dev, any update on generic "Null" support?
@g.kishore: I think it’s supported in filter predicates @chinmay.cerebro ^^
@ho0001vu: what happens if we don't use a predicate? for numeric type I'd like to see nulls instead of some default value
@g.kishore: I see, we dont have the support for that
@darshants.darshan1: Hi folks, I tried to build the pinot project and seeing build errors. Occasionally, I see a "re-try"/"sync" type error as it takes a while to sync from public maven to our internal aritfactory. Unfortunately, upon several re-tries, I still see the errors. 1) [*INFO*] Pinot Google Cloud Storage ......................... *FAILURE* [ 0.555 s] 2)[*ERROR*] Failed to execute goal on project pinot-gcs: *Could not resolve dependencies for project org.apache.pinot:pinot-gcs:jar:0.6.0-SNAPSHOT: Failed to collect dependencies at com.google.cloud:google-cloud-storage:jar:1.102.0 -> com.google.cloud:google-cloud-core-http:jar:1.91.3 -> com.google.api:gax-httpjson:jar:0.66.1*: Failed to read artifact descriptor for com.google.api:gax-httpjson:jar:0.66.1: Could not transfer artifact com.google.api:gax-httpjson:pom:0.66.1 from/to central Any thoughts/suggestions?
@darshants.darshan1: This is resolved; we had to re-build the cache in our internal artifactory.
@mayanks: Thanks for confirming.
@karinwolok1: :pinot: Pinot Committer & PMC @npawar made this "Intro to Pinot" video to help people better understand when and how to use it. :brain: It covers a wide range of things, including highlighted use cases to the capabilities of Pinot and how it achieves real-time analytics. Check it out!
@changliu: @changliu has joined the channel
#random
@sambitdixit: @sambitdixit has joined the channel
@changliu: @changliu has joined the channel
#troubleshooting
@ho0001vu: @ho0001vu has joined the channel
@elon.azoulay: Anyone ever see this error? It occured on a pinot-server, BaseCombiner: Timed out while polling results block:
@elon.azoulay:
@mayanks: Is it an expensive query?
@elon.azoulay: But it looks like the error came from a server though
@elon.azoulay: It wasn't expensive, we bounced the servers and the query latency went from 25s to < 1 second after that.
@elon.azoulay: And we saw no red flags, cpu, network, disk usage etc. were fine
@elon.azoulay: Also I wasn't aware those combiner services run on the servers, maybe the logs are messed up in google cloud?
@afilipchik: @mayanks yep, error is from the server
@mayanks: Sorry, my original message was incorrect. this is from server level combine.
@mayanks: Was there GC?
@elon.azoulay: I can check
@elon.azoulay: 1 sec, also thanks so much for the super quick response!
@elon.azoulay: We need to edit the jmx config, looks like we didn't expose that metric
@elon.azoulay: Will do this
@mayanks: @elon.azoulay, just to add a bit more context. The part of the code throwing exception is waiting on worker threads that are executing the query on a bunch of segments. `numBlocksMerged: 0` seems to suggest none of those threads returned within timeout.
@afilipchik: @mayanks some context
@elon.azoulay: thanks! So what would likely cause that?
@mayanks: It is typically things like expensive query, GC, VM freeze
@afilipchik: our pinot cluster stalled in prod, with 3 out of 5 nodes having those log messages. Their appearance correlate with the time observed query latencies went up
@afilipchik: rolling restart fixed the problem for now, but we don't have a root cause
@mayanks: This is the most common stack trace you will see in case of latency spikes. This is not root cause, but side effect of some other problem.
@afilipchik: hmmm
@afilipchik: what can cause an issue that is treated by restart?
@mayanks: GC, High read load
@elon.azoulay: We're using jdk8 - do you recommend us changing to g1gc? Right now it's using concurrent mark sweep.
@mayanks: Yes, please move to g1gc
@elon.azoulay: Will do!
@mayanks: we have not used CMS in Pinot production at all at LinkedIn
@mayanks: Are these realtime nodes?
@elon.azoulay: Yep
@mayanks: Are you using all the realtime optimizations (like offheap etc)?
@afilipchik: they are hybrid
@afilipchik: realtime + offline
@elon.azoulay: I believe so, mmap for segments, is that how you do off heap?
@elon.azoulay: Any jvm options you recommend?
@mayanks: how much data per node?
@mayanks: most of the time we use xms=xmx=16G
@elon.azoulay: 6 nodes x 100gb-280gb (for the older nodes w more realtime data)
@mayanks: ingestion rate?
@elon.azoulay: and ~27gb memory
@mayanks: ok, you likely need more heap than 16G for sure
@elon.azoulay: But if we' re using mmap for segments that would imply we need to leave space for offheap, right?
@mayanks: also something like: `<value>-XX:MaxGCPauseMillis=20</value>`
@elon.azoulay: and how do we take advantage of offheap, is that just the segment loadmode = mmap?
@mayanks: ``` <value>-XX:+UseG1GC</value> <value>-XX:+ParallelRefProcEnabled</value> <value>-XX:+DisableExplicitGC</value>```
@mayanks: I think there's another setting for mmap. Let me find.
@elon.azoulay: thanks!
@elon.azoulay: we ingest ~6-8mb/s but that will be increasing
@afilipchik: do we need to set MaxDirectMemory?
@elon.azoulay: @mayanks, you're really help us, thanks!
@mayanks: @afilipchik it is not needed, but in practice, we set it to size of main-memory - xmx
@afilipchik: got it, thank you! do you do it to allocate additional ram? or just historical?
@elon.azoulay: sorry, just want to make sure we know how to take advantage of offheap, I thought it was just setting the load mode to MMAP, lmk if there's anything else
@mayanks:
@mayanks: @elon.azoulay ^^
#pinot-dev
@ho0001vu: @ho0001vu has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
