<h3><u>#general</u></h3><br><strong>@mayanks: </strong>Hello Pinot community,
we at LinkedIn, are planning to host a Virtual Meetup for Apache Pinot on Aug
27. We would like to have one presenter from the community share their
success-story/journey/etc with Pinot. Please reply to this thread if you would
be interested in speaking, with a brief on what you would like to talk
about.<br><strong>@christian: </strong>@christian has joined the
channel<br><strong>@qiaochu: </strong>@qiaochu has joined the
channel<br><h3><u>#random</u></h3><br><strong>@christian: </strong>@christian
has joined the channel<br><strong>@qiaochu: </strong>@qiaochu has joined the
channel<br><h3><u>#feat-text-search</u></h3><br><strong>@christian:
</strong>@christian has joined the
channel<br><h3><u>#troubleshooting</u></h3><br><strong>@sosyalmedya.oguzhan:
</strong>can someone help me about maven enforcer rules?
:disappointed:<br><strong>@yash.agarwal: </strong>I am getting the following
exception when querying the controller with a specific sql query.
```Caught exception while sending query request
java.io.IOException: Failed : HTTP error code : 500
at
org.apache.pinot.controller.api.resources.PinotQueryResource.sendPostRaw(PinotQueryResource.java:284)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
at
org.apache.pinot.controller.api.resources.PinotQueryResource.sendRequestRaw(PinotQueryResource.java:322)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
at
org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java:204)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
at
org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:131)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]```
If I directly query the broker I am getting
```<html>
<head>
<title>Grizzly 2.4.4</title>
<style>
<!--div.header
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#003300;font-size:22px;-moz-border-radius-topleft:
10px;border-top-left-radius: 10px;-moz-border-radius-topright:
10px;border-top-right-radius: 10px;padding-left: 5px}div.body
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:#FFFFCC;font-size:16px;padding-top:10px;padding-bottom:10px;padding-left:10px}div.footer
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#666633;font-size:14px;-moz-border-radius-bottomleft:
10px;border-bottom-left-radius: 10px;-moz-border-radius-bottomright:
10px;border-bottom-right-radius: 10px;padding-left: 5px}BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}B
{font-family:Tahoma,Arial,sans-serif;color:black;}A {color : black;}HR {color :
#999966;}
-->
</style>
</head>
<body>
<div class="header">Request failed.</div>
<div class="body">Request failed.</div>
<div class="footer">Grizzly 2.4.4</div>
</body>
</html>```
I have recently added a config in broker
```pinot.broker.timeoutMs=120000```
Any Ideas ?<br><strong>@yash.agarwal: </strong>There are no logs in broker.
Other queries are working fine, this query would scan 1.7 Bil recods and do a
sum of 3 metrics and distinctcount of 1 dimension<br><strong>@yash.agarwal:
</strong>It might be because of high memory usage on broker. as i am seeing
usage spikes upto 60 - 80 %<br><strong>@mayanks: </strong>Yes, distinctcount on
1.7B records may the likely issue. Have you considered using HLL based
approximate distinctcount?<br><strong>@yash.agarwal: </strong>We have tried HLL
in the past. We are not looking for approximations.<br><strong>@yash.agarwal:
</strong>Also seems like that is the problem, I tried without the distinctcount
and it is returning successfully in 2 secs.<br><strong>@mailtobuchi:
</strong>@mailtobuchi has left the channel<br><strong>@christian:
</strong>@christian has joined the
channel<br><h3><u>#pinot-dev</u></h3><br><strong>@mayanks: </strong>`mvn clean
install -DskipTests -Pbin-dist` takes awfully long time, it seems to have
creeped up quite a bit over the years. Any ideas on how to make the build
faster?<br><strong>@dlavoie: </strong>~ 10 minutes on my own CI server.
```[INFO] Pinot .............................................. SUCCESS [ 22.428
s]
[INFO] Pinot Service Provider Interface ................... SUCCESS [ 7.931 s]
[INFO] Pinot Common ....................................... SUCCESS [ 32.668 s]
[INFO] Pinot Plugins ...................................... SUCCESS [ 0.563 s]
[INFO] Pinot Input Format ................................. SUCCESS [ 0.510 s]
[INFO] Pinot Avro Base .................................... SUCCESS [ 1.694 s]
[INFO] Pinot Avro ......................................... SUCCESS [ 5.284 s]
[INFO] Pinot Csv .......................................... SUCCESS [ 0.844 s]
[INFO] Pinot JSON ......................................... SUCCESS [ 0.738 s]
[INFO] Pinot Core ......................................... SUCCESS [ 22.112 s]
[INFO] Pinot Server ....................................... SUCCESS [ 9.760 s]
[INFO] Pinot Controller ................................... SUCCESS [01:25 min]
[INFO] Pinot Broker ....................................... SUCCESS [ 4.878 s]
[INFO] Pinot Clients ...................................... SUCCESS [ 0.230 s]
[INFO] Pinot Java Client .................................. SUCCESS [ 0.750 s]
[INFO] Pinot JDBC Client .................................. SUCCESS [ 0.975 s]
[INFO] Pinot Minion ....................................... SUCCESS [ 2.538 s]
[INFO] Pinot Confluent Avro ............................... SUCCESS [ 3.746 s]
[INFO] Pinot ORC .......................................... SUCCESS [ 4.028 s]
[INFO] Pinot Parquet ...................................... SUCCESS [ 6.670 s]
[INFO] Pinot Thrift ....................................... SUCCESS [ 1.268 s]
[INFO] Pinot Protocol Buffers ............................. SUCCESS [ 1.329 s]
[INFO] Pluggable Pinot file system ........................ SUCCESS [ 0.354 s]
[INFO] Pinot Azure Data Lake Storage ...................... SUCCESS [ 5.128 s]
[INFO] Pinot Hadoop Filesystem ............................ SUCCESS [ 1.115 s]
[INFO] Pinot Google Cloud Storage ......................... SUCCESS [ 5.645 s]
[INFO] Pinot Amazon S3 .................................... SUCCESS [ 11.483 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [ 0.568 s]
[INFO] Pinot Batch Ingestion Common ....................... SUCCESS [ 0.785 s]
[INFO] Pinot Batch Ingestion for Spark .................... SUCCESS [ 11.263 s]
[INFO] Pinot Batch Ingestion for Hadoop ................... SUCCESS [ 3.663 s]
[INFO] Pinot Batch Ingestion Standalone ................... SUCCESS [ 0.985 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [ 0.448 s]
[INFO] Pinot Ingestion Common ............................. SUCCESS [ 1.763 s]
[INFO] Pinot Hadoop ....................................... SUCCESS [ 34.624 s]
[INFO] Pinot Spark ........................................ SUCCESS [ 35.108 s]
[INFO] Pinot Stream Ingestion ............................. SUCCESS [ 0.581 s]
[INFO] Pinot Kafka Base ................................... SUCCESS [ 1.323 s]
[INFO] Pinot Kafka 0.9 .................................... SUCCESS [ 9.815 s]
[INFO] Pinot Kafka 2.0 .................................... SUCCESS [ 5.919 s]
[INFO] Pinot Tools ........................................ SUCCESS [01:47 min]
[INFO] Pinot Integration Tests ............................ SUCCESS [ 18.378 s]
[INFO] Pinot Perf ......................................... SUCCESS [ 7.437 s]
[INFO] Pinot Distribution ................................. SUCCESS [ 38.707
s]```
Overall Pinot Controller and Pinot tools are the suspects<br><strong>@dlavoie:
</strong>Oh wait, don't mind that, it's a `mvn deploy` command to my personal
artifactory.<br><strong>@g.kishore: </strong>shaded jars take
time<br><strong>@dlavoie: </strong>If we had proper version definition and not
placeholders, maybe you could rebuild module selectively and not the whole
thing everytime something changes ?
:slightly_smiling_face:<br><strong>@dlavoie: </strong>I'm personnaly
maintaining a forked of pinot where I apply `mvn -f src/pom.xml --batch-mode
release:update-versions -DdevelopmentVersion=0.5.0-SNAPSHOT` and it just makes
life easier when recompiling.<br><strong>@mayanks: </strong>Don’t be so selfish
@dlavoie, spread the kindness :stuck_out_tongue:<br><strong>@dlavoie:
</strong>dhaha! It will impact the release proccess and I haven't taken the
time to assert the impact of moving to fixed version in all our poms :wink:
That's in my backlog<br><strong>@g.kishore: </strong>i also find it painful
that we cant compile a single module anymore, I am sure we were able to do
this<br><strong>@moradi.sajjad: </strong>A design doc for inconsistency
problems around time boundary calculation in pinot brokers:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMc9VK8AZw4xfCWnhVjqO8F0hEXgV9LklcwhZkvCiU2HHynhCXcIWc42UTjQHNwGE8ZryeBZDuJXpGEVgCdbS-2FE2qp9xQpwGJFPCs3e6Uify3eGK2_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTwT-2FWrA-2FsGEELDvpFu6KY8-2FxdJWDLq5C-2FVUOv2WYSm6toZju5MCCiSZ65Q0yHD23EjRYErL28TdCm6MsOuVkKXhRY9tg8Cdl75stg4wIjLRTSH0T62zO9alwJVxjBW-2BjPZmo5-2B2R4tg02ZqoYhUJ6kHt8N-2BPMoKCkwJRMJGwYXkoLILEH-2BoDTHJUn7ESCVC1ME-3D><br><strong>@moradi.sajjad:
</strong>please review and leave your comments.<br><strong>@mayanks:
</strong>Thanks @moradi.sajjad<br>
