[GitHub] [incubator-druid] surekhasaharan commented on a change in pull request #7653: Refactor SQLMetadataSegmentManager; Change contract of REST methods in DataSourcesResource
surekhasaharan commented on a change in pull request #7653: Refactor
SQLMetadataSegmentManager; Change contract of REST methods in
DataSourcesResource
URL: https://github.com/apache/incubator-druid/pull/7653#discussion_r287892431
##
File path:
server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java
##
@@ -197,366 +340,511 @@ private Runnable createPollTaskForStartOrder(long
startOrder)
}
@Override
- @LifecycleStop
- public void stop()
+ public boolean isPollingDatabasePeriodically()
{
-ReentrantReadWriteLock.WriteLock lock = startStopLock.writeLock();
+// isPollingDatabasePeriodically() is synchronized together with
startPollingDatabasePeriodically(),
+// stopPollingDatabasePeriodically() and poll() to ensure that the latest
currentStartPollingOrder is always
+// visible. readLock should be used to avoid unexpected performance
degradation of DruidCoordinator.
+ReentrantReadWriteLock.ReadLock lock = startStopPollLock.readLock();
lock.lock();
try {
- if (!isStarted()) {
+ return currentStartPollingOrder >= 0;
+}
+finally {
+ lock.unlock();
+}
+ }
+
+ @Override
+ public void stopPollingDatabasePeriodically()
+ {
+ReentrantReadWriteLock.WriteLock lock = startStopPollLock.writeLock();
+lock.lock();
+try {
+ if (!isPollingDatabasePeriodically()) {
return;
}
- dataSources = null;
- currentStartOrder = -1;
- exec.shutdownNow();
- exec = null;
+ periodicPollTaskFuture.cancel(false);
+ latestDatabasePoll = null;
+
+ // NOT nulling dataSources, allowing to query the latest polled data
even when this SegmentsMetadata object is
+ // stopped.
+
+ currentStartPollingOrder = -1;
}
finally {
lock.unlock();
}
}
- private Pair usedPayloadMapper(
- final int index,
- final ResultSet resultSet,
- final StatementContext context
- ) throws SQLException
+ private void awaitOrPerformDatabasePoll()
{
+// Double-checked locking with awaitLatestDatabasePoll() call playing the
role of the "check".
+if (awaitLatestDatabasePoll()) {
+ return;
+}
+ReentrantReadWriteLock.WriteLock lock = startStopPollLock.writeLock();
+lock.lock();
try {
- return new Pair<>(
- jsonMapper.readValue(resultSet.getBytes("payload"),
DataSegment.class),
- resultSet.getBoolean("used")
- );
+ if (awaitLatestDatabasePoll()) {
+return;
+ }
+ OnDemandDatabasePoll newOnDemandUpdate = new OnDemandDatabasePoll();
+ this.latestDatabasePoll = newOnDemandUpdate;
+ doOnDemandPoll(newOnDemandUpdate);
}
-catch (IOException e) {
- throw new RuntimeException(e);
+finally {
+ lock.unlock();
}
}
/**
- * Gets a list of all datasegments that overlap the provided interval along
with thier used status.
+ * If the latest {@link DatabasePoll} is a {@link PeriodicDatabasePoll}, or
an {@link OnDemandDatabasePoll} that is
+ * made not longer than {@link #periodicPollDelay} from now, awaits for it
and returns true; returns false otherwise,
+ * meaning that a new on-demand database poll should be initiated.
*/
- private List> getDataSegmentsOverlappingInterval(
- final String dataSource,
- final Interval interval
- )
+ private boolean awaitLatestDatabasePoll()
{
-return connector.inReadOnlyTransaction(
-(handle, status) -> handle.createQuery(
-StringUtils.format(
-"SELECT used, payload FROM %1$s WHERE dataSource = :dataSource
AND start < :end AND %2$send%2$s > :start",
-getSegmentsTable(),
-connector.getQuoteString()
-)
-)
-.setFetchSize(connector.getStreamingFetchSize())
-.bind("dataSource", dataSource)
-.bind("start", interval.getStart().toString())
-.bind("end", interval.getEnd().toString())
-.map(this::usedPayloadMapper)
-.list()
-);
+DatabasePoll latestDatabasePoll = this.latestDatabasePoll;
+if (latestDatabasePoll instanceof PeriodicDatabasePoll) {
+ Futures.getUnchecked(((PeriodicDatabasePoll)
latestDatabasePoll).firstPollCompletionFuture);
+ return true;
+}
+if (latestDatabasePoll instanceof OnDemandDatabasePoll) {
+ long periodicPollDelayNanos =
TimeUnit.MILLISECONDS.toNanos(periodicPollDelay.getMillis());
+ OnDemandDatabasePoll latestOnDemandPoll = (OnDemandDatabasePoll)
latestDatabasePoll;
+ boolean latestUpdateIsFresh =
latestOnDemandPoll.nanosElapsedFromInitiation() < periodicPollDelayNanos;
+ if (latestUpdateIsFresh) {
+Futures.getUnchecked(latestOnDemandPoll.pollCompletionFuture);
+return true;
+ }
+ // Latest on-demand update is not fresh. Fall through to return false
from this method.
+} else {
+ assert latestDatabasePoll == null;
+
[GitHub] [incubator-druid] surekhasaharan commented on a change in pull request #7653: Refactor SQLMetadataSegmentManager; Change contract of REST methods in DataSourcesResource
surekhasaharan commented on a change in pull request #7653: Refactor
SQLMetadataSegmentManager; Change contract of REST methods in
DataSourcesResource
URL: https://github.com/apache/incubator-druid/pull/7653#discussion_r287931169
##
File path:
server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java
##
@@ -713,6 +1021,8 @@ public DataSegment map(int index, ResultSet r,
StatementContext ctx) throws SQLE
}
catch (IOException e) {
log.makeAlert(e, "Failed to read segment from
db.").emit();
+ // If one entry is database is corrupted, doPoll()
should continue to work overall. See
Review comment:
nit: If one entry "is" -> "in" database ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] surekhasaharan commented on a change in pull request #7653: Refactor SQLMetadataSegmentManager; Change contract of REST methods in DataSourcesResource
surekhasaharan commented on a change in pull request #7653: Refactor
SQLMetadataSegmentManager; Change contract of REST methods in
DataSourcesResource
URL: https://github.com/apache/incubator-druid/pull/7653#discussion_r287874396
##
File path: docs/content/operations/api-reference.md
##
@@ -268,7 +276,9 @@ Runs a [Kill task](../ingestion/tasks.html) for a given
interval and datasource.
* `/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}`
-Disables a segment.
+Marks as unused a segment of a data source. Returns a JSON object of the form
`{"segmentStateChanged": "}` with
+the boolean indicating if the state of the segment has been changed (that is,
the segment was marked as used) as the
Review comment:
"segment was marked as used" -> "segment was marked as unused"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #7770: Web console: Adding a server view that can display all servers
clintropolis commented on issue #7770: Web console: Adding a server view that
can display all servers
URL: https://github.com/apache/incubator-druid/pull/7770#issuecomment-496354597
travis failure looks legitimate
https://travis-ci.org/apache/incubator-druid/jobs/537985537#L1726
```
[INFO] FAIL src/components/sql-control/sql-control.spec.tsx
[INFO] ● Test suite failed to run
[INFO]
[INFO] TypeScript diagnostics (customize using
`[jest-config].globals.ts-jest.diagnostics` option):
[INFO] src/components/sql-control/sql-control.spec.tsx:30:9 - error
TS2322: Type '(query: string, bypassCache: boolean, wrapQuery: boolean) =>
void' is not assignable to type '(query: string, context: Record,
wrapQuery: boolean) => void'.
[INFO] Types of parameters 'bypassCache' and 'context' are
incompatible.
[INFO] Type 'Record' is not assignable to type
'boolean'.
[INFO]
[INFO] 30 onRun={(query: string, bypassCache: boolean,
wrapQuery: boolean) => {}}
[INFO]~
[INFO]
[INFO] src/components/sql-control/sql-control.tsx:61:3
[INFO] 61 onRun: (query: string, context: Record,
wrapQuery: boolean) => void;
[INFO] ~
[INFO] The expected type comes from property 'onRun' which is
declared here on type 'IntrinsicAttributes &
IntrinsicClassAttributes & Readonly & Readonly<{
children?: ReactNode; }>'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] viongpanzi commented on a change in pull request #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec
viongpanzi commented on a change in pull request #7716: Fix memory problem
(OOM/FGC) when expression is used in metricsSpec
URL: https://github.com/apache/incubator-druid/pull/7716#discussion_r287909479
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
##
@@ -196,7 +246,7 @@ static BaseFloatColumnValueSelector
makeColumnValueSelectorWithFloatDefault(
if (fieldName != null) {
return metricFactory.makeColumnValueSelector(fieldName);
} else {
- final Expr expr = Parser.parse(fieldExpression, macroTable);
+ final Expr expr = parseIfAbsent(fieldExpression, macroTable);
Review comment:
@himanshug Thanks for your review. Sorry for the misunderstood! I need to
learn more about github and I'll pay attention next time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (e46bdf0 -> 58a6f0d)
This is an automated email from the ASF dual-hosted git repository. xvrl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from e46bdf0 Remove Codehaus references from the tests (#7773) new 484afe0 exclude hadoop system dependency new 58a6f0d Enable compiling against Java 9+ (tests disabled) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml | 3 +++ extensions-contrib/ambari-metrics-emitter/pom.xml | 6 + pom.xml | 29 +++ 3 files changed, 38 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] 02/02: Enable compiling against Java 9+ (tests disabled)
This is an automated email from the ASF dual-hosted git repository.
xvrl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
commit 58a6f0d5d05c736e1ac1fc8e3308676f880a3296
Author: Xavier Léauté
AuthorDate: Fri Apr 19 08:59:54 2019 -0700
Enable compiling against Java 9+ (tests disabled)
This change only enables compilation to ensure code compiles against
recent Java versions going forward. Tests are still disabled in this
profile until test failures are addressed.
---
.travis.yml | 3 +++
pom.xml | 29 +
2 files changed, 32 insertions(+)
diff --git a/.travis.yml b/.travis.yml
index 16c6d42..be35906 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -31,6 +31,9 @@ cache:
matrix:
include:
+ # Java 11 build
+- jdk: openjdk11
+
# license checks
- env:
- NAME="license checks"
diff --git a/pom.xml b/pom.xml
index daadaaa..193951a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -73,6 +73,7 @@
1.8
1.8
+8
UTF-8
4.1.0
2.12.0
@@ -1348,6 +1349,7 @@
org.apache.maven.plugins
maven-compiler-plugin
+3.8.1
${maven.compiler.source}
${maven.compiler.target}
@@ -1368,6 +1370,33 @@
+java-9+
+
+[9,)
+
+
+
+
+org.apache.maven.plugins
+maven-compiler-plugin
+true
+
+
+${java.version}
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+true
+
+
+
+
+
+
strict
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] xvrl merged pull request #7693: Enable compiling against Java 11 (tests disabled)
xvrl merged pull request #7693: Enable compiling against Java 11 (tests disabled) URL: https://github.com/apache/incubator-druid/pull/7693 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] 01/02: exclude hadoop system dependency
This is an automated email from the ASF dual-hosted git repository.
xvrl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
commit 484afe0afbbb3531374683162d5c73f40e83adc2
Author: Xavier Léauté
AuthorDate: Sun May 19 09:35:18 2019 -0700
exclude hadoop system dependency
ambari depends on hadoop-annotation. Under linux, it sometimes activates a
maven profile that includes an explicit dependency on
${java.home}/../lib/tools.jar
This change excludes this dependency, since this jar is no longer
included with Java 9. It also seems seems unlikely that we would require
this dependency at runtime, since it is only enabled on some platforms.
---
extensions-contrib/ambari-metrics-emitter/pom.xml | 6 ++
1 file changed, 6 insertions(+)
diff --git a/extensions-contrib/ambari-metrics-emitter/pom.xml
b/extensions-contrib/ambari-metrics-emitter/pom.xml
index 761a19d..47bd42e 100644
--- a/extensions-contrib/ambari-metrics-emitter/pom.xml
+++ b/extensions-contrib/ambari-metrics-emitter/pom.xml
@@ -61,6 +61,12 @@
org.codehaus.jackson
jackson-mapper-asl
+
+
+ jdk.tools
+ jdk.tools
+
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7367: Lookback query
stale[bot] commented on issue #7367: Lookback query URL: https://github.com/apache/incubator-druid/pull/7367#issuecomment-496330349 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pdeva commented on issue #7696: show 'number' column in tables of router ui
pdeva commented on issue #7696: show 'number' column in tables of router ui URL: https://github.com/apache/incubator-druid/issues/7696#issuecomment-496329494 Either and/or both are useful This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #1877: Add timePart extractionFn (Or make timeFormat more useful)
vogievetsky commented on issue #1877: Add timePart extractionFn (Or make timeFormat more useful) URL: https://github.com/apache/incubator-druid/issues/1877#issuecomment-496329030 This is solved by Druid SQL's `TIME_EXTRACT`: http://druid.io/docs/latest/querying/sql (and the under-the-hood math-expr that it generates) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky closed issue #1877: Add timePart extractionFn (Or make timeFormat more useful)
vogievetsky closed issue #1877: Add timePart extractionFn (Or make timeFormat more useful) URL: https://github.com/apache/incubator-druid/issues/1877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7696: show 'number' column in tables of router ui
vogievetsky commented on issue #7696: show 'number' column in tables of router ui URL: https://github.com/apache/incubator-druid/issues/7696#issuecomment-496325103 Are you suggesting an index column in the rows of the table or some sort of number indicating the total count of rows in the table? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch compile-jdk11 deleted (was 7d74586)
This is an automated email from the ASF dual-hosted git repository. xvrl pushed a change to branch compile-jdk11 in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. was 7d74586 exclude hadoop system dependency This change permanently discards the following revisions: discard 7d74586 exclude hadoop system dependency discard 57d075c Enable compiling against Java 9+ (tests disabled) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] 01/02: Enable compiling against Java 9+ (tests disabled)
This is an automated email from the ASF dual-hosted git repository.
xvrl pushed a commit to branch compile-jdk11
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
commit 57d075c58d05de7435e11e47c6b666991589a420
Author: Xavier Léauté
AuthorDate: Fri Apr 19 08:59:54 2019 -0700
Enable compiling against Java 9+ (tests disabled)
This change only enables compilation to ensure code compiles against
recent Java versions going forward. Tests are still disabled in this
profile until test failures are addressed.
---
.travis.yml | 3 +++
pom.xml | 29 +
2 files changed, 32 insertions(+)
diff --git a/.travis.yml b/.travis.yml
index 16c6d42..be35906 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -31,6 +31,9 @@ cache:
matrix:
include:
+ # Java 11 build
+- jdk: openjdk11
+
# license checks
- env:
- NAME="license checks"
diff --git a/pom.xml b/pom.xml
index ba9d713..50d39b7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -73,6 +73,7 @@
1.8
1.8
+8
UTF-8
4.1.0
2.12.0
@@ -1347,6 +1348,7 @@
org.apache.maven.plugins
maven-compiler-plugin
+3.8.0
${maven.compiler.source}
${maven.compiler.target}
@@ -1367,6 +1369,33 @@
+java-9+
+
+[9,)
+
+
+
+
+org.apache.maven.plugins
+maven-compiler-plugin
+true
+
+
+${java.version}
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+true
+
+
+
+
+
+
strict
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] 02/02: exclude hadoop system dependency
This is an automated email from the ASF dual-hosted git repository.
xvrl pushed a commit to branch compile-jdk11
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
commit 7d74586e0ed8621c15432f7f31f42f642cb60bbb
Author: Xavier Léauté
AuthorDate: Sun May 19 09:35:18 2019 -0700
exclude hadoop system dependency
ambari depends on hadoop-annotation. Under linux, it sometimes activates a
maven profile that includes an explicit dependency on
${java.home}/../lib/tools.jar
This change excludes this dependency, since this jar is no longer
included with Java 9. It also seems seems unlikely that we would require
this dependency at runtime, since it is only enabled on some platforms.
---
extensions-contrib/ambari-metrics-emitter/pom.xml | 6 ++
1 file changed, 6 insertions(+)
diff --git a/extensions-contrib/ambari-metrics-emitter/pom.xml
b/extensions-contrib/ambari-metrics-emitter/pom.xml
index dc83ba5..73e2263 100644
--- a/extensions-contrib/ambari-metrics-emitter/pom.xml
+++ b/extensions-contrib/ambari-metrics-emitter/pom.xml
@@ -61,6 +61,12 @@
org.codehaus.jackson
jackson-mapper-asl
+
+
+ jdk.tools
+ jdk.tools
+
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch compile-jdk11 created (now 7d74586)
This is an automated email from the ASF dual-hosted git repository. xvrl pushed a change to branch compile-jdk11 in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. at 7d74586 exclude hadoop system dependency This branch includes the following new commits: new 57d075c Enable compiling against Java 9+ (tests disabled) new 7d74586 exclude hadoop system dependency The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7775: [Backport] Web console: fix cache bypass in SQL view
fjy merged pull request #7775: [Backport] Web console: fix cache bypass in SQL view URL: https://github.com/apache/incubator-druid/pull/7775 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch 0.15.0-incubating updated: fix cache bypass (#7775)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.15.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.15.0-incubating by this push:
new cf0cd1f fix cache bypass (#7775)
cf0cd1f is described below
commit cf0cd1fc6d001f5385c898c5db14ab58c2cf1df0
Author: Vadim Ogievetsky
AuthorDate: Mon May 27 16:02:21 2019 -0700
fix cache bypass (#7775)
---
web-console/src/views/sql-view.tsx | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/web-console/src/views/sql-view.tsx
b/web-console/src/views/sql-view.tsx
index bdd297c..abd5f75 100644
--- a/web-console/src/views/sql-view.tsx
+++ b/web-console/src/views/sql-view.tsx
@@ -114,7 +114,7 @@ export class SqlView extends React.Component {
header: true
};
- if (wrapQuery) {
+ if (bypassCache) {
queryPayload.context = {
useCache: false,
populateCache: false
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky opened a new pull request #7775: [Backport] Web console: fix cache bypass in SQL view
vogievetsky opened a new pull request #7775: [Backport] Web console: fix cache bypass in SQL view URL: https://github.com/apache/incubator-druid/pull/7775 This is a backport of a single line from https://github.com/apache/incubator-druid/pull/7770 There is a bug that the SQL view cache bypass is actually keyed on `wrapQuery` instead of `bypassCache`. This is a regression in 0.15.0 (vs 0.14.x) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7772: Bump Apache Avro to 1.9.0
gianm commented on issue #7772: Bump Apache Avro to 1.9.0 URL: https://github.com/apache/incubator-druid/pull/7772#issuecomment-496309175 Would it be possible to update Pig too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch 0.15.0-incubating updated: Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode. (#7765) (#7774)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch 0.15.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.15.0-incubating by this push:
new 04929e4 Fix LookupSerdeModule double-binding in
Coordinator-as-Overlord mode. (#7765) (#7774)
04929e4 is described below
commit 04929e480c01a455d86fe5f9dff2201c3f7cc007
Author: Gian Merlino
AuthorDate: Mon May 27 14:10:06 2019 -0700
Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode.
(#7765) (#7774)
Fixes #7762.
---
services/src/main/java/org/apache/druid/cli/CliCoordinator.java | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/services/src/main/java/org/apache/druid/cli/CliCoordinator.java
b/services/src/main/java/org/apache/druid/cli/CliCoordinator.java
index c1ab5b9..98bb958 100644
--- a/services/src/main/java/org/apache/druid/cli/CliCoordinator.java
+++ b/services/src/main/java/org/apache/druid/cli/CliCoordinator.java
@@ -272,10 +272,12 @@ public class CliCoordinator extends ServerRunnable
}
);
-modules.add(new LookupSerdeModule());
-
if (beOverlord) {
modules.addAll(new CliOverlord().getModules(false));
+} else {
+ // Only add LookupSerdeModule if !beOverlord, since CliOverlord includes
it, and having two copies causes
+ // the injector to get confused due to having multiple bindings for the
same classes.
+ modules.add(new LookupSerdeModule());
}
return modules;
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis merged pull request #7774: [Backport] Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode.
clintropolis merged pull request #7774: [Backport] Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode. URL: https://github.com/apache/incubator-druid/pull/7774 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
sashidhar commented on a change in pull request #7764: #7316 Use
Map.putIfAbsent() instead of containsKey() + put()
URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287867297
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
##
@@ -508,13 +508,14 @@ public int getNumSplits()
// segments to olders.
// timelineSegments are sorted in order of interval
-int index = 0;
+int[] index = {0};
for (TimelineObjectHolder timelineHolder :
Lists.reverse(timelineSegments)) {
for (PartitionChunk chunk : timelineHolder.getObject()) {
for (String metric : chunk.getObject().getMetrics()) {
- if (!uniqueMetrics.containsKey(metric)) {
-uniqueMetrics.put(metric, index++);
+ uniqueMetrics.computeIfAbsent(metric, k -> {
+return index[0]++;
}
Review comment:
Using index variable as int as is, the compiler complains `"Variables used
in lambda should be final or effectively final"`. The fix is to use an integer
array with one element. Let me know if this is right.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
sashidhar commented on a change in pull request #7764: #7316 Use
Map.putIfAbsent() instead of containsKey() + put()
URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287866951
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
##
@@ -204,87 +204,87 @@ public Firehose connect(InputRowParser inputRowParser,
File temporaryDirectory)
segmentIds
);
-try {
- final List> timeLineSegments =
getTimeline();
-
- // Download all segments locally.
- // Note: this requires enough local storage space to fit all of the
segments, even though
- // IngestSegmentFirehose iterates over the segments in series. We may
want to change this
- // to download files lazily, perhaps sharing code with
PrefetchableTextFilesFirehoseFactory.
- final SegmentLoader segmentLoader =
segmentLoaderFactory.manufacturate(temporaryDirectory);
- Map segmentFileMap = Maps.newLinkedHashMap();
- for (TimelineObjectHolder holder :
timeLineSegments) {
-for (PartitionChunk chunk : holder.getObject()) {
- final DataSegment segment = chunk.getObject();
- if (!segmentFileMap.containsKey(segment)) {
-segmentFileMap.put(segment,
segmentLoader.getSegmentFiles(segment));
+final List> timeLineSegments =
getTimeline();
+
+// Download all segments locally.
+// Note: this requires enough local storage space to fit all of the
segments, even though
+// IngestSegmentFirehose iterates over the segments in series. We may want
to change this
+// to download files lazily, perhaps sharing code with
PrefetchableTextFilesFirehoseFactory.
+final SegmentLoader segmentLoader =
segmentLoaderFactory.manufacturate(temporaryDirectory);
+Map segmentFileMap = Maps.newLinkedHashMap();
+for (TimelineObjectHolder holder : timeLineSegments) {
+ for (PartitionChunk chunk : holder.getObject()) {
+final DataSegment segment = chunk.getObject();
+
+segmentFileMap.computeIfAbsent(segment, k -> {
+ try {
+return segmentLoader.getSegmentFiles(segment);
}
-}
- }
+ catch (SegmentLoadingException e) {
+throw new RuntimeException(e);
+ }
+});
- final List dims;
- if (dimensions != null) {
-dims = dimensions;
- } else if
(inputRowParser.getParseSpec().getDimensionsSpec().hasCustomDimensions()) {
-dims =
inputRowParser.getParseSpec().getDimensionsSpec().getDimensionNames();
- } else {
-dims = getUniqueDimensions(
-timeLineSegments,
-
inputRowParser.getParseSpec().getDimensionsSpec().getDimensionExclusions()
-);
}
+}
- final List metricsList = metrics == null ?
getUniqueMetrics(timeLineSegments) : metrics;
+final List dims;
+if (dimensions != null) {
+ dims = dimensions;
+} else if
(inputRowParser.getParseSpec().getDimensionsSpec().hasCustomDimensions()) {
+ dims =
inputRowParser.getParseSpec().getDimensionsSpec().getDimensionNames();
+} else {
+ dims = getUniqueDimensions(
+timeLineSegments,
+
inputRowParser.getParseSpec().getDimensionsSpec().getDimensionExclusions()
+ );
+}
- final List adapters = Lists.newArrayList(
- Iterables.concat(
- Iterables.transform(
- timeLineSegments,
- new Function,
Iterable>()
- {
+final List metricsList = metrics == null ?
getUniqueMetrics(timeLineSegments) : metrics;
+
+final List adapters = Lists.newArrayList(
+Iterables.concat(
+Iterables.transform(
+ timeLineSegments,
+ new Function,
Iterable>() {
+@Override
+public Iterable apply(final
TimelineObjectHolder holder)
+{
+ return
+Iterables.transform(
+ holder.getObject(),
+ new Function,
WindowedStorageAdapter>() {
@Override
-public Iterable apply(final
TimelineObjectHolder holder)
+public WindowedStorageAdapter apply(final
PartitionChunk input)
{
- return
- Iterables.transform(
- holder.getObject(),
- new Function,
WindowedStorageAdapter>()
- {
-@Override
-public WindowedStorageAdapter apply(final
PartitionChunk input)
-{
- final DataSegment segment =
input.getObject();
- try {
-return new WindowedStorageAdapter(
-new
[GitHub] [incubator-druid] sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
sashidhar commented on a change in pull request #7764: #7316 Use
Map.putIfAbsent() instead of containsKey() + put()
URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287866880
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
##
@@ -204,87 +204,87 @@ public Firehose connect(InputRowParser inputRowParser,
File temporaryDirectory)
segmentIds
);
-try {
- final List> timeLineSegments =
getTimeline();
-
- // Download all segments locally.
- // Note: this requires enough local storage space to fit all of the
segments, even though
- // IngestSegmentFirehose iterates over the segments in series. We may
want to change this
- // to download files lazily, perhaps sharing code with
PrefetchableTextFilesFirehoseFactory.
- final SegmentLoader segmentLoader =
segmentLoaderFactory.manufacturate(temporaryDirectory);
- Map segmentFileMap = Maps.newLinkedHashMap();
- for (TimelineObjectHolder holder :
timeLineSegments) {
-for (PartitionChunk chunk : holder.getObject()) {
- final DataSegment segment = chunk.getObject();
- if (!segmentFileMap.containsKey(segment)) {
-segmentFileMap.put(segment,
segmentLoader.getSegmentFiles(segment));
+final List> timeLineSegments =
getTimeline();
+
+// Download all segments locally.
+// Note: this requires enough local storage space to fit all of the
segments, even though
+// IngestSegmentFirehose iterates over the segments in series. We may want
to change this
+// to download files lazily, perhaps sharing code with
PrefetchableTextFilesFirehoseFactory.
+final SegmentLoader segmentLoader =
segmentLoaderFactory.manufacturate(temporaryDirectory);
+Map segmentFileMap = Maps.newLinkedHashMap();
+for (TimelineObjectHolder holder : timeLineSegments) {
+ for (PartitionChunk chunk : holder.getObject()) {
+final DataSegment segment = chunk.getObject();
+
+segmentFileMap.computeIfAbsent(segment, k -> {
+ try {
+return segmentLoader.getSegmentFiles(segment);
Review comment:
segmentLoader.getSegmentFiles(segment) throws SegmentLoadingException, which
needs to be caught and re thrown as RuntimeException. This makes the outer try
catch redundant (compilation error) as the SegmentLoadingException has already
been caught.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put() URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287864332 ## File path: .idea/inspectionProfiles/Druid.xml ## @@ -306,6 +303,11 @@ + Review comment: I've made the changes. Some of the changes are straight forward. I'll comment on those respective changes which are notable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put() URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287863927 ## File path: .idea/inspectionProfiles/Druid.xml ## @@ -161,7 +158,7 @@ - + Review comment: @leventov, thanks for the info. I've reverted the changes auto made by the IDE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7773: Remove Codehaus references from the tests
fjy merged pull request #7773: Remove Codehaus references from the tests URL: https://github.com/apache/incubator-druid/pull/7773 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Remove Codehaus references from the tests (#7773)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new e46bdf0 Remove Codehaus references from the tests (#7773)
e46bdf0 is described below
commit e46bdf082e85edc478a361a4fca001055b05d552
Author: Fokko Driesprong
AuthorDate: Mon May 27 19:51:14 2019 +0200
Remove Codehaus references from the tests (#7773)
---
.../apache/druid/server/lookup/LoadingLookupFactoryTest.java | 12 ++--
.../druid/server/lookup/PollingLookupSerDeserTest.java | 4 ++--
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git
a/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/LoadingLookupFactoryTest.java
b/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/LoadingLookupFactoryTest.java
index 810686f..adda834 100644
---
a/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/LoadingLookupFactoryTest.java
+++
b/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/LoadingLookupFactoryTest.java
@@ -20,6 +20,7 @@
package org.apache.druid.server.lookup;
+import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonTypeName;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.druid.query.lookup.LookupExtractorFactory;
@@ -27,7 +28,6 @@ import org.apache.druid.segment.TestHelper;
import org.apache.druid.server.lookup.cache.loading.LoadingCache;
import org.apache.druid.server.lookup.cache.loading.OffHeapLoadingCache;
import org.apache.druid.server.lookup.cache.loading.OnHeapLoadingCache;
-import org.codehaus.jackson.annotate.JsonCreator;
import org.easymock.EasyMock;
import org.junit.Assert;
import org.junit.Test;
@@ -38,11 +38,11 @@ import java.util.List;
public class LoadingLookupFactoryTest
{
- DataFetcher dataFetcher = EasyMock.createMock(DataFetcher.class);
- LoadingCache lookupCache = EasyMock.createStrictMock(LoadingCache.class);
- LoadingCache reverseLookupCache =
EasyMock.createStrictMock(LoadingCache.class);
- LoadingLookup loadingLookup = EasyMock.createMock(LoadingLookup.class);
- LoadingLookupFactory loadingLookupFactory = new LoadingLookupFactory(
+ private final DataFetcher dataFetcher =
EasyMock.createMock(DataFetcher.class);
+ private final LoadingCache lookupCache =
EasyMock.createStrictMock(LoadingCache.class);
+ private final LoadingCache> reverseLookupCache =
EasyMock.createStrictMock(LoadingCache.class);
+ private final LoadingLookup loadingLookup =
EasyMock.createMock(LoadingLookup.class);
+ private final LoadingLookupFactory loadingLookupFactory = new
LoadingLookupFactory(
dataFetcher,
lookupCache,
reverseLookupCache,
diff --git
a/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/PollingLookupSerDeserTest.java
b/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/PollingLookupSerDeserTest.java
index 350acb5..eef78c0 100644
---
a/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/PollingLookupSerDeserTest.java
+++
b/extensions-core/lookups-cached-single/src/test/java/org/apache/druid/server/lookup/PollingLookupSerDeserTest.java
@@ -19,6 +19,7 @@
package org.apache.druid.server.lookup;
+import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonTypeName;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.druid.jackson.DefaultObjectMapper;
@@ -26,7 +27,6 @@ import org.apache.druid.query.lookup.LookupExtractorFactory;
import org.apache.druid.server.lookup.cache.polling.OffHeapPollingCache;
import org.apache.druid.server.lookup.cache.polling.OnHeapPollingCache;
import org.apache.druid.server.lookup.cache.polling.PollingCacheFactory;
-import org.codehaus.jackson.annotate.JsonCreator;
import org.joda.time.Period;
import org.junit.Assert;
import org.junit.Test;
@@ -53,7 +53,7 @@ public class PollingLookupSerDeserTest
}
private final PollingCacheFactory cacheFactory;
- private DataFetcher dataFetcher = new MockDataFetcher();
+ private final DataFetcher dataFetcher = new MockDataFetcher();
public PollingLookupSerDeserTest(PollingCacheFactory cacheFactory)
{
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7771: How to get the SQL or the entire query by queryId?
gianm commented on issue #7771: How to get the SQL or the entire query by queryId? URL: https://github.com/apache/incubator-druid/issues/7771#issuecomment-496266499 If you want to see the query that was issued, turn on request logging: http://druid.io/docs/latest/configuration/index.html#request-logging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
leventov commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put() URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287846810 ## File path: .idea/inspectionProfiles/Druid.xml ## @@ -161,7 +158,7 @@ - + Review comment: When using different versions of IntelliJ IDEA, it makes some seemingly arbitrary changes to the config by itself. As long as it doesn't break the TeamCity build (and if the PR author uses the latest version of IntelliJ), it should be fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov edited a comment on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status
leventov edited a comment on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status URL: https://github.com/apache/incubator-druid/issues/7571#issuecomment-496264473 Regarding interning - we have to get rid of them anyway: #7395 explains how. If a segment first appears in `BrokerServerView`, it can be created as `SegmentWithOvershadowedStatus(isOvershadowed=false)`. Then if it later appears in `MetadataSegmentView` and it is not actually overshadowed, we reuse the existing object by probing `BrokerServerView`. If the arrived object has `isOvershadowed=true`, we just lose the opportunity to deduplicate this segment, but we expect it for the minority of segments, so that's fine. Regarding `SegmentWithOvershadowedStatus(isOvershadowed=false)` objects in `BrokerServerView` while the actual status is unknown: it seems weird at first glance but it's actually fine. The rule is that to probe `isOvershadowed` status we should *always* go through `MetadataSegmentView`. To guard against misuse, we can prohibit casting to `SegmentWithOvershadowedStatus` in the codebase on the Checkstyle or IntelliJ level. > What do you mean by 'prohibit' @leventov? I followed the link, and I don't see a suggestion about prohibiting equals and hashCode. (Just about prohibiting using DataSegments as keys in a map, which seems reasonable, given that usually people probably meant to use SegmentId.) Sorry, I didn't re-check what did I propose there. No, I don't suggest to prohibit. I suggest exactly what is written in https://github.com/apache/incubator-druid/issues/6358#issuecomment-489102183. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status
leventov commented on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status URL: https://github.com/apache/incubator-druid/issues/7571#issuecomment-496264473 Regarding interning - we have to get rid of them anyway: #7395 explains, how. If a segment first appears in `BrokerServerView`, it can be created as `SegmentWithOvershadowedStatus(isOvershadowed=false)`. Then if it later appears in `MetadataSegmentView` and it is not actually overshadowed, we reuse the existing object by probing `BrokerServerView`. If the arrived object has `isOvershadowed=true`, we just lose the opportunity to deduplicate this segment, but we expect it for the minority of segments, so that's fine. Regarding `SegmentWithOvershadowedStatus(isOvershadowed=false)` objects in `BrokerServerView` while the actual status is unknown: it seems weird at first glance but it's actually fine. The rule is that to probe `isOvershadowed` status we should *always* go through `MetadataSegmentView`. To guard against misuse, we can prohibit casting to `SegmentWithOvershadowedStatus` in the codebase on the Checkstyle or IntelliJ level. > What do you mean by 'prohibit' @leventov? I followed the link, and I don't see a suggestion about prohibiting equals and hashCode. (Just about prohibiting using DataSegments as keys in a map, which seems reasonable, given that usually people probably meant to use SegmentId.) Sorry, I didn't re-check what did I propose there. No, I don't suggest to prohibit. I suggest exactly what is written in https://github.com/apache/incubator-druid/issues/6358#issuecomment-489102183. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug closed issue #7715: Memory problem (OOM/FGC) when expression is used in metricsSpec
himanshug closed issue #7715: Memory problem (OOM/FGC) when expression is used in metricsSpec URL: https://github.com/apache/incubator-druid/issues/7715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec
himanshug merged pull request #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec URL: https://github.com/apache/incubator-druid/pull/7716 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)
This is an automated email from the ASF dual-hosted git repository.
himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 42cf078 Fix memory problem (OOM/FGC) when expression is used in
metricsSpec (#7716)
42cf078 is described below
commit 42cf07884345fa5fba70a17156678dc6c37ab3d1
Author: BIGrey
AuthorDate: Tue May 28 00:46:17 2019 +0800
Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)
* AggregatorUtil should cache parsed expression to avoid memory problem
(OOM/FGC) when Expression is used in metricsSpec
* remove debug log check in Parser.parse
* remove cache and use suppliers.memorize
---
.../druid/benchmark/datagen/BenchmarkSchemas.java | 18 +
.../druid/query/aggregation/AggregatorUtil.java| 20 ++---
.../aggregation/SimpleDoubleAggregatorFactory.java | 10 ++-
.../aggregation/SimpleFloatAggregatorFactory.java | 10 ++-
.../aggregation/SimpleLongAggregatorFactory.java | 10 ++-
.../incremental/OnheapIncrementalIndexTest.java| 94 ++
6 files changed, 139 insertions(+), 23 deletions(-)
diff --git
a/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
b/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
index cda9f47..3e35a66 100644
---
a/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
+++
b/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
@@ -21,6 +21,7 @@ package org.apache.druid.benchmark.datagen;
import com.google.common.collect.ImmutableList;
import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.math.expr.ExprMacroTable;
import org.apache.druid.query.aggregation.AggregatorFactory;
import org.apache.druid.query.aggregation.CountAggregatorFactory;
import org.apache.druid.query.aggregation.DoubleMinAggregatorFactory;
@@ -85,6 +86,14 @@ public class BenchmarkSchemas
basicSchemaIngestAggs.add(new DoubleMinAggregatorFactory("minFloatZipf",
"metFloatZipf"));
basicSchemaIngestAggs.add(new HyperUniquesAggregatorFactory("hyper",
"dimHyperUnique"));
+List basicSchemaIngestAggsExpression = new
ArrayList<>();
+basicSchemaIngestAggsExpression.add(new CountAggregatorFactory("rows"));
+basicSchemaIngestAggsExpression.add(new
LongSumAggregatorFactory("sumLongSequential", null, "if(sumLongSequential>0 &&
dimSequential>100 || dimSequential<10 ||
metLongSequential>3000,sumLongSequential,0)", ExprMacroTable.nil()));
+basicSchemaIngestAggsExpression.add(new
LongMaxAggregatorFactory("maxLongUniform", "metLongUniform"));
+basicSchemaIngestAggsExpression.add(new
DoubleSumAggregatorFactory("sumFloatNormal", null, "if(sumFloatNormal>0 &&
dimSequential>100 || dimSequential<10 ||
metLongSequential>3000,sumFloatNormal,0)", ExprMacroTable.nil()));
+basicSchemaIngestAggsExpression.add(new
DoubleMinAggregatorFactory("minFloatZipf", "metFloatZipf"));
+basicSchemaIngestAggsExpression.add(new
HyperUniquesAggregatorFactory("hyper", "dimHyperUnique"));
+
Interval basicSchemaDataInterval = Intervals.utc(0, 100);
BenchmarkSchemaInfo basicSchema = new BenchmarkSchemaInfo(
@@ -93,7 +102,16 @@ public class BenchmarkSchemas
basicSchemaDataInterval,
true
);
+
+BenchmarkSchemaInfo basicSchemaExpression = new BenchmarkSchemaInfo(
+basicSchemaColumns,
+basicSchemaIngestAggsExpression,
+basicSchemaDataInterval,
+true
+);
+
SCHEMA_MAP.put("basic", basicSchema);
+SCHEMA_MAP.put("expression", basicSchemaExpression);
}
static { // simple single string column and count agg schema, no rollup
diff --git
a/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
b/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
index bf1edd2..e5e5f51 100644
---
a/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
+++
b/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
@@ -24,8 +24,6 @@ import org.apache.druid.guice.annotations.PublicApi;
import org.apache.druid.java.util.common.Pair;
import org.apache.druid.math.expr.Expr;
import org.apache.druid.math.expr.ExprEval;
-import org.apache.druid.math.expr.ExprMacroTable;
-import org.apache.druid.math.expr.Parser;
import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
import org.apache.druid.segment.BaseDoubleColumnValueSelector;
import org.apache.druid.segment.BaseFloatColumnValueSelector;
@@ -184,9 +182,8 @@ public class AggregatorUtil
*/
static BaseFloatColumnValueSelector makeColumnValueSelectorWithFloatDefault(
final ColumnSelectorFactory metricFactory,
- final ExprMacroTable macroTable,
@Nullable final String
[GitHub] [incubator-druid] himanshug commented on a change in pull request #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec
himanshug commented on a change in pull request #7716: Fix memory problem
(OOM/FGC) when expression is used in metricsSpec
URL: https://github.com/apache/incubator-druid/pull/7716#discussion_r287845275
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
##
@@ -196,7 +246,7 @@ static BaseFloatColumnValueSelector
makeColumnValueSelectorWithFloatDefault(
if (fieldName != null) {
return metricFactory.makeColumnValueSelector(fieldName);
} else {
- final Expr expr = Parser.parse(fieldExpression, macroTable);
+ final Expr expr = parseIfAbsent(fieldExpression, macroTable);
Review comment:
@viongpanzi changes look good, thanks.
also, not sure if you know/understood but when I raised
https://github.com/viongpanzi/incubator-druid/pull/1 , I would expect you to
merge the PR instead of copying the changes so that original commit stays
alongside your commits . something for next time :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7774: [Backport] Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode.
gianm opened a new pull request #7774: [Backport] Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode. URL: https://github.com/apache/incubator-druid/pull/7774 Backport of #7765 to 0.15.0-incubating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pcarrier commented on issue #7628: 0.14.2-incubating release notes
pcarrier commented on issue #7628: 0.14.2-incubating release notes URL: https://github.com/apache/incubator-druid/issues/7628#issuecomment-496233482 @clintropolis seems to be propagated to https://archive.apache.org/dist/incubator/druid/ Could we push a git tag? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on issue #7772: Bump Apache Avro to 1.9.0
Fokko commented on issue #7772: Bump Apache Avro to 1.9.0 URL: https://github.com/apache/incubator-druid/pull/7772#issuecomment-496213370 It seems to clash with Apache Pig which still uses Avro 1.7.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put()
sashidhar commented on a change in pull request #7764: #7316 Use Map.putIfAbsent() instead of containsKey() + put() URL: https://github.com/apache/incubator-druid/pull/7764#discussion_r287790197 ## File path: .idea/inspectionProfiles/Druid.xml ## @@ -306,6 +303,11 @@ + Review comment: Sure @gianm , I'll make the necessary changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko opened a new pull request #7773: Remove Codehaus references from the tests
Fokko opened a new pull request #7773: Remove Codehaus references from the tests URL: https://github.com/apache/incubator-druid/pull/7773 While updating Apache Avro I've stumbled upon these. I believe they should be imported from fasterxml (Jackson 2.x) instead of Codehaus (Jackson 1.x). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] viongpanzi commented on a change in pull request #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec
viongpanzi commented on a change in pull request #7716: Fix memory problem
(OOM/FGC) when expression is used in metricsSpec
URL: https://github.com/apache/incubator-druid/pull/7716#discussion_r287784384
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
##
@@ -196,7 +246,7 @@ static BaseFloatColumnValueSelector
makeColumnValueSelectorWithFloatDefault(
if (fieldName != null) {
return metricFactory.makeColumnValueSelector(fieldName);
} else {
- final Expr expr = Parser.parse(fieldExpression, macroTable);
+ final Expr expr = parseIfAbsent(fieldExpression, macroTable);
Review comment:
@himanshug sorry for the delayed update! Can you review this patch again?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] viongpanzi removed a comment on issue #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec
viongpanzi removed a comment on issue #7716: Fix memory problem (OOM/FGC) when expression is used in metricsSpec URL: https://github.com/apache/incubator-druid/pull/7716#issuecomment-496001204 @gianm would you like to look at this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko opened a new pull request #7772: Bump Apache Avro to 1.9.0
Fokko opened a new pull request #7772: Bump Apache Avro to 1.9.0 URL: https://github.com/apache/incubator-druid/pull/7772 Apache Avro 1.9.0 brings a lot of new features: * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default * Remove support for Hadoop 1.x * Move from Jackson 1.x to 2.9 * Add ZStandard Codec * Lots of updates on the dependencies to fix CVE's * Remove Jackson classes from public API * Apache Avro is built by default with Java 8 * Apache Avro is compiled and tested with Java 11 to guarantee compatibility * Apache Avro MapReduce is compiled and tested with Hadoop 3 * Apache Avro is now leaner, multiple dependencies were removed: guava, paranamer, commons-codec, and commons-logging * Introduce JMH Performance Testing Framework * Add Snappy support for C++ DataFile * and many, many more! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on issue #7701: Bump Jackson to 2.9.9
Fokko commented on issue #7701: Bump Jackson to 2.9.9 URL: https://github.com/apache/incubator-druid/pull/7701#issuecomment-496179878 I'm familiar with the pain of classpath issues in combination with Jackson, Guava etc. Currently, we don't run Druid with Hadoop, so I'm not able to test this easily. I can try to get something running in Docker, but this will take some time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
