[incubator-druid] branch master updated (a066cc5 -> 8cb213a)
This is an automated email from the ASF dual-hosted git repository. vogievetsky pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from a066cc5 Fix groupMapping endpoint URIs in druid-basic-security doc (#8847) add 8cb213a Web console: Fix missing include future flag for byPeriod rules (#8859) No new revisions were added by this update. Summary of changes: .../src/components/rule-editor/rule-editor.scss| 7 + .../src/components/rule-editor/rule-editor.tsx | 34 -- web-console/src/utils/load-rule.ts | 19 +++- .../src/views/datasource-view/datasource-view.tsx | 5 4 files changed, 50 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky closed issue #8855: web console: Load rule includeFuture flag is not visible or editable
vogievetsky closed issue #8855: web console: Load rule includeFuture flag is not visible or editable URL: https://github.com/apache/incubator-druid/issues/8855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky merged pull request #8859: Web console: Fix missing include future flag for byPeriod rules
vogievetsky merged pull request #8859: Web console: Fix missing include future flag for byPeriod rules URL: https://github.com/apache/incubator-druid/pull/8859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8838: Web console: replace (and remove) old consoles
lgtm-com[bot] commented on issue #8838: Web console: replace (and remove) old consoles URL: https://github.com/apache/incubator-druid/pull/8838#issuecomment-553201060 This pull request **fixes 28 alerts** when merging d22c178b4bb6ce5f5bd075f25f9bf445f48743a2 into a066cc5648f763935c8e56c23c3810ea90e4955b - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-3bc80a2d2e5c2b391a26acd91fdc231b6a9b0897) **fixed alerts:** * 25 for Unused variable, import, function or class * 1 for Useless assignment to local variable * 1 for Comparison between inconvertible types * 1 for Incomplete string escaping or encoding This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7008: Wish list: a way to query the peons via the management proxy
stale[bot] commented on issue #7008: Wish list: a way to query the peons via the management proxy URL: https://github.com/apache/incubator-druid/issues/7008#issuecomment-553193966 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7006: Wish list: A DruidSQL table for MiddleManagers (workers)
stale[bot] commented on issue #7006: Wish list: A DruidSQL table for MiddleManagers (workers) URL: https://github.com/apache/incubator-druid/issues/7006#issuecomment-553193959 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7004: Wish list: An API endpoint to get all timeBoundries (for all datasources) out of Druid in one call
stale[bot] commented on issue #7004: Wish list: An API endpoint to get all timeBoundries (for all datasources) out of Druid in one call URL: https://github.com/apache/incubator-druid/issues/7004#issuecomment-553193964 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8823: Add InputSource and InputFormat interfaces
clintropolis commented on a change in pull request #8823: Add InputSource and
InputFormat interfaces
URL: https://github.com/apache/incubator-druid/pull/8823#discussion_r345523058
##
File path:
core/src/main/java/org/apache/druid/data/input/AbstractInputSource.java
##
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input;
+
+import com.google.common.base.Preconditions;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+/**
+ * Abstract class for {@link InputSource}. This class provides a default
implementation of {@link #reader} with
+ * a sanity check. Child classes should implement one of {@link
#formattableReader} or {@link #unformattableReader}
+ * depending on {@link #needsFormat()}.
+ */
+public abstract class AbstractInputSource implements InputSource
+{
+ @Override
+ public InputSourceReader reader(
+ InputRowSchema inputRowSchema,
+ @Nullable InputFormat inputFormat,
+ @Nullable File temporaryDirectory
+ )
+ {
+if (needsFormat()) {
+ return formattableReader(
+ inputRowSchema,
+ Preconditions.checkNotNull(inputFormat, "inputFormat"),
+ temporaryDirectory
+ );
+} else {
+ return unformattableReader(inputRowSchema, temporaryDirectory);
+}
+ }
+
+ protected InputSourceReader formattableReader(
+ InputRowSchema inputRowSchema,
+ @Nullable InputFormat inputFormat,
+ @Nullable File temporaryDirectory
+ )
+ {
+throw new UnsupportedOperationException("Implement this method properly if
needsFormat() = true");
+ }
+
+ protected InputSourceReader unformattableReader(InputRowSchema
inputRowSchema, @Nullable File temporaryDirectory)
Review comment:
i think `implicitlyFormattedReader` or `fixedFormatReader` might be a more
intuitive name for this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8823: Add InputSource and InputFormat interfaces
clintropolis commented on a change in pull request #8823: Add InputSource and
InputFormat interfaces
URL: https://github.com/apache/incubator-druid/pull/8823#discussion_r345522332
##
File path:
core/src/main/java/org/apache/druid/data/input/InputSourceReader.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input;
+
+import org.apache.druid.data.input.impl.InputEntityIteratingReader;
+import org.apache.druid.guice.annotations.ExtensionPoint;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+
+import java.io.IOException;
+
+/**
+ * InputSourceReader reads data from {@link InputSource} and returns a {@link
CloseableIterator} of
+ * {@link InputRow}. See {@link InputSource} for an example usage.
+ *
+ * Implementations of this class can use {@link InputEntity} and {@link
InputEntityReader}. {@link InputFormat}
+ * can be useful to understand how to create an InputEntityReader.
+ *
+ * See {@link InputEntityIteratingReader} as an example.
+ */
+@ExtensionPoint
+public interface InputSourceReader
+{
+ CloseableIterator read() throws IOException;
+
+ CloseableIterator sample() throws IOException;
Review comment:
To handle potential 1:many mapping of raw input records to `InputRow`, we
probably want a new wrapper type that associates a `List` to the
'raw' data
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8823: Add InputSource and InputFormat interfaces
clintropolis commented on a change in pull request #8823: Add InputSource and
InputFormat interfaces
URL: https://github.com/apache/incubator-druid/pull/8823#discussion_r345521999
##
File path: core/src/main/java/org/apache/druid/data/input/TextReader.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input;
+
+import org.apache.commons.io.LineIterator;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.CloseableIterator;
+import org.apache.druid.java.util.common.parsers.ParseException;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+
+/**
+ * Abstract {@link InputEntityReader} for text format readers such as CSV or
JSON.
+ */
+public abstract class TextReader implements InputEntityReader
+{
+ private final InputRowSchema inputRowSchema;
+
+ public TextReader(InputRowSchema inputRowSchema)
+ {
+this.inputRowSchema = inputRowSchema;
+ }
+
+ public InputRowSchema getInputRowSchema()
+ {
+return inputRowSchema;
+ }
+
+ @Override
+ public CloseableIterator read(InputEntity source, File
temporaryDirectory) throws IOException
+ {
+return lineIterator(source).map(line -> {
+ try {
+return readLine(line);
+ }
+ catch (IOException e) {
+throw new ParseException(e, "Unable to parse row [%s]", line);
+ }
+});
+ }
+
+ @Override
+ public CloseableIterator sample(InputEntity source, File
temporaryDirectory)
+ throws IOException
+ {
+return lineIterator(source).map(line -> {
+ try {
+return InputRowPlusRaw.of(readLine(line), StringUtils.toUtf8(line));
+ }
+ catch (ParseException e) {
+return InputRowPlusRaw.of(StringUtils.toUtf8(line), e);
+ }
+ catch (IOException e) {
+throw new RuntimeException(e);
+ }
+});
+ }
+
+ private CloseableIterator lineIterator(InputEntity source) throws
IOException
+ {
+final LineIterator delegate = new LineIterator(
+new InputStreamReader(source.open(), StringUtils.UTF8_STRING)
+);
+final int numHeaderLines = getNumHeaderLinesToSkip();
+for (int i = 0; i < numHeaderLines && delegate.hasNext(); i++) {
+ delegate.nextLine(); // skip lines
+}
+if (needsToProcessHeaderLine() && delegate.hasNext()) {
+ processHeaderLine(delegate.nextLine());
+}
+
+return new CloseableIterator()
+{
+ @Override
+ public boolean hasNext()
+ {
+return delegate.hasNext();
+ }
+
+ @Override
+ public String next()
+ {
+return delegate.nextLine();
+ }
+
+ @Override
+ public void close() throws IOException
+ {
+delegate.close();
+ }
+};
+ }
+
+ /**
+ * Parses the given line into {@link InputRow}.
+ */
+ public abstract InputRow readLine(String line) throws IOException,
ParseException;
Review comment:
I think this should be future proofed to return a list of `InputRow` (for
explode spec)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8859: Web console: Fix missing include future flag for byPeriod rules
vogievetsky commented on issue #8859: Web console: Fix missing include future flag for byPeriod rules URL: https://github.com/apache/incubator-druid/pull/8859#issuecomment-553166455 LGTM after CI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 commented on a change in pull request #8859: Web console: Fix missing include future flag for byPeriod rules
renevan10 commented on a change in pull request #8859: Web console: Fix missing
include future flag for byPeriod rules
URL: https://github.com/apache/incubator-druid/pull/8859#discussion_r345497349
##
File path: web-console/src/views/datasource-view/datasource-view.tsx
##
@@ -642,11 +642,6 @@ GROUP BY 1`;
];
} else {
return goToActions.concat([
-{
- icon: IconNames.GANTT_CHART,
- title: 'Go to tasks',
- onAction: () => goToTask(datasource),
-},
Review comment:
I noticed this was duplicated, if you look carefully there will be two "Go
to tasks" under each datasource(unless they are disabled).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 commented on a change in pull request #8859: Web console: Fix missing include future flag for byPeriod rules
renevan10 commented on a change in pull request #8859: Web console: Fix missing
include future flag for byPeriod rules
URL: https://github.com/apache/incubator-druid/pull/8859#discussion_r345495745
##
File path: web-console/src/utils/load-rule.ts
##
@@ -41,7 +42,13 @@ export class RuleUtil {
return (
rule.type +
(rule.period ? `(${rule.period})` : '') +
- (rule.interval ? `(${rule.interval})` : '')
+ (rule.interval ? `(${rule.interval})` : '') +
+ (rule.includeFuture !== false &&
+ (rule.type === 'loadByPeriod' ||
+rule.type === 'dropByPeriod' ||
+rule.type === 'broadcastByPeriod')
Review comment:
I've made the changes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 commented on a change in pull request #8859: Web console: Fix missing include future flag for byPeriod rules
renevan10 commented on a change in pull request #8859: Web console: Fix missing
include future flag for byPeriod rules
URL: https://github.com/apache/incubator-druid/pull/8859#discussion_r345495728
##
File path: web-console/src/components/rule-editor/rule-editor.scss
##
@@ -30,4 +30,11 @@
flex: 1;
}
}
+
+ .byPeriod {
Review comment:
I've made the changes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #8859: Web console: Fix missing include future flag for byPeriod rules
vogievetsky commented on a change in pull request #8859: Web console: Fix
missing include future flag for byPeriod rules
URL: https://github.com/apache/incubator-druid/pull/8859#discussion_r345476443
##
File path: web-console/src/utils/load-rule.ts
##
@@ -41,7 +42,13 @@ export class RuleUtil {
return (
rule.type +
(rule.period ? `(${rule.period})` : '') +
- (rule.interval ? `(${rule.interval})` : '')
+ (rule.interval ? `(${rule.interval})` : '') +
+ (rule.includeFuture !== false &&
+ (rule.type === 'loadByPeriod' ||
+rule.type === 'dropByPeriod' ||
+rule.type === 'broadcastByPeriod')
Review comment:
this logic should be extracted into a utility function
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #8859: Web console: Fix missing include future flag for byPeriod rules
vogievetsky commented on a change in pull request #8859: Web console: Fix
missing include future flag for byPeriod rules
URL: https://github.com/apache/incubator-druid/pull/8859#discussion_r345475987
##
File path: web-console/src/components/rule-editor/rule-editor.scss
##
@@ -30,4 +30,11 @@
flex: 1;
}
}
+
+ .byPeriod {
Review comment:
please use `kebab-case` for class names
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #8859: Web console: Fix missing include future flag for byPeriod rules
vogievetsky commented on a change in pull request #8859: Web console: Fix
missing include future flag for byPeriod rules
URL: https://github.com/apache/incubator-druid/pull/8859#discussion_r345476663
##
File path: web-console/src/views/datasource-view/datasource-view.tsx
##
@@ -642,11 +642,6 @@ GROUP BY 1`;
];
} else {
return goToActions.concat([
-{
- icon: IconNames.GANTT_CHART,
- title: 'Go to tasks',
- onAction: () => goToTask(datasource),
-},
Review comment:
why did you remove this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-553141979 I just tried my patch, it's not working This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] suneet-amp commented on a change in pull request #8822: optimize numeric column null value checking for low filter selectivity (more rows)
suneet-amp commented on a change in pull request #8822: optimize numeric column
null value checking for low filter selectivity (more rows)
URL: https://github.com/apache/incubator-druid/pull/8822#discussion_r345466852
##
File path:
processing/src/main/java/org/apache/druid/collections/bitmap/PeekableIteratorAdapter.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.collections.bitmap;
+
+import com.google.common.base.Preconditions;
+import org.roaringbitmap.IntIterator;
+import org.roaringbitmap.PeekableIntIterator;
+
+public class PeekableIteratorAdapter
implements PeekableIntIterator
+{
+ static final int NOT_SET = -1;
+ final TIntIterator baseIterator;
+ int mark = NOT_SET;
+
+ PeekableIteratorAdapter(TIntIterator iterator)
+ {
+this.baseIterator = Preconditions.checkNotNull(iterator, "iterator");
+ }
+
+ @Override
+ public void advanceIfNeeded(int i)
+ {
+while (mark < i && baseIterator.hasNext()) {
Review comment:
sgtm
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8857: Add reference to druid.storage.type
jihoonson commented on a change in pull request #8857: Add reference to druid.storage.type URL: https://github.com/apache/incubator-druid/pull/8857#discussion_r345461025 ## File path: docs/development/extensions-core/s3.md ## @@ -61,6 +63,7 @@ As an example, to set the region to 'us-east-1' through system properties: |`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](#server-side-encryption) for more details.|None| |`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| |`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| +|`druid.storage.type`|Global deep storage provider. Must be set to `s3` to make use of this extension.|`local`| Review comment: Ok, it sounds good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fstolba commented on a change in pull request #8857: Add reference to druid.storage.type
fstolba commented on a change in pull request #8857: Add reference to druid.storage.type URL: https://github.com/apache/incubator-druid/pull/8857#discussion_r345460456 ## File path: docs/development/extensions-core/s3.md ## @@ -61,6 +63,7 @@ As an example, to set the region to 'us-east-1' through system properties: |`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](#server-side-encryption) for more details.|None| |`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| |`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| +|`druid.storage.type`|Global deep storage provider. Must be set to `s3` to make use of this extension.|`local`| Review comment: Yes that probably makes sense to just mark it as mandatory. We could suggest people might want to use `s3` considering the page they're viewing. Regarding the table ordering it looks like the knobs are grouped at least by their namespace so I put it with the other `druid.storage` parameters. Seeing that the first paragraph in the Configuration section mentions that this knob is mandatory it seems to me one would be able to quickly grasp the importance of this parameter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8744: support for array expressions in TransformSpec with ExpressionTransform
gianm commented on a change in pull request #8744: support for array
expressions in TransformSpec with ExpressionTransform
URL: https://github.com/apache/incubator-druid/pull/8744#discussion_r345459910
##
File path:
processing/src/main/java/org/apache/druid/segment/virtual/ExpressionSelectors.java
##
@@ -514,15 +509,45 @@ public void inspectRuntimeShape(RuntimeShapeInspector
inspector)
/**
* Selectors are not consistent in treatment of null, [], and [null], so
coerce [] to [null]
*/
- private static Object coerceListDimToStringArray(List val)
+ // suppressed because calling toArray creates Object[] instead of Long[]
which makes ExprEval.bestEffortOf sad
+ @SuppressWarnings("SimplifyStreamApiCallChains")
+ public static Object coerceListToArray(List val)
{
-Object[] arrayVal = val.stream().map(x -> x != null ? x.toString() :
x).toArray(String[]::new);
-if (arrayVal.length > 0) {
- return arrayVal;
+if (val != null && val.size() > 0) {
+ Object firstElement = val.get(0);
Review comment:
Expressions support lists that contain nulls, right? What happens if the
first element is null? Also, what happens if the first element is an Integer
but later ones are Doubles? i.e. JSON `[1, 2.0]`.
It may be better to examine all elements using some kind of binary type
conversion rules (like long + int = long, etc).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8822: optimize numeric column null value checking for low filter selectivity (more rows)
gianm commented on a change in pull request #8822: optimize numeric column null
value checking for low filter selectivity (more rows)
URL: https://github.com/apache/incubator-druid/pull/8822#discussion_r345456789
##
File path:
processing/src/main/java/org/apache/druid/collections/bitmap/PeekableIteratorAdapter.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.collections.bitmap;
+
+import com.google.common.base.Preconditions;
+import org.roaringbitmap.IntIterator;
+import org.roaringbitmap.PeekableIntIterator;
+
+public class PeekableIteratorAdapter
implements PeekableIntIterator
+{
+ static final int NOT_SET = -1;
+ final TIntIterator baseIterator;
+ int mark = NOT_SET;
+
+ PeekableIteratorAdapter(TIntIterator iterator)
+ {
+this.baseIterator = Preconditions.checkNotNull(iterator, "iterator");
+ }
+
+ @Override
+ public void advanceIfNeeded(int i)
+ {
+while (mark < i && baseIterator.hasNext()) {
Review comment:
I'd suggest dealing with this, if at all, just through javadocs saying that
these interfaces should only be used with nonnegative ints. In practice these
iterators are used for iterating over bitmaps that represent row numbers. It's
doubly-impossible to see negative numbers: bitmaps cannot store negative ints,
and row numbers cannot be negative either.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-553124223 The example is `druid.storage.type=local` btw This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8857: Add reference to druid.storage.type
jihoonson commented on a change in pull request #8857: Add reference to druid.storage.type URL: https://github.com/apache/incubator-druid/pull/8857#discussion_r345454907 ## File path: docs/development/extensions-core/s3.md ## @@ -61,6 +63,7 @@ As an example, to set the region to 'us-east-1' through system properties: |`druid.storage.sse.type`|Server-side encryption type. Should be one of `s3`, `kms`, and `custom`. See the below [Server-side encryption section](#server-side-encryption) for more details.|None| |`druid.storage.sse.kms.keyId`|AWS KMS key ID. This is used only when `druid.storage.sse.type` is `kms` and can be empty to use the default key ID.|None| |`druid.storage.sse.custom.base64EncodedKey`|Base64-encoded key. Should be specified if `druid.storage.sse.type` is `custom`.|None| +|`druid.storage.type`|Global deep storage provider. Must be set to `s3` to make use of this extension.|`local`| Review comment: The default value is `local` and so this doc is technically correct. But, I'm a bit worried about whether it could confuse people. How about changing it to `Must be set.` instead of `local`? Also, I guess you added it here based on an alphabetical order but there's no actual ordering I think. How about moving it to the top of the table since it must be set? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus edited a comment on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus edited a comment on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-553123020 ``` 11:33:28.944 [main] ERROR org.apache.druid.indexer.JobHelper - Exception in retry loop java.lang.IllegalArgumentException: Pathname /druid/indexer/foo/2019-11-12T04:00:00.000Z_2019-11-12T05:00:00.000Z/2019-11-12T10:04:03.778Z/0/index.zip.0 from hdfs://us2/druid/indexer/foo/2019-11-12T04:00:00.000Z_2019-11-12T05:00:00.000Z/2019-11-12T10:04:03.778Z/0/index.zip.0 is not a valid DFS filename. at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:217) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:476) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:473) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:473) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:414) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929) ~[hadoop-common-2.8.5.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) ~[hadoop-common-2.8.5.jar:?] at org.apache.druid.indexer.JobHelper$2.push(JobHelper.java:452) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) [hadoop-common-2.8.5.jar:?] at com.sun.proxy.$Proxy78.push(Unknown Source) [?:?] at org.apache.druid.indexer.JobHelper.serializeOutIndex(JobHelper.java:469) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at org.apache.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:827) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at org.apache.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:579) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) [hadoop-mapreduce-client-core-2.8.5.jar:?] at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) [hadoop-mapreduce-client-core-2.8.5.jar:?] at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) [hadoop-mapreduce-client-core-2.8.5.jar:?] at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) [hadoop-mapreduce-client-app-2.8.5.jar:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_222] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) [hadoop-mapreduce-client-app-2.8.5.jar:?] ``` we upgraded to 2.8.5 right around the same time as druid 0.16 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-553123020 ```11:33:28.944 [main] ERROR org.apache.druid.indexer.JobHelper - Exception in retry loop java.lang.IllegalArgumentException: Pathname /druid/indexer/foo/2019-11-12T04:00:00.000Z_2019-11-12T05:00:00.000Z/2019-11-12T10:04:03.778Z/0/index.zip.0 from hdfs://us2/druid/indexer/foo/2019-11-12T04:00:00.000Z_2019-11-12T05:00:00.000Z/2019-11-12T10:04:03.778Z/0/index.zip.0 is not a valid DFS filename. at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:217) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:476) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:473) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:473) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:414) ~[hadoop-hdfs-client-2.8.5.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929) ~[hadoop-common-2.8.5.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) ~[hadoop-common-2.8.5.jar:?] at org.apache.druid.indexer.JobHelper$2.push(JobHelper.java:452) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) [hadoop-common-2.8.5.jar:?] at com.sun.proxy.$Proxy78.push(Unknown Source) [?:?] at org.apache.druid.indexer.JobHelper.serializeOutIndex(JobHelper.java:469) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at org.apache.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:827) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at org.apache.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:579) [druid-indexing-hadoop-0.16.0-lqm1.jar:0.16.0-lqm1] at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) [hadoop-mapreduce-client-core-2.8.5.jar:?] at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) [hadoop-mapreduce-client-core-2.8.5.jar:?] at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) [hadoop-mapreduce-client-core-2.8.5.jar:?] at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) [hadoop-mapreduce-client-app-2.8.5.jar:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_222] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) [hadoop-common-2.8.5.jar:?] at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) [hadoop-mapreduce-client-app-2.8.5.jar:?] ``` - we upgraded to 2.8.5 right around the same time as druid 0.16 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
gianm commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-553118126 Hmm, I tried the [Hadoop tutorial](https://druid.apache.org/docs/latest/tutorials/tutorial-batch-hadoop.html), which uses HDFS deep storage, and it worked ok for me. I wonder if something weird is going on with your setup. Do you have a stack trace from the reduce task that fails? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fstolba commented on issue #8857: Add reference to druid.storage.type
fstolba commented on issue #8857: Add reference to druid.storage.type URL: https://github.com/apache/incubator-druid/pull/8857#issuecomment-553116852 @jihoonson Done. I was thinking about adding all the possible values of the parameter to the table but since this is S3s documentation page it's probably not that helpful here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] chandrashekhar-saini opened a new issue #8860: Java JDBC Connectivity and Select Query Exectuion
chandrashekhar-saini opened a new issue #8860: Java JDBC Connectivity and Select Query Exectuion URL: https://github.com/apache/incubator-druid/issues/8860 HI Team, I need your help, please share sample code to connect Druid Database using Java. I have a requirement i need to get data from druid database by running select query using Java Program. What I need. 1. Java Program to connect to druid Database. 2. How to Select Data from Druid Using Java Code. I have tables in Druid Database Table : EMPLOYEE Column : EMP_NAME, AGE, ADDRESS, DATE I need to select Name and Address Using Java Program. Please share the code. Regards, Shekhar This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] glasser commented on a change in pull request #5149: Automatic pendingSegments cleanup
glasser commented on a change in pull request #5149: Automatic pendingSegments
cleanup
URL: https://github.com/apache/incubator-druid/pull/5149#discussion_r345362611
##
File path:
server/src/main/java/io/druid/server/coordinator/DruidCoordinatorCleanupPendingSegments.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to Metamarkets Group Inc. (Metamarkets) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. Metamarkets licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package io.druid.server.coordinator;
+
+import com.google.common.base.Preconditions;
+import com.google.inject.Inject;
+import io.druid.client.ImmutableDruidDataSource;
+import io.druid.client.indexing.IndexingServiceClient;
+import io.druid.indexer.TaskStatusPlus;
+import io.druid.java.util.common.DateTimes;
+import io.druid.java.util.common.guava.Comparators;
+import io.druid.java.util.common.logger.Logger;
+import io.druid.server.coordinator.helper.DruidCoordinatorHelper;
+import org.joda.time.DateTime;
+import org.joda.time.Period;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DruidCoordinatorCleanupPendingSegments implements
DruidCoordinatorHelper
+{
+ private static final Logger log = new
Logger(DruidCoordinatorCleanupPendingSegments.class);
+ private static final Period KEEP_PENDING_SEGMENTS_OFFSET = new Period("P1D");
+
+ private final IndexingServiceClient indexingServiceClient;
+
+ @Inject
+ public DruidCoordinatorCleanupPendingSegments(IndexingServiceClient
indexingServiceClient)
+ {
+this.indexingServiceClient = indexingServiceClient;
+ }
+
+ @Override
+ public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams
params)
+ {
+final List createdTimes = new ArrayList<>();
+createdTimes.add(
+indexingServiceClient
+.getRunningTasks()
+.stream()
+.map(TaskStatusPlus::getCreatedTime)
+.min(Comparators.naturalNullsFirst())
+.orElse(DateTimes.nowUtc()) // If there is no running tasks, this
returns the current time.
+);
+createdTimes.add(
+indexingServiceClient
+.getPendingTasks()
+.stream()
+.map(TaskStatusPlus::getCreatedTime)
+.min(Comparators.naturalNullsFirst())
+.orElse(DateTimes.nowUtc()) // If there is no pending tasks, this
returns the current time.
+);
+createdTimes.add(
+indexingServiceClient
+.getWaitingTasks()
+.stream()
+.map(TaskStatusPlus::getCreatedTime)
+.min(Comparators.naturalNullsFirst())
+.orElse(DateTimes.nowUtc()) // If there is no waiting tasks, this
returns the current time.
+);
+
+final TaskStatusPlus completeTaskStatus =
indexingServiceClient.getLastCompleteTask();
+if (completeTaskStatus != null) {
+ createdTimes.add(completeTaskStatus.getCreatedTime());
+}
+createdTimes.sort(Comparators.naturalNullsFirst());
+
+// There should be at least one createdTime because the current time is
added to the 'createdTimes' list if there
Review comment:
Is this comment accurate? I don't see where it's implemented, which suggests
that this helper can crash in a brand new cluster.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8859: Web console: Fix missing include future flag for byPeriod rules
lgtm-com[bot] commented on issue #8859: Web console: Fix missing include future flag for byPeriod rules URL: https://github.com/apache/incubator-druid/pull/8859#issuecomment-553015141 This pull request **introduces 2 alerts** when merging 419b25b6337e2e7573d822f600efa02bf95f077d into a066cc5648f763935c8e56c23c3810ea90e4955b - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-713bf6c845aa55b6c6d706847c7eb3cbc5c42965) **new alerts:** * 2 for Comparison between inconvertible types This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] renevan10 opened a new pull request #8859: Web console: Fix missing include future flag for byPeriod rules
renevan10 opened a new pull request #8859: Web console: Fix missing include future flag for byPeriod rules URL: https://github.com/apache/incubator-druid/pull/8859 When adding a by period rule, the user can now decide whether to include the future through a flag. By default it is set to true.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] glasser edited a comment on issue #8856: web console: display index_parallel progress
glasser edited a comment on issue #8856: web console: display index_parallel progress URL: https://github.com/apache/incubator-druid/issues/8856#issuecomment-553009726 That makes sense. My understanding is that the "reports" concept which isn't currently used by `index_parallel` tasks is an end-of-task-only thing and can't be used for progress, right? Another alternative would be a way to get things like progress into the overlord's understanding of task status. (ps hope to meet you tonight!) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] glasser edited a comment on issue #8856: web console: display index_parallel progress
glasser edited a comment on issue #8856: web console: display index_parallel progress URL: https://github.com/apache/incubator-druid/issues/8856#issuecomment-553009726 That makes sense. My understanding is that the "reports" concept which isn't currently used by `index_parallel` tasks is an end-of-task-only thing and can't be used for progress, right? Another alternative would be a way to get things like progress into the overlord's understanding of task status. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] glasser commented on issue #8856: web console: display index_parallel progress
glasser commented on issue #8856: web console: display index_parallel progress URL: https://github.com/apache/incubator-druid/issues/8856#issuecomment-553009726 That makes sense. My understanding is that the "reports" concept which isn't currently used by `index_parallel` tasks is an end-of-task-only thing and can't be used for progress, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pjain1 commented on a change in pull request #8841: add readiness endpoints to processes having initialization delays
pjain1 commented on a change in pull request #8841: add readiness endpoints to processes having initialization delays URL: https://github.com/apache/incubator-druid/pull/8841#discussion_r345325843 ## File path: docs/operations/api-reference.md ## @@ -752,7 +752,11 @@ Returns segment information lists including server locations for the given datas * `/druid/broker/v1/loadstatus` -Returns a flag indicating if the Broker knows about all segments in Zookeeper. This can be used to know when a Broker process is ready to be queried after a restart. +Returns a flag indicating if the Broker knows about all segments in the cluster. This can be used to know when a Broker process is ready to be queried after a restart. + +* `/druid/broker/v1/readiness` Review comment: @nishantmonu51 Historical already has `/druid/historical/v1/readiness` so just extended it for broker - `/druid/broker/v1/readiness` to be consistent, but can change it to just `/readiness` if required, no preference from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pjain1 commented on a change in pull request #8841: add readiness endpoints to processes having initialization delays
pjain1 commented on a change in pull request #8841: add readiness endpoints to processes having initialization delays URL: https://github.com/apache/incubator-druid/pull/8841#discussion_r345325843 ## File path: docs/operations/api-reference.md ## @@ -752,7 +752,11 @@ Returns segment information lists including server locations for the given datas * `/druid/broker/v1/loadstatus` -Returns a flag indicating if the Broker knows about all segments in Zookeeper. This can be used to know when a Broker process is ready to be queried after a restart. +Returns a flag indicating if the Broker knows about all segments in the cluster. This can be used to know when a Broker process is ready to be queried after a restart. + +* `/druid/broker/v1/readiness` Review comment: Historical already has `/druid/historical/v1/readiness` so just extended it for broker - `/druid/broker/v1/readiness` to be consistent, but can change it to just `/readiness` if required, no preference from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 merged pull request #8847: Fix groupMapping endpoint URIs in druid-basic-security doc
nishantmonu51 merged pull request #8847: Fix groupMapping endpoint URIs in druid-basic-security doc URL: https://github.com/apache/incubator-druid/pull/8847 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #8841: add readiness endpoints to processes having initialization delays
nishantmonu51 commented on a change in pull request #8841: add readiness endpoints to processes having initialization delays URL: https://github.com/apache/incubator-druid/pull/8841#discussion_r345279741 ## File path: docs/operations/api-reference.md ## @@ -752,7 +752,11 @@ Returns segment information lists including server locations for the given datas * `/druid/broker/v1/loadstatus` -Returns a flag indicating if the Broker knows about all segments in Zookeeper. This can be used to know when a Broker process is ready to be queried after a restart. +Returns a flag indicating if the Broker knows about all segments in the cluster. This can be used to know when a Broker process is ready to be queried after a restart. + +* `/druid/broker/v1/readiness` Review comment: agree with @sascha-coenen readiness is a commonly used term for health checks. One change though would be to have a generic endpoint */readiness* instead of node type specific endpoints. would suggest removing /druid//v1 prefix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552945372 and there is no alternative to `index_hadoop` for raw files in HDFS, at least it's not obvious from the documentation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552940672 to be more precise, it's the hadoop reduce task that fails This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus edited a comment on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus edited a comment on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552939930 It's working for compact and index_kafka. However index_hadoop is written by Hadoop itself, I think that's the problem. I just removed the broken commit from my PR. ``` druid.storage.type=hdfs druid.storage.storageDirectory=hdfs://us2/druid/indexer ``` and yes, I have been doing this for a while. I remember filing a bug back in the days. And now it's back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus edited a comment on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus edited a comment on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552939930 It's working for compact and index_kafka. However index_hadoop is written by Hadoop itself, I think that's the problem. I just removed the broken commit from my PR. ```druid.storage.type=hdfs druid.storage.storageDirectory=hdfs://us2/druid/indexer ``` and yes, I have been doing this for a while. I remember filing a bug back in the days. And now it's back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: index_hadoop tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552939930 It's working for compact and index_kafka. However index_hadoop is written by Hadoop itself, I think that's the problem. I just removed the broken commit from my PR. ```druid.storage.type=hdfs druid.storage.storageDirectory=hdfs://us2/druid/indexer``` and yes, I have been doing this for a while. I remember filing a bug back in the days. And now it's back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8840: hadoop_index tasks fail on wrong file format when run inside indexer
gianm commented on issue #8840: hadoop_index tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552936310 What are your `druid.storage.*` properties? I am looking at the code that handles HDFS path colon replacement, and it hasn't changed in a while, and I don't see a reason for it to behave differently on MM/Peon vs Indexer. Also, we do have other users on 0.16.0 + HDFS deep storage. So it should still work… But if it doesn't please feed us some more clues and we can get to the bottom of it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sixtus commented on issue #8840: hadoop_index tasks fail on wrong file format when run inside indexer
sixtus commented on issue #8840: hadoop_index tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552935676 Yes, this is druid 0.16.0. I just fixed the config bug and reverted and the bug is back. I am not sure if it's running on peons (didn't validate that), but it sure isn't working when running on indexer. My PR has 2 commits. I was expecting it to work after the first and it didn't. But I think this was just a jar caching issue. I am about to test it again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8840: hadoop_index tasks fail on wrong file format when run inside indexer
gianm commented on issue #8840: hadoop_index tasks fail on wrong file format when run inside indexer URL: https://github.com/apache/incubator-druid/issues/8840#issuecomment-552934069 Hi @sixtus, Is this in Druid 0.16.0? Am I hearing you right that when you run on 0.16.0 using MM + peons, everything works fine, but when you run on 0.16.0 with an Indexer, you get this behavior where `:` are not replaced with `_` in segment names? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6790: chooseBestServer bug in CostBalancerStrategy
stale[bot] commented on issue #6790: chooseBestServer bug in CostBalancerStrategy URL: https://github.com/apache/incubator-druid/issues/6790#issuecomment-552914672 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed pull request #8098: Kafka lookup custom Json and Jq extractor
stale[bot] closed pull request #8098: Kafka lookup custom Json and Jq extractor URL: https://github.com/apache/incubator-druid/pull/8098 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #8098: Kafka lookup custom Json and Jq extractor
stale[bot] commented on issue #8098: Kafka lookup custom Json and Jq extractor URL: https://github.com/apache/incubator-druid/pull/8098#issuecomment-552914645 This pull request/issue has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8838: Web console: replace (and remove) old consoles
lgtm-com[bot] commented on issue #8838: Web console: replace (and remove) old consoles URL: https://github.com/apache/incubator-druid/pull/8838#issuecomment-552877364 This pull request **fixes 28 alerts** when merging e026739e0147b2922d217634723fc355c6a5f7b3 into df2f77c58d599e74b097027aea46368d378bbeea - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-c86a04a180f610bbf26345e061a6847457eddd27) **fixed alerts:** * 25 for Unused variable, import, function or class * 1 for Useless assignment to local variable * 1 for Comparison between inconvertible types * 1 for Incomplete string escaping or encoding This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #4133: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.orc.OrcNewSplit not found
stale[bot] commented on issue #4133: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.orc.OrcNewSplit not found URL: https://github.com/apache/incubator-druid/issues/4133#issuecomment-552848212 This issue is no longer marked as stale. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] chenhaiyan commented on issue #4133: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.orc.OrcNewSplit not found
chenhaiyan commented on issue #4133: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.orc.OrcNewSplit not found URL: https://github.com/apache/incubator-druid/issues/4133#issuecomment-552848198 @stale > This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [d...@druid.apache.org](mailto:d...@druid.apache.org) list. Thank you for your contributions. Sorry about this,I didn't see my mail for a long time . As for the question,it is ok to close. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fstolba opened a new issue #8858: export-metadata throws java.lang.NullPointerException
fstolba opened a new issue #8858: export-metadata throws
java.lang.NullPointerException
URL: https://github.com/apache/incubator-druid/issues/8858
I'm trying to migrate my segments from local storage to S3. When running
`export-metadata` with parameters `-b`/`-k` (rewriting deep storage location to
S3), it crashes while exporting the segments table. Exporting without rewriting
works fine.
I suspect it might be due to the fact that not all segments in the table
refer to the same storage location. Recently created and new segments are
actually stored in the target bucket already (storage location in
`common/runtime.conf` was changed to reflect the S3 configuration).
### Affected Version
0.16
### Description
```
bash-4.2# java -classpath "lib/*"
-Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml
-Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[]
org.apache.druid.cli.Main tools export-metadata --connectURI
"jdbc:derby://localhost:1527/var/druid/metadata.db;" -o /tmp -b druid-bucket -k
druid/segments
Nov 12, 2019 10:11:27 AM org.hibernate.validator.internal.util.Version
INFO: HV01: Hibernate Validator 5.1.3.Final
2019-11-12T10:11:27,504 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.common.config.NullValueHandlingConfig]
from props[druid.generic.] as
[org.apache.druid.common.config.NullValueHandlingConfig@3da05287]
2019-11-12T10:11:27,552 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.guice.ExtensionsConfig] from
props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true,
directory='extensions', useExtensionClassloaderFirst=false,
hadoopDependenciesDir='hadoop-dependencies',
hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false,
loadList=[]}]
2019-11-12T10:11:27,640 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.guice.ModulesConfig] from
props[druid.modules.] as [ModulesConfig{excludeList=[]}]
2019-11-12T10:11:27,795 INFO [main]
org.apache.druid.initialization.Initialization - Adding implementation
[org.apache.druid.common.aws.AWSModule] for class [interface
org.apache.druid.initialization.DruidModule] from classpath extension
2019-11-12T10:11:27,797 INFO [main]
org.apache.druid.initialization.Initialization - Adding implementation
[org.apache.druid.common.gcp.GcpModule] for class [interface
org.apache.druid.initialization.DruidModule] from classpath extension
2019-11-12T10:11:28,042 INFO [main] org.apache.druid.guice.Jerseys - Adding
Jersey resource: org.apache.druid.server.StatusResource
2019-11-12T10:11:28,476 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[interface org.apache.druid.server.security.Escalator] from
props[druid.escalator.] as [NoopEscalator{}]
2019-11-12T10:11:28,486 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.curator.CuratorConfig] from
props[druid.zk.service.] as [org.apache.druid.curator.CuratorConfig@3015db78]
2019-11-12T10:11:28,490 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.curator.ExhibitorConfig] from
props[druid.exhibitor.service.] as
[org.apache.druid.curator.ExhibitorConfig@6b760460]
2019-11-12T10:11:28,543 INFO [main] org.apache.curator.utils.Compatibility -
Running in ZooKeeper 3.4.x compatibility mode
2019-11-12T10:11:28,545 WARN [main]
org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30).
Pinning to 29
2019-11-12T10:11:28,583 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.server.initialization.ZkPathsConfig] from
props[druid.zk.paths.] as
[org.apache.druid.server.initialization.ZkPathsConfig@58d3f4be]
2019-11-12T10:11:28,591 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.server.security.AuthConfig] from
props[druid.auth.] as [AuthConfig{authenticatorChain=null, authorizers=null,
unsecuredPaths=[], allowUnauthenticatedHttpOptions=false}]
2019-11-12T10:11:28,642 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class
org.apache.druid.server.metrics.DruidMonitorSchedulerConfig] from
props[druid.monitoring.] as
[org.apache.druid.server.metrics.DruidMonitorSchedulerConfig@7bc6d27a]
2019-11-12T10:11:28,646 INFO [main] org.apache.druid.guice.JsonConfigurator
- Loaded class[class org.apache.druid.server.metrics.MonitorsConfig] from
props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
2019-11-12T10:11:28,647 INFO [main]
org.apache.druid.server.emitter.EmitterModule - Underlying emitter for
ServiceEmitter: org.apache.druid.java.util.emitter.core.NoopEmitter@1ac45389
2019-11-12T10:11:28,647 INFO [main]
org.apache.druid.server.emitter.EmitterModule - Extra service dimensions:
[GitHub] [incubator-druid] anshbansal commented on issue #8835: Ingestion task fails with NullPointerException during BUILD_SEGMENTS phase
anshbansal commented on issue #8835: Ingestion task fails with NullPointerException during BUILD_SEGMENTS phase URL: https://github.com/apache/incubator-druid/issues/8835#issuecomment-552811869 Observed behaviour related to this is that this is not always reproducible. The same task succeeded in the environment at other times. That suggests something related to difference in environment. Going through the druid code it seems there are retries configured (default 3 retries) for fetching the data in some time (default 60s). This could be an environment issue in the setup which is causing those fetches to not be complete in the given time duration. But the NullPointerException should still be improved on druid's side. In the current form this exception makes it hard to pinpoint the problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lgtm-com[bot] commented on issue #8838: Web console: replace (and remove) old consoles
lgtm-com[bot] commented on issue #8838: Web console: replace (and remove) old consoles URL: https://github.com/apache/incubator-druid/pull/8838#issuecomment-552809357 This pull request **fixes 28 alerts** when merging 14f5e93fcb6f3d8ad59c7d1c15d87bb81c9668f5 into df2f77c58d599e74b097027aea46368d378bbeea - [view on LGTM.com](https://lgtm.com/projects/g/apache/incubator-druid/rev/pr-c102e3680f6d6df9c32f0e2b793c5f0900eefd01) **fixed alerts:** * 25 for Unused variable, import, function or class * 1 for Useless assignment to local variable * 1 for Comparison between inconvertible types * 1 for Incomplete string escaping or encoding This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] anshbansal commented on issue #8835: Ingestion task fails with NullPointerException during BUILD_SEGMENTS phase
anshbansal commented on issue #8835: Ingestion task fails with NullPointerException during BUILD_SEGMENTS phase URL: https://github.com/apache/incubator-druid/issues/8835#issuecomment-552786056 This seems to be the PR which introduced these changes https://github.com/apache/incubator-druid/pull/8257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
