[GitHub] [druid] vogievetsky commented on a change in pull request #9587: Use auto-form for add an edit lookups
vogievetsky commented on a change in pull request #9587: Use auto-form for add
an edit lookups
URL: https://github.com/apache/druid/pull/9587#discussion_r405279936
##
File path: web-console/src/dialogs/lookup-edit-dialog/lookup-edit-dialog.tsx
##
@@ -93,8 +194,365 @@ export const LookupEditDialog = React.memo(function
LookupEditDialog(props: Look
}
}
- const disableSubmit =
-lookupName === '' || lookupVersion === '' || lookupTier === '' ||
!validJson(lookupSpec);
+ const fields = [
+{
+ name: 'type',
+ type: 'string',
+ suggestions: ['map', 'cachedNamespace'],
+ adjustment: (model: LookupSpec) => {
+if (model.type === 'map' && model.extractionNamespace &&
model.extractionNamespace.type) {
+ return model;
+}
+model.extractionNamespace = { type: 'uri', namespaceParseSpec: {
format: 'csv' } };
+return model;
+ },
+},
+{
+ name: 'map',
+ type: 'json',
+ defined: (model: LookupSpec) => {
+return model.type === 'map';
+ },
+},
+{
+ name: 'extractionNamespace.type',
+ type: 'string',
+ label: 'Globally cached lookup type',
+ placeholder: 'uri',
+ suggestions: ['uri', 'jdbc'],
+ defined: (model: LookupSpec) => model.type === 'cachedNamespace',
+},
+{
+ name: 'extractionNamespace.uriPrefix',
+ type: 'string',
+ label: 'URI prefix',
+ info:
+'A URI which specifies a directory (or other searchable resource) in
which to search for files',
+ placeholder: 's3://bucket/some/key/prefix/',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.fileRegex',
+ type: 'string',
+ label: 'File regex',
+ placeholder: 'renames-[0-9]*\\.gz',
+ info:
+'Optional regex for matching the file name under uriPrefix. Only used
if uriPrefix is used',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.format',
+ type: 'string',
+ label: 'Format',
+ defaultValue: 'csv',
+ suggestions: ['csv', 'tsv', 'customJson', 'simpleJson'],
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.columns',
+ type: 'string-array',
+ label: 'Columns',
+ placeholder: `["key", "value"]`,
+ info: 'The list of columns in the csv file',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.keyColumn',
+ type: 'string',
+ label: 'Key column',
+ placeholder: 'Key',
+ info: 'The name of the column containing the key',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.valueColumn',
+ type: 'string',
+ label: 'Value column',
+ placeholder: 'Value',
+ info: 'The name of the column containing the value',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.hasHeaderRow',
+ type: 'boolean',
+ label: 'Has header row',
+ defaultValue: false,
+ info: `A flag to indicate that column information can be extracted from
the input files' header row`,
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamesp
[GitHub] [druid] vogievetsky commented on a change in pull request #9587: Use auto-form for add an edit lookups
vogievetsky commented on a change in pull request #9587: Use auto-form for add
an edit lookups
URL: https://github.com/apache/druid/pull/9587#discussion_r405281007
##
File path: web-console/src/dialogs/lookup-edit-dialog/lookup-edit-dialog.tsx
##
@@ -93,8 +194,365 @@ export const LookupEditDialog = React.memo(function
LookupEditDialog(props: Look
}
}
- const disableSubmit =
-lookupName === '' || lookupVersion === '' || lookupTier === '' ||
!validJson(lookupSpec);
+ const fields = [
+{
+ name: 'type',
+ type: 'string',
+ suggestions: ['map', 'cachedNamespace'],
+ adjustment: (model: LookupSpec) => {
+if (model.type === 'map' && model.extractionNamespace &&
model.extractionNamespace.type) {
+ return model;
+}
+model.extractionNamespace = { type: 'uri', namespaceParseSpec: {
format: 'csv' } };
+return model;
+ },
+},
+{
+ name: 'map',
+ type: 'json',
+ defined: (model: LookupSpec) => {
+return model.type === 'map';
+ },
+},
+{
+ name: 'extractionNamespace.type',
+ type: 'string',
+ label: 'Globally cached lookup type',
+ placeholder: 'uri',
+ suggestions: ['uri', 'jdbc'],
+ defined: (model: LookupSpec) => model.type === 'cachedNamespace',
+},
+{
+ name: 'extractionNamespace.uriPrefix',
+ type: 'string',
+ label: 'URI prefix',
+ info:
+'A URI which specifies a directory (or other searchable resource) in
which to search for files',
+ placeholder: 's3://bucket/some/key/prefix/',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.fileRegex',
+ type: 'string',
+ label: 'File regex',
+ placeholder: 'renames-[0-9]*\\.gz',
+ info:
+'Optional regex for matching the file name under uriPrefix. Only used
if uriPrefix is used',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.format',
+ type: 'string',
+ label: 'Format',
+ defaultValue: 'csv',
+ suggestions: ['csv', 'tsv', 'customJson', 'simpleJson'],
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.columns',
+ type: 'string-array',
+ label: 'Columns',
+ placeholder: `["key", "value"]`,
+ info: 'The list of columns in the csv file',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.keyColumn',
+ type: 'string',
+ label: 'Key column',
+ placeholder: 'Key',
+ info: 'The name of the column containing the key',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.valueColumn',
+ type: 'string',
+ label: 'Value column',
+ placeholder: 'Value',
+ info: 'The name of the column containing the value',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.hasHeaderRow',
+ type: 'boolean',
+ label: 'Has header row',
+ defaultValue: false,
+ info: `A flag to indicate that column information can be extracted from
the input files' header row`,
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamesp
[GitHub] [druid] vogievetsky commented on a change in pull request #9587: Use auto-form for add an edit lookups
vogievetsky commented on a change in pull request #9587: Use auto-form for add
an edit lookups
URL: https://github.com/apache/druid/pull/9587#discussion_r405281463
##
File path: web-console/src/dialogs/lookup-edit-dialog/lookup-edit-dialog.tsx
##
@@ -93,8 +194,365 @@ export const LookupEditDialog = React.memo(function
LookupEditDialog(props: Look
}
}
- const disableSubmit =
-lookupName === '' || lookupVersion === '' || lookupTier === '' ||
!validJson(lookupSpec);
+ const fields = [
+{
+ name: 'type',
+ type: 'string',
+ suggestions: ['map', 'cachedNamespace'],
+ adjustment: (model: LookupSpec) => {
+if (model.type === 'map' && model.extractionNamespace &&
model.extractionNamespace.type) {
+ return model;
+}
+model.extractionNamespace = { type: 'uri', namespaceParseSpec: {
format: 'csv' } };
+return model;
+ },
+},
+{
+ name: 'map',
+ type: 'json',
+ defined: (model: LookupSpec) => {
+return model.type === 'map';
+ },
+},
+{
+ name: 'extractionNamespace.type',
+ type: 'string',
+ label: 'Globally cached lookup type',
+ placeholder: 'uri',
+ suggestions: ['uri', 'jdbc'],
+ defined: (model: LookupSpec) => model.type === 'cachedNamespace',
+},
+{
+ name: 'extractionNamespace.uriPrefix',
+ type: 'string',
+ label: 'URI prefix',
+ info:
+'A URI which specifies a directory (or other searchable resource) in
which to search for files',
+ placeholder: 's3://bucket/some/key/prefix/',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.fileRegex',
+ type: 'string',
+ label: 'File regex',
+ placeholder: 'renames-[0-9]*\\.gz',
+ info:
+'Optional regex for matching the file name under uriPrefix. Only used
if uriPrefix is used',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.format',
+ type: 'string',
+ label: 'Format',
+ defaultValue: 'csv',
+ suggestions: ['csv', 'tsv', 'customJson', 'simpleJson'],
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.columns',
+ type: 'string-array',
+ label: 'Columns',
+ placeholder: `["key", "value"]`,
+ info: 'The list of columns in the csv file',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.keyColumn',
+ type: 'string',
+ label: 'Key column',
+ placeholder: 'Key',
+ info: 'The name of the column containing the key',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.valueColumn',
+ type: 'string',
+ label: 'Value column',
+ placeholder: 'Value',
+ info: 'The name of the column containing the value',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.hasHeaderRow',
+ type: 'boolean',
+ label: 'Has header row',
+ defaultValue: false,
+ info: `A flag to indicate that column information can be extracted from
the input files' header row`,
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamesp
[GitHub] [druid] vogievetsky commented on a change in pull request #9587: Use auto-form for add an edit lookups
vogievetsky commented on a change in pull request #9587: Use auto-form for add
an edit lookups
URL: https://github.com/apache/druid/pull/9587#discussion_r405280023
##
File path: web-console/src/dialogs/lookup-edit-dialog/lookup-edit-dialog.tsx
##
@@ -93,8 +194,365 @@ export const LookupEditDialog = React.memo(function
LookupEditDialog(props: Look
}
}
- const disableSubmit =
-lookupName === '' || lookupVersion === '' || lookupTier === '' ||
!validJson(lookupSpec);
+ const fields = [
+{
+ name: 'type',
+ type: 'string',
+ suggestions: ['map', 'cachedNamespace'],
+ adjustment: (model: LookupSpec) => {
+if (model.type === 'map' && model.extractionNamespace &&
model.extractionNamespace.type) {
+ return model;
+}
+model.extractionNamespace = { type: 'uri', namespaceParseSpec: {
format: 'csv' } };
+return model;
+ },
+},
+{
+ name: 'map',
+ type: 'json',
+ defined: (model: LookupSpec) => {
+return model.type === 'map';
+ },
+},
+{
+ name: 'extractionNamespace.type',
+ type: 'string',
+ label: 'Globally cached lookup type',
+ placeholder: 'uri',
+ suggestions: ['uri', 'jdbc'],
+ defined: (model: LookupSpec) => model.type === 'cachedNamespace',
+},
+{
+ name: 'extractionNamespace.uriPrefix',
+ type: 'string',
+ label: 'URI prefix',
+ info:
+'A URI which specifies a directory (or other searchable resource) in
which to search for files',
+ placeholder: 's3://bucket/some/key/prefix/',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.fileRegex',
+ type: 'string',
+ label: 'File regex',
+ placeholder: 'renames-[0-9]*\\.gz',
+ info:
+'Optional regex for matching the file name under uriPrefix. Only used
if uriPrefix is used',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.format',
+ type: 'string',
+ label: 'Format',
+ defaultValue: 'csv',
+ suggestions: ['csv', 'tsv', 'customJson', 'simpleJson'],
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri',
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.columns',
+ type: 'string-array',
+ label: 'Columns',
+ placeholder: `["key", "value"]`,
+ info: 'The list of columns in the csv file',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.keyColumn',
+ type: 'string',
+ label: 'Key column',
+ placeholder: 'Key',
+ info: 'The name of the column containing the key',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.valueColumn',
+ type: 'string',
+ label: 'Value column',
+ placeholder: 'Value',
+ info: 'The name of the column containing the value',
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+ model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+},
+{
+ name: 'extractionNamespace.namespaceParseSpec.hasHeaderRow',
+ type: 'boolean',
+ label: 'Has header row',
+ defaultValue: false,
+ info: `A flag to indicate that column information can be extracted from
the input files' header row`,
+ defined: (model: LookupSpec) =>
+model.type === 'cachedNamespace' &&
+!!model.extractionNamespace &&
+model.extractionNamespace.type === 'uri' &&
+model.extractionNamespace.namespaceParseSpec &&
+(model.extractionNamesp
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405269070
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableFloatColumnSelector.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableFloatColumnSelector extends TestFloatColumnSelector
+{
+
+ private final Float[] floats;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableFloatColumnSelector(Float[] floats)
+ {
+this.floats = floats;
+ }
+
+ @Override
+ public float getFloat()
+ {
+if (floats[index] != null) {
+ return floats[index];
+} else {
+ return NullHandling.ZERO_FLOAT;
Review comment:
@jihoonson, I changed it here in the previous commit:
https://github.com/apache/druid/commit/e2639e571c40069828a24be815eb67670f5aaf1e
But then the CI Build Sql compatibility test
[failed](https://travis-ci.org/github/apache/druid/builds/672332557) for the
Float case, please see below:
```
[ERROR]
testFloatWrappingDimensionSelector(org.apache.druid.segment.WrappingDimensionSelectorTest)
Time elapsed: 0.008 s <<< ERROR!
java.lang.UnsupportedOperationException
```
I assumed calling `getFloat()` was ok after reading through some druid sql
compatibility documentation. Please correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405269070
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableFloatColumnSelector.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableFloatColumnSelector extends TestFloatColumnSelector
+{
+
+ private final Float[] floats;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableFloatColumnSelector(Float[] floats)
+ {
+this.floats = floats;
+ }
+
+ @Override
+ public float getFloat()
+ {
+if (floats[index] != null) {
+ return floats[index];
+} else {
+ return NullHandling.ZERO_FLOAT;
Review comment:
@jihoonson, I changed it here in the previous commit:
https://github.com/apache/druid/commit/e2639e571c40069828a24be815eb67670f5aaf1e
But then the CI Build Sql compatibility test
[failed](https://travis-ci.org/github/apache/druid/builds/672332557) for the
Float case, please see below:
```
[ERROR]
testFloatWrappingDimensionSelector(org.apache.druid.segment.WrappingDimensionSelectorTest)
Time elapsed: 0.008 s <<< ERROR!
java.lang.UnsupportedOperationException
```
I assumed calling this was ok for `float`, but not `long` and `double`
types. Please correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405269070
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableFloatColumnSelector.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableFloatColumnSelector extends TestFloatColumnSelector
+{
+
+ private final Float[] floats;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableFloatColumnSelector(Float[] floats)
+ {
+this.floats = floats;
+ }
+
+ @Override
+ public float getFloat()
+ {
+if (floats[index] != null) {
+ return floats[index];
+} else {
+ return NullHandling.ZERO_FLOAT;
Review comment:
@jihoonson, I changed it here in the previous commit:
https://github.com/apache/druid/commit/e2639e571c40069828a24be815eb67670f5aaf1e
But then the CI Build Sql compatibility test failed for the Float case,
please see below:
```
[ERROR]
testFloatWrappingDimensionSelector(org.apache.druid.segment.WrappingDimensionSelectorTest)
Time elapsed: 0.008 s <<< ERROR!
java.lang.UnsupportedOperationException
```
I assumed calling this was ok for `float`, but not `long` and `double`
types. Please correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405269070
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableFloatColumnSelector.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableFloatColumnSelector extends TestFloatColumnSelector
+{
+
+ private final Float[] floats;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableFloatColumnSelector(Float[] floats)
+ {
+this.floats = floats;
+ }
+
+ @Override
+ public float getFloat()
+ {
+if (floats[index] != null) {
+ return floats[index];
+} else {
+ return NullHandling.ZERO_FLOAT;
Review comment:
@jihoonson, I changed it here in the previous commit:
https://github.com/apache/druid/commit/e2639e571c40069828a24be815eb67670f5aaf1e
But then the CI Build Sql compatibility test
[failed[(https://travis-ci.org/github/apache/druid/builds/672332557) for the
Float case, please see below:
```
[ERROR]
testFloatWrappingDimensionSelector(org.apache.druid.segment.WrappingDimensionSelectorTest)
Time elapsed: 0.008 s <<< ERROR!
java.lang.UnsupportedOperationException
```
I assumed calling this was ok for `float`, but not `long` and `double`
types. Please correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhi751993 commented on issue #9207: Load streaming data from Apache Kafka Problem
abhi751993 commented on issue #9207: Load streaming data from Apache Kafka Problem URL: https://github.com/apache/druid/issues/9207#issuecomment-610761708 Hey @qinyz, were you able to find a solution to this problem? I'm facing the same problem too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #5855: Error enabling http emitter module by set metrics of "io.druid.server.metrics.HistoricalMetricsMonitor" and "io.druid.server.metrics.QueryCountSta
stale[bot] commented on issue #5855: Error enabling http emitter module by set metrics of "io.druid.server.metrics.HistoricalMetricsMonitor" and "io.druid.server.metrics.QueryCountStatsMonitor" URL: https://github.com/apache/druid/issues/5855#issuecomment-610753096 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #5855: Error enabling http emitter module by set metrics of "io.druid.server.metrics.HistoricalMetricsMonitor" and "io.druid.server.metrics.QueryCountStatsMoni
stale[bot] closed issue #5855: Error enabling http emitter module by set metrics of "io.druid.server.metrics.HistoricalMetricsMonitor" and "io.druid.server.metrics.QueryCountStatsMonitor" URL: https://github.com/apache/druid/issues/5855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405253477
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableFloatColumnSelector.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableFloatColumnSelector extends TestFloatColumnSelector
+{
+
+ private final Float[] floats;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableFloatColumnSelector(Float[] floats)
+ {
+this.floats = floats;
+ }
+
+ @Override
+ public float getFloat()
+ {
+if (floats[index] != null) {
+ return floats[index];
+} else {
+ return NullHandling.ZERO_FLOAT;
Review comment:
Would you please fix here too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405254175
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+if (doubles[index] != null) {
+ return doubles[index];
+} else if (NullHandling.replaceWithDefault()) {
+ return NullHandling.ZERO_DOUBLE;
+} else {
+ throw new UnsupportedOperationException();
Review comment:
nit: I don't think it matters much, but perhaps `NullPointerException` or
`InvalidStateException` would be more appropriate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch 0.18.0 updated: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown (#9610) (#9641)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.18.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/0.18.0 by this push:
new 325f299 Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
(#9610) (#9641)
325f299 is described below
commit 325f29977740b1ff446f01b98179e377760e47f4
Author: Maytas Monsereenusorn <52679095+mayt...@users.noreply.github.com>
AuthorDate: Tue Apr 7 16:58:27 2020 -1000
Fix NPE in RemoteTaskRunner event handler causes JVM shutdown (#9610)
(#9641)
* Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
* address comments
* fix compile
* fix checkstyle
* fix lgtm
* fix merge
* fix test
* fix tests
* change scope
* address comments
* address comments
---
extensions-contrib/ambari-metrics-emitter/pom.xml | 12 ++
.../druid/indexing/overlord/RemoteTaskRunner.java | 232 -
.../indexing/overlord/RemoteTaskRunnerTest.java| 36
licenses.yaml | 2 +-
pom.xml| 2 +-
.../druid/curator/discovery/DiscoveryModule.java | 12 ++
6 files changed, 191 insertions(+), 105 deletions(-)
diff --git a/extensions-contrib/ambari-metrics-emitter/pom.xml
b/extensions-contrib/ambari-metrics-emitter/pom.xml
index 7cd69ea..fe94d09 100644
--- a/extensions-contrib/ambari-metrics-emitter/pom.xml
+++ b/extensions-contrib/ambari-metrics-emitter/pom.xml
@@ -131,6 +131,18 @@
JUnitParams
test
+
+ org.codehaus.jackson
+ jackson-core-asl
+ ${codehaus.jackson.version}
+ test
+
+
+ org.codehaus.jackson
+ jackson-mapper-asl
+ ${codehaus.jackson.version}
+ test
+
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
index fe6e8be..dbaadf9 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
@@ -45,6 +45,7 @@ import com.google.common.util.concurrent.SettableFuture;
import org.apache.commons.lang.mutable.MutableInt;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.PathChildrenCache;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
import org.apache.curator.utils.ZKPaths;
import org.apache.druid.concurrent.LifecycleLock;
import org.apache.druid.curator.CuratorUtils;
@@ -969,116 +970,141 @@ public class RemoteTaskRunner implements
WorkerTaskRunner, TaskLogStreamer
);
// Add status listener to the watcher for status changes
- zkWorker.addListener(
- (client, event) -> {
-final String taskId;
-final RemoteTaskRunnerWorkItem taskRunnerWorkItem;
-synchronized (statusLock) {
- try {
-switch (event.getType()) {
- case CHILD_ADDED:
- case CHILD_UPDATED:
-taskId =
ZKPaths.getNodeFromPath(event.getData().getPath());
-final TaskAnnouncement announcement = jsonMapper.readValue(
-event.getData().getData(), TaskAnnouncement.class
-);
-
-log.info(
-"Worker[%s] wrote %s status for task [%s] on [%s]",
-zkWorker.getWorker().getHost(),
-announcement.getTaskStatus().getStatusCode(),
-taskId,
-announcement.getTaskLocation()
-);
-
-// Synchronizing state with ZK
-statusLock.notifyAll();
-
-final RemoteTaskRunnerWorkItem tmp;
-if ((tmp = runningTasks.get(taskId)) != null) {
- taskRunnerWorkItem = tmp;
-} else {
- final RemoteTaskRunnerWorkItem newTaskRunnerWorkItem =
new RemoteTaskRunnerWorkItem(
- taskId,
- announcement.getTaskType(),
- zkWorker.getWorker(),
- TaskLocation.unknown(),
- announcement.getTaskDataSource()
- );
- final RemoteTaskRunnerWorkItem existingItem =
runningTasks.putIfAbsent(
- taskId,
- newTaskRunnerWorkItem
- );
- if (existingItem == null) {
-log.warn(
-"Worker[%s] announced a status for a task I didn't
k
[GitHub] [druid] fjy merged pull request #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
fjy merged pull request #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown URL: https://github.com/apache/druid/pull/9641 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] calvinhkf commented on a change in pull request #9636: Align library version
calvinhkf commented on a change in pull request #9636: Align library version URL: https://github.com/apache/druid/pull/9636#discussion_r405226327 ## File path: extendedset/pom.xml ## @@ -58,7 +58,7 @@ junit junit - 4.8.1 + 4.12 Review comment: Yes. It's better. I have made suggested changes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm opened a new pull request #9644: [WIP] Add API to trigger a compaction by the coordinator for integration tests
maytasm opened a new pull request #9644: [WIP] Add API to trigger a compaction by the coordinator for integration tests URL: https://github.com/apache/druid/pull/9644 Add API to trigger a compaction by the coordinator for integration tests ### Description Implement a new API to manually trigger the compaction by the coordinator. This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] mcbrewster opened a new pull request #9643: add new text to lookup action dialog
mcbrewster opened a new pull request #9643: add new text to lookup action dialog URL: https://github.com/apache/druid/pull/9643 Updates the text in in the lookup action dialog as it takes a little bit of time for a look up to propagate. https://user-images.githubusercontent.com/37322608/78738354-e499a900-790e-11ea-82ad-b6b6ac68f112.png";> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on issue #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on issue #9622: Preserve the null values for numeric type dimensions post-compaction. URL: https://github.com/apache/druid/pull/9622#issuecomment-610708879 @jihoonson, thank you for the review! Could you please take another pass when you get a chance? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405214289
##
File path:
processing/src/test/java/org/apache/druid/segment/TestWrappingDimensionSelector.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestWrappingDimensionSelector
+{
+ @Test
+ public void testLongWrappingDimensionSelector()
+ {
+Long[] vals = new Long[]{24L, null, 50L, 0L, -60L};
+TestNullableLongColumnSelector lngSelector = new
TestNullableLongColumnSelector(vals);
+
+LongWrappingDimensionSelector lngWrapSelector = new
LongWrappingDimensionSelector(lngSelector, null);
+Assert.assertEquals(24L, lngSelector.getLong());
+Assert.assertEquals("24", lngWrapSelector.getValue());
+
+lngSelector.increment();
+Assert.assertEquals(0L, lngSelector.getLong());
Review comment:
Done, that makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405214107
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+return doubles[index] == null ? NullHandling.ZERO_DOUBLE : doubles[index];
+ }
+
+ @Override
+ public boolean isNull()
+ {
+return doubles[index] == null;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null
values for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405213973
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+return doubles[index] == null ? NullHandling.ZERO_DOUBLE : doubles[index];
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction. URL: https://github.com/apache/druid/pull/9622#discussion_r405214013 ## File path: processing/src/test/java/org/apache/druid/segment/TestWrappingDimensionSelector.java ## @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.segment; + +import org.junit.Assert; +import org.junit.Test; + +public class TestWrappingDimensionSelector Review comment: Ack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
jihoonson commented on issue #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown URL: https://github.com/apache/druid/pull/9641#issuecomment-610703502 Restarted This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm commented on issue #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on issue #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown URL: https://github.com/apache/druid/pull/9641#issuecomment-610701493 ^ looks like LGTM failure is intermittent. Can anyone with super power retrigger it for me? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9634: More optimize CNF conversion of filters
jihoonson commented on a change in pull request #9634: More optimize CNF
conversion of filters
URL: https://github.com/apache/druid/pull/9634#discussion_r405205767
##
File path:
processing/src/main/java/org/apache/druid/segment/filter/CnfHelper.java
##
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.filter;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import org.apache.druid.query.filter.BooleanFilter;
+import org.apache.druid.query.filter.Filter;
+
+import javax.annotation.Nonnull;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * A helper class to convert a filter to CNF.
+ *
+ * The methods in this class are mainly adopted from Apache Hive and Apache
Calcite.
+ */
+public class CnfHelper
+{
+ public static Filter toCnf(Filter current)
+ {
+current = pushDownNot(current);
+current = flatten(current);
+current = pull(current);
+current = convertToCNFInternal(current);
+current = flatten(current);
+return current;
+ }
+
+ // A helper function adapted from Apache Hive, see:
+ //
https://github.com/apache/hive/blob/branch-2.0/storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
+ @VisibleForTesting
+ static Filter pushDownNot(Filter current)
+ {
+if (current instanceof NotFilter) {
+ Filter child = ((NotFilter) current).getBaseFilter();
+ if (child instanceof NotFilter) {
+return pushDownNot(((NotFilter) child).getBaseFilter());
+ }
+ if (child instanceof AndFilter) {
+Set children = new HashSet<>();
+for (Filter grandChild : ((AndFilter) child).getFilters()) {
+ children.add(pushDownNot(new NotFilter(grandChild)));
+}
+return new OrFilter(children);
+ }
+ if (child instanceof OrFilter) {
+Set children = new HashSet<>();
+for (Filter grandChild : ((OrFilter) child).getFilters()) {
+ children.add(pushDownNot(new NotFilter(grandChild)));
+}
+return new AndFilter(children);
+ }
+}
+
+if (current instanceof AndFilter) {
+ Set children = new HashSet<>();
+ for (Filter child : ((AndFilter) current).getFilters()) {
+children.add(pushDownNot(child));
+ }
+ return new AndFilter(children);
+}
+
+if (current instanceof OrFilter) {
+ Set children = new HashSet<>();
+ for (Filter child : ((OrFilter) current).getFilters()) {
+children.add(pushDownNot(child));
+ }
+ return new OrFilter(children);
+}
+return current;
+ }
+
+ // A helper function adapted from Apache Hive, see:
+ //
https://github.com/apache/hive/blob/branch-2.0/storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
+ private static Filter convertToCNFInternal(Filter current)
+ {
+if (current instanceof NotFilter) {
+ return new NotFilter(convertToCNFInternal(((NotFilter)
current).getBaseFilter()));
+}
+if (current instanceof AndFilter) {
+ Set children = new HashSet<>();
+ for (Filter child : ((AndFilter) current).getFilters()) {
+children.add(convertToCNFInternal(child));
+ }
+ return new AndFilter(children);
+}
+if (current instanceof OrFilter) {
+ // a list of leaves that weren't under AND expressions
+ List nonAndList = new ArrayList();
+ // a list of AND expressions that we need to distribute
+ List andList = new ArrayList();
+ for (Filter child : ((OrFilter) current).getFilters()) {
+if (child instanceof AndFilter) {
+ andList.add(child);
+} else if (child instanceof OrFilter) {
+ // pull apart the kids of the OR expression
+ nonAndList.addAll(((OrFilter) child).getFilters());
+} else {
+ nonAndList.add(child);
+}
+ }
+ if (!andList.isEmpty()) {
+Set result = new HashSet<>();
+
[GitHub] [druid] jon-wei commented on a change in pull request #9634: More optimize CNF conversion of filters
jon-wei commented on a change in pull request #9634: More optimize CNF
conversion of filters
URL: https://github.com/apache/druid/pull/9634#discussion_r405199617
##
File path:
processing/src/main/java/org/apache/druid/segment/filter/CnfHelper.java
##
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.filter;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import org.apache.druid.query.filter.BooleanFilter;
+import org.apache.druid.query.filter.Filter;
+
+import javax.annotation.Nonnull;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * A helper class to convert a filter to CNF.
+ *
+ * The methods in this class are mainly adopted from Apache Hive and Apache
Calcite.
+ */
+public class CnfHelper
+{
+ public static Filter toCnf(Filter current)
+ {
+current = pushDownNot(current);
+current = flatten(current);
+current = pull(current);
+current = convertToCNFInternal(current);
+current = flatten(current);
+return current;
+ }
+
+ // A helper function adapted from Apache Hive, see:
+ //
https://github.com/apache/hive/blob/branch-2.0/storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
+ @VisibleForTesting
+ static Filter pushDownNot(Filter current)
+ {
+if (current instanceof NotFilter) {
+ Filter child = ((NotFilter) current).getBaseFilter();
+ if (child instanceof NotFilter) {
+return pushDownNot(((NotFilter) child).getBaseFilter());
+ }
+ if (child instanceof AndFilter) {
+Set children = new HashSet<>();
+for (Filter grandChild : ((AndFilter) child).getFilters()) {
+ children.add(pushDownNot(new NotFilter(grandChild)));
+}
+return new OrFilter(children);
+ }
+ if (child instanceof OrFilter) {
+Set children = new HashSet<>();
+for (Filter grandChild : ((OrFilter) child).getFilters()) {
+ children.add(pushDownNot(new NotFilter(grandChild)));
+}
+return new AndFilter(children);
+ }
+}
+
+if (current instanceof AndFilter) {
+ Set children = new HashSet<>();
+ for (Filter child : ((AndFilter) current).getFilters()) {
+children.add(pushDownNot(child));
+ }
+ return new AndFilter(children);
+}
+
+if (current instanceof OrFilter) {
+ Set children = new HashSet<>();
+ for (Filter child : ((OrFilter) current).getFilters()) {
+children.add(pushDownNot(child));
+ }
+ return new OrFilter(children);
+}
+return current;
+ }
+
+ // A helper function adapted from Apache Hive, see:
+ //
https://github.com/apache/hive/blob/branch-2.0/storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
+ private static Filter convertToCNFInternal(Filter current)
+ {
+if (current instanceof NotFilter) {
+ return new NotFilter(convertToCNFInternal(((NotFilter)
current).getBaseFilter()));
+}
+if (current instanceof AndFilter) {
+ Set children = new HashSet<>();
+ for (Filter child : ((AndFilter) current).getFilters()) {
+children.add(convertToCNFInternal(child));
+ }
+ return new AndFilter(children);
+}
+if (current instanceof OrFilter) {
+ // a list of leaves that weren't under AND expressions
+ List nonAndList = new ArrayList();
+ // a list of AND expressions that we need to distribute
+ List andList = new ArrayList();
+ for (Filter child : ((OrFilter) current).getFilters()) {
+if (child instanceof AndFilter) {
+ andList.add(child);
+} else if (child instanceof OrFilter) {
+ // pull apart the kids of the OR expression
+ nonAndList.addAll(((OrFilter) child).getFilters());
+} else {
+ nonAndList.add(child);
+}
+ }
+ if (!andList.isEmpty()) {
+Set result = new HashSet<>();
+
[GitHub] [druid] averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610668434 @himanshug thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610660405 pac4j asks for `scope=openid+profile+email` so I guess that would mean all of above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jcoelh0 opened a new issue #9642: Can't run druid with a kafka supervisor running
jcoelh0 opened a new issue #9642: Can't run druid with a kafka supervisor running URL: https://github.com/apache/druid/issues/9642 Isn't it possible to do a post before running druid? I know it's possible to post a task using the file **\druid-master\examples\bin\post-index-task-main** with ** \druid-master\examples\quickstart\tutorial\wikipedia-index.json**. Why isn't there a way to do this? The only way is through a POST? Like this: `curl -X POST -H 'Content-Type: application/json' -d @supervisor-spec.json http://localhost:8081/druid/indexer/v1/supervisor ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm opened a new pull request #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm opened a new pull request #9641: [Backport] Fix NPE in RemoteTaskRunner event handler causes JVM shutdown URL: https://github.com/apache/druid/pull/9641 Backport of https://github.com/apache/druid/pull/9610 to 0.18.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610653194 @himanshug my team is asking what we want to return such as user id, email address, roles, etc. what should we provide ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405151099
##
File path:
processing/src/test/java/org/apache/druid/segment/TestWrappingDimensionSelector.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestWrappingDimensionSelector
+{
+ @Test
+ public void testLongWrappingDimensionSelector()
+ {
+Long[] vals = new Long[]{24L, null, 50L, 0L, -60L};
+TestNullableLongColumnSelector lngSelector = new
TestNullableLongColumnSelector(vals);
+
+LongWrappingDimensionSelector lngWrapSelector = new
LongWrappingDimensionSelector(lngSelector, null);
+Assert.assertEquals(24L, lngSelector.getLong());
+Assert.assertEquals("24", lngWrapSelector.getValue());
+
+lngSelector.increment();
+Assert.assertEquals(0L, lngSelector.getLong());
Review comment:
`lngSelector.getLong` should be never called when the current value is null
and `NullHandling.sqlCompatible() == true`. Probably better to check
```java
if (NullHandling.sqlCompatible()) {
Assert.assertTrue(lngSelector.isNull());
} else {
Assert.assertEquals(0L, lngSelector.getLong());
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405124829
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+return doubles[index] == null ? NullHandling.ZERO_DOUBLE : doubles[index];
Review comment:
Probably this should return 0 when `doubles[index] == null &&
NullHandling.replaceWithDefault() == true`? Otherwise it probably should throw
an exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405150922
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+return doubles[index] == null ? NullHandling.ZERO_DOUBLE : doubles[index];
+ }
+
+ @Override
+ public boolean isNull()
+ {
+return doubles[index] == null;
Review comment:
Same for other column selectors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction. URL: https://github.com/apache/druid/pull/9622#discussion_r405128094 ## File path: processing/src/test/java/org/apache/druid/segment/TestWrappingDimensionSelector.java ## @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.segment; + +import org.junit.Assert; +import org.junit.Test; + +public class TestWrappingDimensionSelector Review comment: We usually append "Test" at end of the class name instead of the beginning. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405150798
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+return doubles[index] == null ? NullHandling.ZERO_DOUBLE : doubles[index];
+ }
+
+ @Override
+ public boolean isNull()
+ {
+return doubles[index] == null;
Review comment:
Probably this should be `return NullHandling.replaceWithDefault() ? false :
doubles[index] == null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9622: Preserve the null values for numeric type dimensions post-compaction.
jihoonson commented on a change in pull request #9622: Preserve the null values
for numeric type dimensions post-compaction.
URL: https://github.com/apache/druid/pull/9622#discussion_r405124923
##
File path:
processing/src/test/java/org/apache/druid/segment/TestNullableDoubleColumnSelector.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+
+public class TestNullableDoubleColumnSelector extends TestDoubleColumnSelector
+{
+ private final Double[] doubles;
+
+ static {
+NullHandling.initializeForTests();
+ }
+
+ private int index = 0;
+
+ public TestNullableDoubleColumnSelector(Double[] doubles)
+ {
+this.doubles = doubles;
+ }
+
+ @Override
+ public double getDouble()
+ {
+return doubles[index] == null ? NullHandling.ZERO_DOUBLE : doubles[index];
Review comment:
Same for other column selectors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website] branch asf-site updated (a77e868 -> 6546130)
This is an automated email from the ASF dual-hosted git repository. surekha pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/druid-website.git. from a77e868 Fix missing CSS new ea8a0be Autobuild new 08f4493 restore files new 6546130 Merge pull request #75 from implydata/autobuild The 197 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: community/index.html | 12 +++ index.html | 61 +--- 2 files changed, 51 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website] surekhasaharan merged pull request #75: Autobuild
surekhasaharan merged pull request #75: Autobuild URL: https://github.com/apache/druid-website/pull/75 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website] vogievetsky opened a new pull request #75: Autobuild
vogievetsky opened a new pull request #75: Autobuild URL: https://github.com/apache/druid-website/pull/75 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown (#9610)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new b95a1b9 Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
(#9610)
b95a1b9 is described below
commit b95a1b9878588ccc975fd6724a02b18b89c4d46f
Author: Maytas Monsereenusorn <52679095+mayt...@users.noreply.github.com>
AuthorDate: Tue Apr 7 11:53:51 2020 -1000
Fix NPE in RemoteTaskRunner event handler causes JVM shutdown (#9610)
* Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
* address comments
* fix compile
* fix checkstyle
* fix lgtm
* fix merge
* fix test
* fix tests
* change scope
* address comments
* address comments
---
extensions-contrib/ambari-metrics-emitter/pom.xml | 12 ++
.../druid/indexing/overlord/RemoteTaskRunner.java | 232 -
.../indexing/overlord/RemoteTaskRunnerTest.java| 36
licenses.yaml | 2 +-
pom.xml| 2 +-
.../druid/curator/discovery/DiscoveryModule.java | 12 ++
6 files changed, 191 insertions(+), 105 deletions(-)
diff --git a/extensions-contrib/ambari-metrics-emitter/pom.xml
b/extensions-contrib/ambari-metrics-emitter/pom.xml
index 72786ed..aadf49d 100644
--- a/extensions-contrib/ambari-metrics-emitter/pom.xml
+++ b/extensions-contrib/ambari-metrics-emitter/pom.xml
@@ -131,6 +131,18 @@
JUnitParams
test
+
+ org.codehaus.jackson
+ jackson-core-asl
+ ${codehaus.jackson.version}
+ test
+
+
+ org.codehaus.jackson
+ jackson-mapper-asl
+ ${codehaus.jackson.version}
+ test
+
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
index fe6e8be..dbaadf9 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
@@ -45,6 +45,7 @@ import com.google.common.util.concurrent.SettableFuture;
import org.apache.commons.lang.mutable.MutableInt;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.PathChildrenCache;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
import org.apache.curator.utils.ZKPaths;
import org.apache.druid.concurrent.LifecycleLock;
import org.apache.druid.curator.CuratorUtils;
@@ -969,116 +970,141 @@ public class RemoteTaskRunner implements
WorkerTaskRunner, TaskLogStreamer
);
// Add status listener to the watcher for status changes
- zkWorker.addListener(
- (client, event) -> {
-final String taskId;
-final RemoteTaskRunnerWorkItem taskRunnerWorkItem;
-synchronized (statusLock) {
- try {
-switch (event.getType()) {
- case CHILD_ADDED:
- case CHILD_UPDATED:
-taskId =
ZKPaths.getNodeFromPath(event.getData().getPath());
-final TaskAnnouncement announcement = jsonMapper.readValue(
-event.getData().getData(), TaskAnnouncement.class
-);
-
-log.info(
-"Worker[%s] wrote %s status for task [%s] on [%s]",
-zkWorker.getWorker().getHost(),
-announcement.getTaskStatus().getStatusCode(),
-taskId,
-announcement.getTaskLocation()
-);
-
-// Synchronizing state with ZK
-statusLock.notifyAll();
-
-final RemoteTaskRunnerWorkItem tmp;
-if ((tmp = runningTasks.get(taskId)) != null) {
- taskRunnerWorkItem = tmp;
-} else {
- final RemoteTaskRunnerWorkItem newTaskRunnerWorkItem =
new RemoteTaskRunnerWorkItem(
- taskId,
- announcement.getTaskType(),
- zkWorker.getWorker(),
- TaskLocation.unknown(),
- announcement.getTaskDataSource()
- );
- final RemoteTaskRunnerWorkItem existingItem =
runningTasks.putIfAbsent(
- taskId,
- newTaskRunnerWorkItem
- );
- if (existingItem == null) {
-log.warn(
-"Worker[%s] announced a status for a task I didn't
know about, add
[GitHub] [druid] clintropolis merged pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
clintropolis merged pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown URL: https://github.com/apache/druid/pull/9610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] fjy merged pull request #9640: [Backport] check paths used for shuffle intermediary data manager get and delete
fjy merged pull request #9640: [Backport] check paths used for shuffle intermediary data manager get and delete URL: https://github.com/apache/druid/pull/9640 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch 0.18.0 updated: check paths used for shuffle intermediary data manager get and delete (#9630) (#9640)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.18.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/0.18.0 by this push:
new b46d06a check paths used for shuffle intermediary data manager get
and delete (#9630) (#9640)
b46d06a is described below
commit b46d06abc3cc07909b9a52ecb18074937dbcf449
Author: Clint Wylie
AuthorDate: Tue Apr 7 14:33:46 2020 -0700
check paths used for shuffle intermediary data manager get and delete
(#9630) (#9640)
* check paths used for shuffle intermediary data manager get and delete
* add test
* newline
* meh
---
.../java/org/apache/druid/indexer/TaskIdUtils.java | 63
.../org/apache/druid/indexer/TaskIdUtilsTest.java | 111 +
.../druid/indexing/kafka/KafkaConsumerConfigs.java | 4 +-
.../indexing/kafka/supervisor/KafkaSupervisor.java | 4 +-
.../kinesis/supervisor/KinesisSupervisor.java | 4 +-
.../druid/indexing/common/task/AbstractTask.java | 4 +-
.../indexing/common/task/utils/RandomIdUtils.java | 34 ---
.../supervisor/SeekableStreamSupervisor.java | 6 +-
.../indexing/worker/IntermediaryDataManager.java | 3 +
...ermediaryDataManagerManualAddAndDeleteTest.java | 64 +++-
.../tests/indexer/AbstractKafkaIndexerTest.java| 4 +-
server/pom.xml | 6 ++
.../apache/druid/segment/indexing/DataSchema.java | 14 +--
.../druid/segment/indexing/DataSchemaTest.java | 66 ++--
14 files changed, 316 insertions(+), 71 deletions(-)
diff --git a/core/src/main/java/org/apache/druid/indexer/TaskIdUtils.java
b/core/src/main/java/org/apache/druid/indexer/TaskIdUtils.java
new file mode 100644
index 000..a88341b
--- /dev/null
+++ b/core/src/main/java/org/apache/druid/indexer/TaskIdUtils.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexer;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.druid.java.util.common.StringUtils;
+
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class TaskIdUtils
+{
+ private static final Pattern INVALIDCHARS = Pattern.compile("(?s).*[^\\S
].*");
+
+ public static void validateId(String thingToValidate, String
stringToValidate)
+ {
+Preconditions.checkArgument(
+!Strings.isNullOrEmpty(stringToValidate),
+StringUtils.format("%s cannot be null or empty. Please provide a %s.",
thingToValidate, thingToValidate)
+);
+Preconditions.checkArgument(
+!stringToValidate.startsWith("."),
+StringUtils.format("%s cannot start with the '.' character.",
thingToValidate)
+);
+Preconditions.checkArgument(
+!stringToValidate.contains("/"),
+StringUtils.format("%s cannot contain the '/' character.",
thingToValidate)
+);
+Matcher m = INVALIDCHARS.matcher(stringToValidate);
+Preconditions.checkArgument(
+!m.matches(),
+StringUtils.format("%s cannot contain whitespace character except
space.", thingToValidate)
+);
+ }
+
+ public static String getRandomId()
+ {
+final StringBuilder suffix = new StringBuilder(8);
+for (int i = 0; i < Integer.BYTES * 2; ++i) {
+ suffix.append((char) ('a' + ((ThreadLocalRandom.current().nextInt() >>>
(i * 4)) & 0x0F)));
+}
+return suffix.toString();
+ }
+}
diff --git a/core/src/test/java/org/apache/druid/indexer/TaskIdUtilsTest.java
b/core/src/test/java/org/apache/druid/indexer/TaskIdUtilsTest.java
new file mode 100644
index 000..5fed8fb
--- /dev/null
+++ b/core/src/test/java/org/apache/druid/indexer/TaskIdUtilsTest.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this
[GitHub] [druid] himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610626878 yes in the auth server side you would need to add something like `https://druid_router_url:port/druid-ext/druid-pac4j/callback` as redirect uri in the auth server client app configuration. this is the url where auth server redirects browser to, after successful login. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] fjy merged pull request #9639: [Backport] Reuse transformer in stream indexing
fjy merged pull request #9639: [Backport] Reuse transformer in stream indexing URL: https://github.com/apache/druid/pull/9639 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch 0.18.0 updated: Reuse transformer in stream indexing (#9625) (#9639)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.18.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/0.18.0 by this push:
new 4df4eac Reuse transformer in stream indexing (#9625) (#9639)
4df4eac is described below
commit 4df4eacf231501175b61cb830038a1d6ee769601
Author: Jihoon Son
AuthorDate: Tue Apr 7 14:19:30 2020 -0700
Reuse transformer in stream indexing (#9625) (#9639)
* Reuse transformer in stream indexing
* remove unused method
* memoize complied pattern
---
.../druid/data/input/impl/RegexInputFormat.java| 29 +++-
.../apache/druid/data/input/impl/RegexReader.java | 7 +-
.../data/input/impl/RegexInputFormatTest.java | 72 +++
.../SeekableStreamIndexTaskRunner.java | 54 --
.../seekablestream/SettableByteEntityReader.java | 84 ++
.../indexing/seekablestream/StreamChunkParser.java | 81 +
.../druid/segment/transform/TransformSpec.java | 6 --
7 files changed, 283 insertions(+), 50 deletions(-)
diff --git
a/core/src/main/java/org/apache/druid/data/input/impl/RegexInputFormat.java
b/core/src/main/java/org/apache/druid/data/input/impl/RegexInputFormat.java
index 1165516..c3745bc 100644
--- a/core/src/main/java/org/apache/druid/data/input/impl/RegexInputFormat.java
+++ b/core/src/main/java/org/apache/druid/data/input/impl/RegexInputFormat.java
@@ -20,7 +20,10 @@
package org.apache.druid.data.input.impl;
import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
import org.apache.druid.data.input.InputEntity;
import org.apache.druid.data.input.InputEntityReader;
import org.apache.druid.data.input.InputFormat;
@@ -29,12 +32,15 @@ import org.apache.druid.data.input.InputRowSchema;
import javax.annotation.Nullable;
import java.io.File;
import java.util.List;
+import java.util.regex.Pattern;
public class RegexInputFormat implements InputFormat
{
private final String pattern;
private final String listDelimiter;
private final List columns;
+ @JsonIgnore
+ private final Supplier compiledPatternSupplier;
@JsonCreator
public RegexInputFormat(
@@ -46,6 +52,27 @@ public class RegexInputFormat implements InputFormat
this.pattern = pattern;
this.listDelimiter = listDelimiter;
this.columns = columns;
+this.compiledPatternSupplier = Suppliers.memoize(() ->
Pattern.compile(pattern));
+ }
+
+ @JsonProperty
+ public String getPattern()
+ {
+return pattern;
+ }
+
+ @Nullable
+ @JsonProperty
+ public String getListDelimiter()
+ {
+return listDelimiter;
+ }
+
+ @Nullable
+ @JsonProperty
+ public List getColumns()
+ {
+return columns;
}
@Override
@@ -57,6 +84,6 @@ public class RegexInputFormat implements InputFormat
@Override
public InputEntityReader createReader(InputRowSchema inputRowSchema,
InputEntity source, File temporaryDirectory)
{
-return new RegexReader(inputRowSchema, source, pattern, listDelimiter,
columns);
+return new RegexReader(inputRowSchema, source, pattern,
compiledPatternSupplier.get(), listDelimiter, columns);
}
}
diff --git
a/core/src/main/java/org/apache/druid/data/input/impl/RegexReader.java
b/core/src/main/java/org/apache/druid/data/input/impl/RegexReader.java
index 2962ebd..e29e1df 100644
--- a/core/src/main/java/org/apache/druid/data/input/impl/RegexReader.java
+++ b/core/src/main/java/org/apache/druid/data/input/impl/RegexReader.java
@@ -42,7 +42,7 @@ import java.util.regex.Pattern;
public class RegexReader extends TextReader
{
private final String pattern;
- private final Pattern compiled;
+ private final Pattern compiledPattern;
private final Function multiValueFunction;
private List columns;
@@ -51,13 +51,14 @@ public class RegexReader extends TextReader
InputRowSchema inputRowSchema,
InputEntity source,
String pattern,
+ Pattern compiledPattern,
@Nullable String listDelimiter,
@Nullable List columns
)
{
super(inputRowSchema, source);
this.pattern = pattern;
-this.compiled = Pattern.compile(pattern);
+this.compiledPattern = compiledPattern;
final String finalListDelimeter = listDelimiter == null ?
Parsers.DEFAULT_LIST_DELIMITER : listDelimiter;
this.multiValueFunction =
ParserUtils.getMultiValueFunction(finalListDelimeter,
Splitter.on(finalListDelimeter));
this.columns = columns;
@@ -78,7 +79,7 @@ public class RegexReader extends TextReader
private Map parseLine(String line)
{
try {
- final Matcher matcher = compiled.matcher(line);
+ final Matcher matcher = compiledPattern.matcher(line);
if (!matcher.matches()) {
[GitHub] [druid] averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610624500 @himanshug do we have any redirect url setting for Druid SSO, they wanted to know if we have any.They need for authorization code flow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9628: Kerberos in Druid
himanshug commented on issue #9628: Kerberos in Druid URL: https://github.com/apache/druid/issues/9628#issuecomment-610624543 first error could be related to https://stackoverflow.com/questions/47227276/gssexception-defective-token-detected-when-trying-to-authenticate-to-tomcat and that leads to second error This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky commented on a change in pull request #9587: Use auto-form for add an edit lookups
vogievetsky commented on a change in pull request #9587: Use auto-form for add
an edit lookups
URL: https://github.com/apache/druid/pull/9587#discussion_r405116898
##
File path: web-console/src/dialogs/lookup-edit-dialog/lookup-edit-dialog.tsx
##
@@ -93,8 +136,45 @@ export const LookupEditDialog = React.memo(function
LookupEditDialog(props: Look
}
}
- const disableSubmit =
-lookupName === '' || lookupVersion === '' || lookupTier === '' ||
!validJson(lookupSpec);
+ let disableSubmit =
+lookupName === '' ||
+lookupVersion === '' ||
+lookupTier === '' ||
+lookupSpec.type === '' ||
+lookupSpec.type === undefined ||
+(lookupSpec.type === 'map' && lookupSpec.map === undefined) ||
+(lookupSpec.type === 'cachedNamespace' && lookupSpec.extractionNamespace
=== undefined);
+
+ if (!disableSubmit && lookupSpec.type === 'cachedNamespace' &&
lookupSpec.extractionNamespace) {
+const namespaceParseSpec =
lookupSpec.extractionNamespace.namespaceParseSpec;
Review comment:
can you extract his whole check into a function please?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky commented on a change in pull request #9587: Use auto-form for add an edit lookups
vogievetsky commented on a change in pull request #9587: Use auto-form for add
an edit lookups
URL: https://github.com/apache/druid/pull/9587#discussion_r405115366
##
File path: web-console/src/dialogs/lookup-edit-dialog/lookup-edit-dialog.tsx
##
@@ -124,34 +202,302 @@ export const LookupEditDialog = React.memo(function
LookupEditDialog(props: Look
}
/>
-
-
-
- onChange('lookupEditSpec', e)}
-fontSize={12}
-height="40vh"
-width="auto"
-showPrintMargin={false}
-showGutter={false}
-value={lookupSpec}
-editorProps={{ $blockScrolling: Infinity }}
-setOptions={{
- tabSize: 2,
+ {
+ if (
+model.type === 'map' &&
+model.extractionNamespace &&
+model.extractionNamespace.type
+ ) {
+return model;
+ }
+ model.extractionNamespace = { type: 'uri', namespaceParseSpec: {
format: 'csv' } };
+ return model;
+},
+ },
+ {
+name: 'map',
+type: 'json',
+defined: model => {
+ return model.type === 'map';
+},
+ },
+ {
+name: 'extractionNamespace.type',
+type: 'string',
+label: 'Globally cached lookup type',
+placeholder: 'uri',
+suggestions: ['uri', 'jdbc'],
+defined: model => model.type === 'cachedNamespace',
+ },
+ {
+name: 'extractionNamespace.uriPrefix',
+type: 'string',
+label: 'URI prefix',
+info:
+ 'A URI which specifies a directory (or other searchable
resource) in which to search for files',
+placeholder: 's3://bucket/some/key/prefix/',
+defined: model =>
+ model.type === 'cachedNamespace' &&
+ !!model.extractionNamespace &&
+ model.extractionNamespace.type === 'uri',
+ },
+ {
+name: 'extractionNamespace.fileRegex',
+type: 'string',
+label: 'File regex',
+placeholder: 'renames-[0-9]*\\.gz',
+info:
+ 'Optional regex for matching the file name under uriPrefix. Only
used if uriPrefix is used',
+defined: model =>
+ model.type === 'cachedNamespace' &&
+ !!model.extractionNamespace &&
+ model.extractionNamespace.type === 'uri',
+ },
+ {
+name: 'extractionNamespace.namespaceParseSpec.format',
+type: 'string',
+label: 'Format',
+defaultValue: 'csv',
+suggestions: ['csv', 'tsv', 'customJson', 'simpleJson'],
+// todo needs info
+defined: model =>
+ model.type === 'cachedNamespace' &&
+ !!model.extractionNamespace &&
+ model.extractionNamespace.type === 'uri',
+ },
+ {
+name: 'extractionNamespace.namespaceParseSpec.columns',
+type: 'string-array',
+label: 'Columns',
+placeholder: `["key", "value"]`,
+info: 'The list of columns in the csv file',
+defined: model =>
+ model.type === 'cachedNamespace' &&
+ !!model.extractionNamespace &&
+ model.extractionNamespace.type === 'uri' &&
+ model.extractionNamespace.namespaceParseSpec &&
+ (model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+ },
+ {
+name: 'extractionNamespace.namespaceParseSpec.keyColumn',
+type: 'string',
+label: 'Key column',
+placeholder: 'Key',
+info: 'The name of the column containing the key',
+defined: model =>
+ model.type === 'cachedNamespace' &&
+ !!model.extractionNamespace &&
+ model.extractionNamespace.type === 'uri' &&
+ model.extractionNamespace.namespaceParseSpec &&
+ (model.extractionNamespace.namespaceParseSpec.format === 'csv' ||
+model.extractionNamespace.namespaceParseSpec.format === 'tsv'),
+ },
+ {
+name: 'extractionNamespace.namespaceParseSpec.valueColumn',
+type: 'string',
+label: 'Value column',
+placeholder: 'Value',
+info: 'The name of the column containing the value',
+defined: model =>
+ model.type === 'cachedNamespace' &&
+ !!model.extractionNamespace &&
+ model.extractionNamespace.type === 'uri' &&
+ model.extractionNamespace.namespaceParseSpec &&
+ (model.extractionNamespace.namespaceParseSpec.format === 'csv
[GitHub] [druid] lgtm-com[bot] commented on issue #9640: [Backport] check paths used for shuffle intermediary data manager get and delete
lgtm-com[bot] commented on issue #9640: [Backport] check paths used for shuffle intermediary data manager get and delete URL: https://github.com/apache/druid/pull/9640#issuecomment-610608397 This pull request **fixes 2 alerts** when merging 0cca4ea269a5c4f04b5b5049e30835e626decd4f into bc472e85f57216902cd598d4a158f7196c4d254c - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-5213a61ace5c2a88e04361a3efd0268e174cacc7) **fixed alerts:** * 2 for Uncontrolled data used in path expression This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky merged pull request #9567: fix global filter input
vogievetsky merged pull request #9567: fix global filter input URL: https://github.com/apache/druid/pull/9567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky commented on issue #9567: fix global filter input
vogievetsky commented on issue #9567: fix global filter input URL: https://github.com/apache/druid/pull/9567#issuecomment-610605142 Perfect. Thank you for addressing the feedback! 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: fix global filter input (#9567)
This is an automated email from the ASF dual-hosted git repository.
vogievetsky pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 6e50d29 fix global filter input (#9567)
6e50d29 is described below
commit 6e50d29b4e95597337cb389689459b2bc33a4834
Author: mcbrewster <37322608+mcbrews...@users.noreply.github.com>
AuthorDate: Tue Apr 7 14:31:19 2020 -0600
fix global filter input (#9567)
* fix global filter input
* remove clear
* close global filters after clicking apply
* add restFilter
---
.../src/views/load-data-view/load-data-view.tsx| 29 +++---
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git a/web-console/src/views/load-data-view/load-data-view.tsx
b/web-console/src/views/load-data-view/load-data-view.tsx
index 113b104..49d322d 100644
--- a/web-console/src/views/load-data-view/load-data-view.tsx
+++ b/web-console/src/views/load-data-view/load-data-view.tsx
@@ -313,6 +313,7 @@ export interface LoadDataViewState {
selectedFilterIndex: number;
selectedFilter?: DruidFilter;
showGlobalFilter: boolean;
+ newFilterValue?: Record;
// for schema
schemaQueryState: QueryState<{
@@ -2066,7 +2067,7 @@ export class LoadDataView extends
React.PureComponent
{
+ value={newFilterValue}
+ onChange={f => this.setState({ newFilterValue: f })}
+ height="200px"
+/>
+
+
+ {
const curFilter = splitFilter(
deepGet(spec, 'spec.dataSchema.transformSpec.filter'),
);
-const newFilter = joinFilter(deepSet(curFilter, `restFilter`,
f));
+const newFilter = joinFilter(deepSet(curFilter, `restFilter`,
newFilterValue));
this.updateSpec(deepSet(spec,
'spec.dataSchema.transformSpec.filter', newFilter));
+this.setState({ showGlobalFilter: false, newFilterValue:
undefined });
}}
- height="200px"
/>
-
-
-
this.queryForFilter()} />
this.setState({
showGlobalFilter: false })} />
@@ -2116,7 +2122,12 @@ export class LoadDataView extends
React.PureComponent
this.setState({ showGlobalFilter: true })}
+onClick={() =>
+ this.setState({
+showGlobalFilter: true,
+newFilterValue: restFilter,
+ })
+}
/>
);
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated (0ba62cd -> 547c6b2)
This is an automated email from the ASF dual-hosted git repository. surekha pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git. from 0ba62cd Merge pull request #117 from rachelpedreschi/patch-10 add dcc1952 add new committers add a594f60 fix github link new 547c6b2 Merge pull request #119 from apache/add--committers The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: community/index.md | 73 -- 1 file changed, 38 insertions(+), 35 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] 01/01: Merge pull request #119 from apache/add--committers
This is an automated email from the ASF dual-hosted git repository. surekha pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git commit 547c6b28e7a44dce583ecf4570cba17b66da9844 Merge: 0ba62cd a594f60 Author: Surekha AuthorDate: Tue Apr 7 13:29:53 2020 -0700 Merge pull request #119 from apache/add--committers update community page community/index.md | 73 -- 1 file changed, 38 insertions(+), 35 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] surekhasaharan merged pull request #119: update community page
surekhasaharan merged pull request #119: update community page URL: https://github.com/apache/druid-website-src/pull/119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #8481: Auto compaction can get stuck
jihoonson commented on issue #8481: Auto compaction can get stuck URL: https://github.com/apache/druid/issues/8481#issuecomment-610593777 Fixed in #8573. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson closed issue #8481: Auto compaction can get stuck
jihoonson closed issue #8481: Auto compaction can get stuck URL: https://github.com/apache/druid/issues/8481 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] averma111 commented on issue #9628: Kerberos in Druid
averma111 commented on issue #9628: Kerberos in Druid URL: https://github.com/apache/druid/issues/9628#issuecomment-610570491 @himanshug we have cleared cookies several time, everything is working excpet UI is not opening. The first time your open the router you get below error GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) when we fresh it again we get below error Problem accessing /unified-console.html. Reason: org.apache.hadoop.security.authentication.util.SignerException: Invalid signed text: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis opened a new pull request #9640: [Backport] check paths used for shuffle intermediary data manager get and delete
clintropolis opened a new pull request #9640: [Backport] check paths used for shuffle intermediary data manager get and delete URL: https://github.com/apache/druid/pull/9640 Backport of #9630 to 0.18.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on issue #9622: Preserve the null values for numeric type dimensions post-compaction.
abhishekrb19 commented on issue #9622: Preserve the null values for numeric type dimensions post-compaction. URL: https://github.com/apache/druid/pull/9622#issuecomment-610564783 @clintropolis, thanks for the pointer! I went ahead and followed one of your suggestions to add similar selector `TestNullable*ColumnSelector` for each type that returns `null` based on the values initialized. I verified that the newly added test cases didn't work as expected _before_ the fix in this patch, but does _after_ the fix. Could you please take a look? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
averma111 commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610562031 Sure, we will rebuild again and follow the instructions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson opened a new pull request #9639: [Backport] Reuse transformer in stream indexing
jihoonson opened a new pull request #9639: [Backport] Reuse transformer in stream indexing URL: https://github.com/apache/druid/pull/9639 Backport of #9625 to 0.18.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (d267b1c -> 73a6baa)
This is an automated email from the ASF dual-hosted git repository.
ccaominh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.
from d267b1c check paths used for shuffle intermediary data manager get
and delete (#9630)
add 73a6baa change hadoop inputSource IT to use parallel batch ingestion
(#9616)
No new revisions were added by this update.
Summary of changes:
...Test.java => AbstractHdfsInputSourceParallelIndexTest.java} | 10 --
...impleIndexTest.java => ITHdfsToAzureParallelIndexTest.java} | 2 +-
...sSimpleIndexTest.java => ITHdfsToGcsParallelIndexTest.java} | 2 +-
...SimpleIndexTest.java => ITHdfsToHdfsParallelIndexTest.java} | 2 +-
...S3SimpleIndexTest.java => ITHdfsToS3ParallelIndexTest.java} | 2 +-
5 files changed, 12 insertions(+), 6 deletions(-)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{AbstractHdfsInputSourceSimpleIndexTest.java
=> AbstractHdfsInputSourceParallelIndexTest.java} (89%)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{ITHdfsToAzureSimpleIndexTest.java
=> ITHdfsToAzureParallelIndexTest.java} (95%)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{ITHdfsToGcsSimpleIndexTest.java
=> ITHdfsToGcsParallelIndexTest.java} (95%)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{ITHdfsToHdfsSimpleIndexTest.java
=> ITHdfsToHdfsParallelIndexTest.java} (94%)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{ITHdfsToS3SimpleIndexTest.java
=> ITHdfsToS3ParallelIndexTest.java} (95%)
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ccaominh merged pull request #9616: Change Hadoop inputSource integration tests to use parallel batch ingestion
ccaominh merged pull request #9616: Change Hadoop inputSource integration tests to use parallel batch ingestion URL: https://github.com/apache/druid/pull/9616 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9421: Unknown complex types break Druid SQL
jihoonson commented on issue #9421: Unknown complex types break Druid SQL URL: https://github.com/apache/druid/issues/9421#issuecomment-610548834 AFAIK, we have different compatibility policies in different areas. There are two areas I'm aware of. - For APIs, we have `PublicApi` and `ExtensionPoint` annotations. For those APIs annotated with one of them, there could be incompatible changes in those APIs between major releases. For non-annotated APIs, they can change in breaking ways in any releases. - For segment read, we should be able to read any format of segments no matter how old Druid created them (I didn't know this policy when I raised https://github.com/apache/druid/pull/9246). This applies to segment format, all shardSpecs, and all loadSpecs. These policies apply to across all Druid modules including contrib extensions. Based on these policies, if the aggregator type has changed inadvertently, that's a compatibility bug and should be fixed by supporting both old and new types. However, I think your PR might be still useful since people sometimes might want to run Druid without some extensions (for example, they just want to stop using those extensions). Even in that case, as you mentioned, it would be nice that Druid could be still functioning. Your PR could address this use case. I'll review it soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9579: Add Apache Ranger Authorization
himanshug commented on issue #9579: Add Apache Ranger Authorization URL: https://github.com/apache/druid/pull/9579#issuecomment-610536889 (1) sounds good to me (2) sounds good as well but seems not doable unless that version of Ranger is released ? In that can probably leave that as known issue for now and include it in docs for this extension so that users can make an informed decision. I am not yet sure what is the impact of this, so documenting that would be nice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: check paths used for shuffle intermediary data manager get and delete (#9630)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new d267b1c check paths used for shuffle intermediary data manager get
and delete (#9630)
d267b1c is described below
commit d267b1c414b9b55b129729692755273d2a35e304
Author: Clint Wylie
AuthorDate: Tue Apr 7 09:47:18 2020 -0700
check paths used for shuffle intermediary data manager get and delete
(#9630)
* check paths used for shuffle intermediary data manager get and delete
* add test
* newline
* meh
---
.../java/org/apache/druid/indexer/TaskIdUtils.java | 63
.../org/apache/druid/indexer/TaskIdUtilsTest.java | 111 +
.../druid/indexing/kafka/KafkaConsumerConfigs.java | 4 +-
.../indexing/kafka/supervisor/KafkaSupervisor.java | 4 +-
.../kinesis/supervisor/KinesisSupervisor.java | 4 +-
.../druid/indexing/common/task/AbstractTask.java | 4 +-
.../indexing/common/task/utils/RandomIdUtils.java | 34 ---
.../supervisor/SeekableStreamSupervisor.java | 6 +-
.../indexing/worker/IntermediaryDataManager.java | 3 +
...ermediaryDataManagerManualAddAndDeleteTest.java | 64 +++-
.../tests/indexer/AbstractKafkaIndexerTest.java| 4 +-
.../apache/druid/segment/indexing/DataSchema.java | 14 +--
.../druid/segment/indexing/DataSchemaTest.java | 28 +++---
13 files changed, 268 insertions(+), 75 deletions(-)
diff --git a/core/src/main/java/org/apache/druid/indexer/TaskIdUtils.java
b/core/src/main/java/org/apache/druid/indexer/TaskIdUtils.java
new file mode 100644
index 000..a88341b
--- /dev/null
+++ b/core/src/main/java/org/apache/druid/indexer/TaskIdUtils.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexer;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.druid.java.util.common.StringUtils;
+
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class TaskIdUtils
+{
+ private static final Pattern INVALIDCHARS = Pattern.compile("(?s).*[^\\S
].*");
+
+ public static void validateId(String thingToValidate, String
stringToValidate)
+ {
+Preconditions.checkArgument(
+!Strings.isNullOrEmpty(stringToValidate),
+StringUtils.format("%s cannot be null or empty. Please provide a %s.",
thingToValidate, thingToValidate)
+);
+Preconditions.checkArgument(
+!stringToValidate.startsWith("."),
+StringUtils.format("%s cannot start with the '.' character.",
thingToValidate)
+);
+Preconditions.checkArgument(
+!stringToValidate.contains("/"),
+StringUtils.format("%s cannot contain the '/' character.",
thingToValidate)
+);
+Matcher m = INVALIDCHARS.matcher(stringToValidate);
+Preconditions.checkArgument(
+!m.matches(),
+StringUtils.format("%s cannot contain whitespace character except
space.", thingToValidate)
+);
+ }
+
+ public static String getRandomId()
+ {
+final StringBuilder suffix = new StringBuilder(8);
+for (int i = 0; i < Integer.BYTES * 2; ++i) {
+ suffix.append((char) ('a' + ((ThreadLocalRandom.current().nextInt() >>>
(i * 4)) & 0x0F)));
+}
+return suffix.toString();
+ }
+}
diff --git a/core/src/test/java/org/apache/druid/indexer/TaskIdUtilsTest.java
b/core/src/test/java/org/apache/druid/indexer/TaskIdUtilsTest.java
new file mode 100644
index 000..5fed8fb
--- /dev/null
+++ b/core/src/test/java/org/apache/druid/indexer/TaskIdUtilsTest.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of th
[GitHub] [druid] jihoonson merged pull request #9630: check paths used for shuffle intermediary data manager get and delete
jihoonson merged pull request #9630: check paths used for shuffle intermediary data manager get and delete URL: https://github.com/apache/druid/pull/9630 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] surekhasaharan merged pull request #117: Update events.yml
surekhasaharan merged pull request #117: Update events.yml URL: https://github.com/apache/druid-website-src/pull/117 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated: Update events.yml
This is an automated email from the ASF dual-hosted git repository. surekha pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git The following commit(s) were added to refs/heads/master by this push: new 4931b34 Update events.yml new 0ba62cd Merge pull request #117 from rachelpedreschi/patch-10 4931b34 is described below commit 4931b343c5ca043d570f30b371c250a527fe00bf Author: Rachel Pedreschi <53063158+rachelpedres...@users.noreply.github.com> AuthorDate: Fri Apr 3 12:25:25 2020 -0700 Update events.yml --- _data/events.yml | 13 + 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/_data/events.yml b/_data/events.yml index f3bfffe..3502104 100644 --- a/_data/events.yml +++ b/_data/events.yml @@ -1,8 +1,13 @@ -- date: 2020-03-24 - name: "Virtual Meetup- AGA (*Ask Gian Anything*)" +- date: 2020-04-08 + name: "Virtual Meetup- AGA v2.0 (*Ask Gian Anything*)" info: "Hosted by the Bay Area Apache Druid Meetup Group" - link: https://www.meetup.com/druidio/events/269554006/ - + link: https://bit.ly/3dMtdtl + +- date: 2020-04-09 + name: "Virtual Meetup- Apache Druid 101: Analytics for Real-time Intelligence" + info: "Hosted by the Chicago Apache Druid Meetup Group" + link: https://bit.ly/39GE453 + - date: 2020-04-13 name: "Virtual Druid Summit" info: "The comfort of your own laptop" - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9636: Align library version
suneet-s commented on a change in pull request #9636: Align library version URL: https://github.com/apache/druid/pull/9636#discussion_r404938818 ## File path: extendedset/pom.xml ## @@ -58,7 +58,7 @@ junit junit - 4.8.1 + 4.12 Review comment: Thanks for cleaning this up! I looked through all uses of junit in the poms, and it looks like this is the only sub-module that explicitly specifies the version. Can we exclude the version here so that it will pick up the version specified in the top level pom? Similar comment for JUnitParams below This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] bolkedebruin commented on issue #9579: Add Apache Ranger Authorization
bolkedebruin commented on issue #9579: Add Apache Ranger Authorization URL: https://github.com/apache/druid/pull/9579#issuecomment-610358321 @ccaominh @himanshug my suggestion to fix this is 1) replace log4j with a recent version for this dependency - I just built Ranger with the latest log4j and it's a drop in 2) remove kafka. This requires a patch which is in a unreleased version of Ranger. It also not often used This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on issue #9630: check paths used for shuffle intermediary data manager get and delete
lgtm-com[bot] commented on issue #9630: check paths used for shuffle intermediary data manager get and delete URL: https://github.com/apache/druid/pull/9630#issuecomment-610344733 This pull request **fixes 2 alerts** when merging d120c288fecaab124e1953f373ea6c2b9e842d1a into 79522f3e25f434caa2e7e56aa97a064a91af805f - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-5947963af448f01faa00550304f9ca50fe070b42) **fixed alerts:** * 2 for Uncontrolled data used in path expression This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #5180: MetricsSpec fields having type doubleSum returned 0.0 on groupBy query
stale[bot] closed issue #5180: MetricsSpec fields having type doubleSum returned 0.0 on groupBy query URL: https://github.com/apache/druid/issues/5180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #5180: MetricsSpec fields having type doubleSum returned 0.0 on groupBy query
stale[bot] commented on issue #5180: MetricsSpec fields having type doubleSum returned 0.0 on groupBy query URL: https://github.com/apache/druid/issues/5180#issuecomment-610325528 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis opened a new pull request #9638: refactor internal type system
clintropolis opened a new pull request #9638: refactor internal type system
URL: https://github.com/apache/druid/pull/9638
### Description
This PR begins to refactor the Druid internal column type system in an
effort to one day live in a world where we can have complete information about
the types of all columns taking part in a query - at all layers, i.e. a
complete `RowSignature` and `ColumnCapabilities`.
**tl;dr** this PR consolidates `ValueType` enums, adds type information to
`PostAggregator`, and adds finalized type information to `AggregatorFactory`.
The 2 separate `ValueType` enumerations have been consolidated, and now live
in `druid-core`. The methods in `ValueType` which required `druid-processing`
have been migrated into static methods in a new `ValueTypes` static class,
which takes a `ValueType` and produces a `makeNumericWrappingDimensionSelector`
or `makeNewSettableColumnValueSelector` as appropriate. There might be a better
home for this, perhaps `RowSignature` or .. somewhere? `ValueTypes` also
provides a method, `aggregatorTypeNameToType` which centralizes the
`GuavaUtils.getEnumIfPresent` pattern used to convert aggregator type names
into `ValueType`.
`PostAggregator` now has 2 new methods:
```
String getTypeName();
default ValueType getType()
{
return ValueTypes.aggregatorTypeNameToType(getTypeName());
}
```
and will now correctly populate a `RowSignature` instead of setting the type
to `null`.
To support type information on `PostAggregator`, `AggregatorFactory` also
has 3 new methods:
```
public String getFinalizedTypeName()
{
return getTypeName();
}
public ValueType getType()
{
return ValueTypes.aggregatorTypeNameToType(getTypeName());
}
public ValueType getFinalizedType()
{
return ValueTypes.aggregatorTypeNameToType(getFinalizedTypeName());
}
```
so that finalized type information is available to correctly compute the
output type of `FinalizingFieldAccessPostAggregator`. This also means that
finalized type information for aggregators _could_ be available in a finalized
view of a `RowSignature`, but I have not made this modification yet.
Because a lot of postaggs produce `double` typed arrays in their results,
and in anticipation of what I think might be a good idea to do next which is
merge `ValueType` and `ExprType`, I went ahead and added array types to the new
and improved `ValueType` enumeration. The complete set is now:
```
DOUBLE,
FLOAT,
LONG,
STRING,
COMPLEX,
DOUBLE_ARRAY,
LONG_ARRAY,
STRING_ARRAY
```
This is perhaps controversial and worth discussion, but it also doesn't seem
to introduce any ill effects as far as I can tell.
Since this PR adds types for almost every _other_ type of column, we should
now be able to have nearly complete type information for all inputs when we are
constructing an `Expr` for `ExpressionVirtualColumn` and
`ExpressionPostAggregator` and `ExpressionFilter` and anything else using
`Expr`. This means that as a follow-up we should also be able to allow an
`Expr` to compute it's output `ValueType` based on its inputs, making explicit
user specification no longer necessary. I suspect we could also use it for type
specific optimizations to help make expressions faster, and more.
This PR has:
- [ ] been self-reviewed.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
- [ ] been tested in a test Druid cluster.
# Key changed/added classes in this PR
* `ValueType` + `ValueType` = `ValueType`
* `RowSignature`
* like every `PostAggregator` and `AggregatorFactory`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] dark-spark2 commented on issue #8139: KIS throwing error messages
dark-spark2 commented on issue #8139: KIS throwing error messages URL: https://github.com/apache/druid/issues/8139#issuecomment-610237244 Hi, I am using kafka ingestion with 2 replicas on druid 0.17 and experenicing similiar behaviour. Any news regarding this bug? Best Regards, Rotem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-610236177 @averma111 @soumyajose0784 can you add the commit from https://github.com/apache/druid/pull/9637/files in your build then you should be able to specify custom ssl trustStore via following configuration in runtime.properties file... ``` druid.extensions.loadList=["druid-pac4j","simple-client-sslcontext",...] druid.auth.authenticatorChain=["pac4j"] druid.auth.pac4j.cookiePassphrase=whatever druid.auth.pac4j.enableCustomSslContext=true druid.auth.pac4j.oidc.clientID=asdafwqerteg druid.auth.pac4j.oidc.clientSecret=asadfewgtewr druid.auth.pac4j.oidc.discoveryURI=https://dev-960309.okta.com/oauth2/default/.well-known/openid-configuration druid.client.https.protocol=TLS druid.client.https.trustStorePassword=p@ssword druid.client.https.trustStorePath=/etc/security/truststore_ldap.jks ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug opened a new pull request #9637: [WIP]druid-pac4j: add ability for custom ssl trust store for talking to auth
himanshug opened a new pull request #9637: [WIP]druid-pac4j: add ability for custom ssl trust store for talking to auth URL: https://github.com/apache/druid/pull/9637 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on a change in pull request #9610: Fix NPE in
RemoteTaskRunner event handler causes JVM shutdown
URL: https://github.com/apache/druid/pull/9610#discussion_r404593305
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
##
@@ -1081,6 +979,117 @@ private boolean cancelWorkerCleanup(String workerHost)
}
}
+ @VisibleForTesting
+ PathChildrenCacheListener getStatusListener(final Worker worker, final
ZkWorker zkWorker, final SettableFuture retVal)
+ {
+return (client, event) -> {
+ final String taskId;
+ final RemoteTaskRunnerWorkItem taskRunnerWorkItem;
+ synchronized (statusLock) {
+try {
+ switch (event.getType()) { // lgtm
[java/dereferenced-value-may-be-null]
+case CHILD_ADDED:
+case CHILD_UPDATED:
+ taskId = ZKPaths.getNodeFromPath(event.getData().getPath()); //
lgtm [java/dereferenced-value-may-be-null]
Review comment:
- Confirmed that event cannot be null. Removed null check for the event.
- Added a null check blocks for event.getData() with different alert message
than the existing message for catching any Exception. This should be more
helpful to operators. Let me know if you have suggestion on different wording
or other KV data to add to alert. This check is for the CHILD_ADDED,
CHILD_UPDATED, and CHILD_REMOVED events which even.getData() must not be null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on a change in pull request #9610: Fix NPE in
RemoteTaskRunner event handler causes JVM shutdown
URL: https://github.com/apache/druid/pull/9610#discussion_r404593305
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
##
@@ -1081,6 +979,117 @@ private boolean cancelWorkerCleanup(String workerHost)
}
}
+ @VisibleForTesting
+ PathChildrenCacheListener getStatusListener(final Worker worker, final
ZkWorker zkWorker, final SettableFuture retVal)
+ {
+return (client, event) -> {
+ final String taskId;
+ final RemoteTaskRunnerWorkItem taskRunnerWorkItem;
+ synchronized (statusLock) {
+try {
+ switch (event.getType()) { // lgtm
[java/dereferenced-value-may-be-null]
+case CHILD_ADDED:
+case CHILD_UPDATED:
+ taskId = ZKPaths.getNodeFromPath(event.getData().getPath()); //
lgtm [java/dereferenced-value-may-be-null]
Review comment:
- Confirmed that event cannot be null. Removed null check for the event.
- Added a null check blocks with different alert message than the existing
message for catching any Exception. This should be more helpful to operators.
Let me know if you have suggestion on different wording or other KV data to add
to alert. This check is for the CHILD_ADDED, CHILD_UPDATED, and CHILD_REMOVED
events which even.getData() must not be null
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] calvinhkf opened a new pull request #9636: Align library version
calvinhkf opened a new pull request #9636: Align library version URL: https://github.com/apache/druid/pull/9636 Align junit and jUnitParams versions to avoid inconsistency. Update version in one module to make library version consistent with that in other modules. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on a change in pull request #9610: Fix NPE in
RemoteTaskRunner event handler causes JVM shutdown
URL: https://github.com/apache/druid/pull/9610#discussion_r404595527
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
##
@@ -969,116 +970,129 @@ private boolean cancelWorkerCleanup(String workerHost)
);
// Add status listener to the watcher for status changes
- zkWorker.addListener(
- (client, event) -> {
-final String taskId;
-final RemoteTaskRunnerWorkItem taskRunnerWorkItem;
-synchronized (statusLock) {
- try {
-switch (event.getType()) {
- case CHILD_ADDED:
- case CHILD_UPDATED:
-taskId =
ZKPaths.getNodeFromPath(event.getData().getPath());
-final TaskAnnouncement announcement = jsonMapper.readValue(
-event.getData().getData(), TaskAnnouncement.class
-);
-
-log.info(
-"Worker[%s] wrote %s status for task [%s] on [%s]",
-zkWorker.getWorker().getHost(),
-announcement.getTaskStatus().getStatusCode(),
-taskId,
-announcement.getTaskLocation()
-);
-
-// Synchronizing state with ZK
-statusLock.notifyAll();
-
-final RemoteTaskRunnerWorkItem tmp;
-if ((tmp = runningTasks.get(taskId)) != null) {
- taskRunnerWorkItem = tmp;
-} else {
- final RemoteTaskRunnerWorkItem newTaskRunnerWorkItem =
new RemoteTaskRunnerWorkItem(
- taskId,
- announcement.getTaskType(),
- zkWorker.getWorker(),
- TaskLocation.unknown(),
- announcement.getTaskDataSource()
- );
- final RemoteTaskRunnerWorkItem existingItem =
runningTasks.putIfAbsent(
- taskId,
- newTaskRunnerWorkItem
- );
- if (existingItem == null) {
-log.warn(
-"Worker[%s] announced a status for a task I didn't
know about, adding to runningTasks: %s",
-zkWorker.getWorker().getHost(),
-taskId
-);
-taskRunnerWorkItem = newTaskRunnerWorkItem;
- } else {
-taskRunnerWorkItem = existingItem;
- }
-}
-
-if
(!announcement.getTaskLocation().equals(taskRunnerWorkItem.getLocation())) {
-
taskRunnerWorkItem.setLocation(announcement.getTaskLocation());
- TaskRunnerUtils.notifyLocationChanged(listeners, taskId,
announcement.getTaskLocation());
-}
+ zkWorker.addListener(getStatusListener(worker, zkWorker, retVal));
+ zkWorker.start();
+ return retVal;
+}
+catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
-if (announcement.getTaskStatus().isComplete()) {
- taskComplete(taskRunnerWorkItem, zkWorker,
announcement.getTaskStatus());
- runPendingTasks();
-}
-break;
- case CHILD_REMOVED:
-taskId =
ZKPaths.getNodeFromPath(event.getData().getPath());
-taskRunnerWorkItem = runningTasks.remove(taskId);
-if (taskRunnerWorkItem != null) {
- log.info("Task[%s] just disappeared!", taskId);
- taskRunnerWorkItem.setResult(TaskStatus.failure(taskId));
- TaskRunnerUtils.notifyStatusChanged(listeners, taskId,
TaskStatus.failure(taskId));
-} else {
- log.info("Task[%s] went bye bye.", taskId);
-}
-break;
- case INITIALIZED:
-if (zkWorkers.putIfAbsent(worker.getHost(), zkWorker) ==
null) {
- retVal.set(zkWorker);
-} else {
- final String message = StringUtils.format(
- "WTF?! Tried to add already-existing worker[%s]",
- worker.getHost()
- );
- log.makeAlert(message)
- .addData("workerHost", worker.getHost())
- .addData("workerIp", worker.getIp())
- .emit();
- retVal.setException(new IllegalStat
[GitHub] [druid] maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on a change in pull request #9610: Fix NPE in
RemoteTaskRunner event handler causes JVM shutdown
URL: https://github.com/apache/druid/pull/9610#discussion_r404594626
##
File path: extensions-contrib/ambari-metrics-emitter/pom.xml
##
@@ -48,6 +48,18 @@
test-jar
test
+
+ org.codehaus.jackson
+ jackson-core-asl
+ ${codehaus.jackson.version}
+ test
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on a change in pull request #9610: Fix NPE in
RemoteTaskRunner event handler causes JVM shutdown
URL: https://github.com/apache/druid/pull/9610#discussion_r404593305
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/RemoteTaskRunner.java
##
@@ -1081,6 +979,117 @@ private boolean cancelWorkerCleanup(String workerHost)
}
}
+ @VisibleForTesting
+ PathChildrenCacheListener getStatusListener(final Worker worker, final
ZkWorker zkWorker, final SettableFuture retVal)
+ {
+return (client, event) -> {
+ final String taskId;
+ final RemoteTaskRunnerWorkItem taskRunnerWorkItem;
+ synchronized (statusLock) {
+try {
+ switch (event.getType()) { // lgtm
[java/dereferenced-value-may-be-null]
+case CHILD_ADDED:
+case CHILD_UPDATED:
+ taskId = ZKPaths.getNodeFromPath(event.getData().getPath()); //
lgtm [java/dereferenced-value-may-be-null]
Review comment:
- Confirmed that event cannot be null. Removed null check for the event.
- Added a separate catch block for NPE with a different alert message than
the existing message for catching any Exception. This should be more helpful to
operators. Let me know if you have suggestion on different wording or other KV
data to add to alert.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown
maytasm commented on a change in pull request #9610: Fix NPE in RemoteTaskRunner event handler causes JVM shutdown URL: https://github.com/apache/druid/pull/9610#discussion_r404586950 ## File path: extensions-contrib/ambari-metrics-emitter/pom.xml ## @@ -48,6 +48,18 @@ test-jar test + Review comment: No. Other modules such as druid-orc-extensions and druid-hdfs-storage fails. This is because jackson-mapper-asl-1.9.13.jar is pulled in by org.apache.hadoop:hadoop-common:jar:2.8.5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
