[GitHub] [druid] clintropolis merged pull request #9191: [Backport] Suppress CVE-2019-20330 for htrace-core-4.0.1 (#9189)
clintropolis merged pull request #9191: [Backport] Suppress CVE-2019-20330 for htrace-core-4.0.1 (#9189) URL: https://github.com/apache/druid/pull/9191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch 0.17.0 updated: Suppress CVE-2019-20330 for htrace-core-4.0.1 (#9189) (#9191)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a commit to branch 0.17.0 in repository https://gitbox.apache.org/repos/asf/druid.git The following commit(s) were added to refs/heads/0.17.0 by this push: new 47fd6da Suppress CVE-2019-20330 for htrace-core-4.0.1 (#9189) (#9191) 47fd6da is described below commit 47fd6da30c670e4dd7f136e6c549cc751461dcb4 Author: Chi Cao Minh AuthorDate: Wed Jan 15 00:56:06 2020 -0800 Suppress CVE-2019-20330 for htrace-core-4.0.1 (#9189) (#9191) CVE-2019-20330 was updated on 14 Jan 2020, which now gets flagged by the security vulnerability scan. Since the CVE is for jackson-databind, via htrace-core-4.0.1, it can be added to the existing list of security vulnerability suppressions for that dependency. --- owasp-dependency-check-suppressions.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/owasp-dependency-check-suppressions.xml b/owasp-dependency-check-suppressions.xml index cf88f39..22ab0ec 100644 --- a/owasp-dependency-check-suppressions.xml +++ b/owasp-dependency-check-suppressions.xml @@ -194,5 +194,6 @@ CVE-2019-16943 CVE-2019-17267 CVE-2019-17531 +CVE-2019-20330 - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed issue #7251: Segment level Redis cache, keys not balanced when using codis or druid clusters
stale[bot] closed issue #7251: Segment level Redis cache, keys not balanced when using codis or druid clusters URL: https://github.com/apache/druid/issues/7251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #7251: Segment level Redis cache, keys not balanced when using codis or druid clusters
stale[bot] commented on issue #7251: Segment level Redis cache, keys not balanced when using codis or druid clusters URL: https://github.com/apache/druid/issues/7251#issuecomment-574595775 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zihengCat commented on issue #9184: New property in `CoordinatorDynamicConfig` to retain a flexible duration for stale pendingSegments
zihengCat commented on issue #9184: New property in `CoordinatorDynamicConfig` to retain a flexible duration for stale pendingSegments URL: https://github.com/apache/druid/pull/9184#issuecomment-574613943 @clintropolis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #6943: InterruptedExceptions should never be ignored in the code.
stale[bot] commented on issue #6943: InterruptedExceptions should never be ignored in the code. URL: https://github.com/apache/druid/pull/6943#issuecomment-574618097 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9181: Speed up String first/last aggregators when folding isn't needed.
clintropolis commented on a change in pull request #9181: Speed up String
first/last aggregators when folding isn't needed.
URL: https://github.com/apache/druid/pull/9181#discussion_r366825374
##
File path:
processing/src/test/java/org/apache/druid/query/aggregation/last/StringLastBufferAggregatorTest.java
##
@@ -81,6 +82,43 @@ public void testBufferAggregate()
}
+ @Test
+ public void testBufferAggregateWithFoldCheck()
+ {
+final long[] timestamps = {1526724600L, 1526724700L, 1526724800L,
1526725900L, 1526725000L};
+final String[] strings = {"", "", "", "", ""};
+Integer maxStringBytes = 1024;
+
+TestLongColumnSelector longColumnSelector = new
TestLongColumnSelector(timestamps);
+TestObjectColumnSelector objectColumnSelector = new
TestObjectColumnSelector<>(strings);
+
+StringLastAggregatorFactory factory = new StringLastAggregatorFactory(
+"billy", "billy", maxStringBytes
+);
+
+StringLastBufferAggregator agg = new StringLastBufferAggregator(
+longColumnSelector,
+objectColumnSelector,
+maxStringBytes,
+true
+);
+
+ByteBuffer buf = ByteBuffer.allocate(factory.getMaxIntermediateSize());
+int position = 0;
+
+agg.init(buf, position);
+//noinspection ForLoopReplaceableByForEach
+for (int i = 0; i < timestamps.length; i++) {
+ aggregateBuffer(longColumnSelector, objectColumnSelector, agg, buf,
position);
+}
+
+SerializablePairLongString sp = ((SerializablePairLongString) agg.get(buf,
position));
+
+
+Assert.assertEquals("expectec last string value", "", sp.rhs);
Review comment:
nit: typo 'expectec'
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9181: Speed up String first/last aggregators when folding isn't needed.
clintropolis commented on a change in pull request #9181: Speed up String
first/last aggregators when folding isn't needed.
URL: https://github.com/apache/druid/pull/9181#discussion_r366814361
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/first/StringFirstLastUtils.java
##
@@ -33,23 +36,63 @@
{
private static final int NULL_VALUE = -1;
+ /**
+ * Shorten "s" to "maxBytes" chars. Fast and loose because these are *chars*
not *bytes*. Use
+ * {@link #chop(String, int)} for slower, but accurate chopping.
+ */
+ @Nullable
+ public static String fastLooseChop(@Nullable final String s, final int
maxBytes)
+ {
+if (s == null || s.length() <= maxBytes) {
+ return s;
+} else {
+ return s.substring(0, maxBytes);
+}
+ }
+
+ /**
+ * Shorten "s" to what could fit in "maxBytes" bytes as UTF-8.
+ */
@Nullable
public static String chop(@Nullable final String s, final int maxBytes)
{
if (s == null) {
return null;
} else {
- // Shorten firstValue to what could fit in maxBytes as UTF-8.
final byte[] bytes = new byte[maxBytes];
final int len = StringUtils.toUtf8WithLimit(s, ByteBuffer.wrap(bytes));
return new String(bytes, 0, len, StandardCharsets.UTF_8);
}
}
+ /**
+ * Returns whether a given value selector *might* contain
SerializablePairLongString objects.
+ */
+ public static boolean selectorNeedsFoldCheck(
+ final BaseObjectColumnValueSelector valueSelector,
+ @Nullable final ColumnCapabilities valueSelectorCapabilities
+ )
+ {
+if (valueSelectorCapabilities != null &&
valueSelectorCapabilities.getType() != ValueType.COMPLEX) {
+ // Known, non-complex type.
+ return false;
+}
+
+if (valueSelector instanceof NilColumnValueSelector) {
+ // Nil column, definitely no SerializablePairLongStrings.
+ return false;
+}
+
+// Check if the reported class could possibly be
SerializablePairLongString.
+final Class clazz = valueSelector.classOfObject();
+return clazz.isAssignableFrom(SerializablePairLongString.class)
Review comment:
Hmm, it looks to me like this method only returns true if a selector
definitely is a `SerializablePairLongString `. Should this just be returning
true if we made it here?
Alternatively, should `needsFoldCheck` be rebranded as `isFold` and the
`aggregate` methods of the first last heap and buffer aggs just be reading a
`SerializablePairLongString` from the value selector directly instead of
calling `readPairFromSelectors`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] leventov commented on issue #9134: Fix concurrency of ComplexMetrics.java
leventov commented on issue #9134: Fix concurrency of ComplexMetrics.java URL: https://github.com/apache/druid/pull/9134#issuecomment-574679563 @capistrant ideally there should be concurrent unit tests, but I wouldn't obsess about it here. (I should probably change the language of the corresponding checklist item because creating correct concurrent unit tests is so hard that it should be justified only in a minority of cases.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] leventov closed issue #9107: Fix concurrency of ComplexMetrics
leventov closed issue #9107: Fix concurrency of ComplexMetrics URL: https://github.com/apache/druid/issues/9107 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] leventov merged pull request #9134: Fix concurrency of ComplexMetrics.java
leventov merged pull request #9134: Fix concurrency of ComplexMetrics.java URL: https://github.com/apache/druid/pull/9134 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (b287711 -> 4716e0b)
This is an automated email from the ASF dual-hosted git repository. leventov pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from b287711 Suppress CVE-2019-20330 for htrace-core-4.0.1 (#9189) add 4716e0b Fix concurrency of ComplexMetrics.java (#9134) No new revisions were added by this update. Summary of changes: .../apache/druid/segment/serde/ComplexMetrics.java | 43 +++--- 1 file changed, 29 insertions(+), 14 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] VladimirIordanov commented on issue #8989: Support different types of Autoscalers
VladimirIordanov commented on issue #8989: Support different types of Autoscalers URL: https://github.com/apache/druid/pull/8989#issuecomment-574688872 Hi @gianm. Any chance for this PR to be reviewed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] kou64yama opened a new pull request #9193: Fix comment in distribution/docker/druid.sh
kou64yama opened a new pull request #9193: Fix comment in distribution/docker/druid.sh URL: https://github.com/apache/druid/pull/9193 ### Description Fix `DRUID_CONFIG` in `distribution/docker/druid.sh` comment to `DRUID_CONFIG_COMMON`. In fact, this script references `DRUID_CONFIG_COMMON`. https://github.com/apache/druid/blob/b2877119d07a973b19aa48cb1e278779bf88e874/distribution/docker/druid.sh#L116-L121 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ritik02 opened a new issue #9194: [Materialized Views] Updating of derivative DataSource dimensions or metrics
ritik02 opened a new issue #9194: [Materialized Views] Updating of derivative
DataSource dimensions or metrics
URL: https://github.com/apache/druid/issues/9194
Hey,
I have started using Materialized views in druid and things seems to work
fine .
Only issue i came across is i am not able to update dimensions or metrics in
existing spec sheet of derivative datasource .
I get the following error in Overlord logs -
org.apache.druid.indexing.materializedview.MaterializedViewSupervisor -
Failed to start MaterializedViewSupervisor-Click_Dump_MView. Metadata in
database(DerivedDataSourceMetadata{baseDataSource=Click_Dump,
dimensions=[sld_dot_tld, ad_position, customer_id, keyword_term,
ad_display_url], metrics=[m_net_bid, m_count]}) is different from new
dataSource metadata(MaterializedViewSupervisorSpec{baseDataSource=Click_Dump,
dimensions=[sld_dot_tld, click_status, ad_position, customer_id, keyword_term,
ad_display_url], metrics=[m_net_bid, m_count]})
My base datasource is Click_Dump and derivate is CLick_Dump_MView.
Is creating a new materialized view with new set of dimensions and metrics
the only way?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ccaominh commented on issue #9190: Docs: move search to the left
ccaominh commented on issue #9190: Docs: move search to the left URL: https://github.com/apache/druid/pull/9190#issuecomment-574777675 The security vulnerabilities travis job failure is unrelated to this PR. Please merge master to get the fix: https://github.com/apache/druid/pull/9189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367026592
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
+
+ public DoubleAnyAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null && !valueSelector.isNull()) {
+ foundValue = valueSelector.getDouble();
+}
+ }
+
+ @Override
+ public Object get()
+ {
+return foundValue;
+ }
+
+ @Override
+ public float getFloat()
+ {
+return foundValue.floatValue();
+ }
+
+ @Override
+ public long getLong()
+ {
+return foundValue.longValue();
+ }
+
+ @Override
+ public double getDouble()
+ {
+return foundValue;
+ }
+
+ @Override
+ public void close()
+ {
+
Review comment:
nit: Add a comment explaining why this function is empty to indicate it is
intentional
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367030965
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367042756
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.putDouble(position, NULL_VALUE);
+ }
+
+ @Override
+ public void aggregate(ByteBuffer buf, int position)
+ {
+if (buf.getDouble(position) == NULL_VALUE && !valueSelector.isNull()) {
+ buf.putDouble(position, valueSelector.getDouble());
+}
+ }
+
+ @Override
+ public Object get(ByteBuffer buf, int position)
+ {
+return buf.getDouble(position);
+ }
+
+ @Override
+ public float getFloat(ByteBuffer buf, int position)
+ {
+return (float) buf.getDouble(position);
+ }
+
+ @Override
+ public long getLong(ByteBuffer buf, int position)
+ {
+return (long) buf.getDouble(position);
+ }
+
+ @Override
+ public double getDouble(ByteBuffer buf, int position)
+ {
+return buf.getDouble(position);
+ }
+
+ @Override
+ public void close()
+ {
+
Review comment:
nit: Comment for empty function please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367026167
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
+
+ public DoubleAnyAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null && !valueSelector.isNull()) {
+ foundValue = valueSelector.getDouble();
+}
+ }
+
+ @Override
+ public Object get()
+ {
+return foundValue;
+ }
+
+ @Override
+ public float getFloat()
+ {
+return foundValue.floatValue();
Review comment:
This can throw an NPE if isNull is true. I can't tell if this is called from
a hot loop or not. @clintropolis or @jon-wei might know how we should deal with
this. I see this potential NPE in other aggregators as well.
If you switch to using a primitive in the class, then all these getters can
simply cast the local variable
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367036857
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
Review comment:
I don't understand why both the parameters here are `fieldName` and both the
parameters in getCombiningFactory are `name` - I see this pattern used in all
the aggregators
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r366709685 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java ## @@ -121,6 +121,11 @@ public static final byte MEAN_CACHE_TYPE_ID = 0x41; + public static final byte LONG_ANY_CACHE_TYPE_ID = 0x42; + public static final byte DOUBLE_ANY_CACHE_TYPE_ID = 0x42; Review comment: I think you need a different byte for each type id here. Also nitpicking here - can you add a comment above the group saying this is for the any aggregators This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367031257
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367033326
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
Review comment:
nit: `Collections.singletonList(fieldName)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: comm
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367047216
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/first/StringAggregatorUtils.java
##
@@ -87,7 +87,7 @@ public static void writePair(
if (pair.rhs != null) {
mutationBuffer.position(position + Long.BYTES + Integer.BYTES);
- mutationBuffer.limit(maxStringBytes);
+ mutationBuffer.limit(position + Long.BYTES + Integer.BYTES +
maxStringBytes);
Review comment:
Why was this change needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367040205
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
Review comment:
Is this so that missing values show up first? Is that the behavior we want?
I don't know the answer...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367024014
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
Review comment:
I think you should store this as a primitive so we don't have to do
un-necessary boxing. Also I think we need another variable in here to track
whether the value is null or not. We would need to implement `isNull` for the
aggregator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367042441
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
Review comment:
I know this is an existing pattern, but I prefer having a package private
all args constructor to make unit testing easier and have the JsonCreator
constructor call the all args constructor. Otherwise the tests need to rely on
the static implementation of `ColumnHolder#storeDoubleAsFloat` which can be a
huge pain to try and mock correctly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367043979
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final double NULL_VALUE = 0;
Review comment:
0 is a valid value for a double. I think `Double.NaN` is safer to indicate
that the value is not found.
Similar comments to `DoubleAnyAggregator` - I think you need to store a byte
to indicate whether or not the value has been found since 0, NaN, etc. are all
valid values that can show up in a double column.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367032686
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] suneet-s commented on a change in pull request #9187: Implement ANY aggregator
suneet-s commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367020210 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java ## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.query.aggregation.any; + +import org.apache.druid.query.aggregation.Aggregator; +import org.apache.druid.segment.BaseDoubleColumnValueSelector; + +public class DoubleAnyAggregator implements Aggregator Review comment: nit: javadocs please - I know most of the aggregators don't have docs, but I think explaining how nulls should be handled in this class is valuable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367059233
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
+
+ public DoubleAnyAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null && !valueSelector.isNull()) {
+ foundValue = valueSelector.getDouble();
+}
+ }
+
+ @Override
+ public Object get()
+ {
+return foundValue;
+ }
+
+ @Override
+ public float getFloat()
+ {
+return foundValue.floatValue();
Review comment:
(Double/Float/Long)AnyAggregators are created by the
(Double/Float/Long)AnyAggregatorFactory. The
(Double/Float/Long)AnyAggregatorFactory extends from
NullableNumericAggregatorFactory. NullableNumericAggregatorFactory will handle
the Null for the (Double/Float/Long)AnyAggregators. If
useDefaultValueForNull=true then we don't have to worry about null since all
null will be convert to non-null (i.e. 0) before aggregation. If
useDefaultValueForNull=false then the NullableNumericAggregatorFactory will
wrap the AnyAggregators in NullableNumericAggregator which already have isNull
check. (Those methods like getFloat also won't be call if isNull is true)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367060153
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
Review comment:
As mentioned before we don't have to worry about null. This is just
considering which is more optimize between checking if Double is null or
checking if boolean is T/F (and I guess memory for storing one Double vs.
storing one double and one Boolean)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9155: Update docs to use index_parallel instead of index
jihoonson commented on a change in pull request #9155: Update docs to use index_parallel instead of index URL: https://github.com/apache/druid/pull/9155#discussion_r367076121 ## File path: docs/tutorials/tutorial-ingestion-spec.md ## @@ -490,44 +421,57 @@ The `dataSchema` is shared across all task types, but each task type has its own ## Define the input source -Now let's define our input source, which is specified in an `ioConfig` object. Each task type has its own type of `ioConfig`. The native batch task uses "firehoses" to read input data, so let's configure a "local" firehose to read the example netflow data we saved earlier: +Now let's define our input source, which is specified in an `ioConfig` object. Each task type has its own type of `ioConfig`. To read input data, we need to specify an `inputSource`. The the example netflow data we saved earlier needs to be read from a local file, which is configured below: Review comment: typo: `The the` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9155: Update docs to use index_parallel instead of index
suneet-s commented on a change in pull request #9155: Update docs to use index_parallel instead of index URL: https://github.com/apache/druid/pull/9155#discussion_r367078545 ## File path: docs/tutorials/tutorial-ingestion-spec.md ## @@ -490,44 +421,57 @@ The `dataSchema` is shared across all task types, but each task type has its own ## Define the input source -Now let's define our input source, which is specified in an `ioConfig` object. Each task type has its own type of `ioConfig`. The native batch task uses "firehoses" to read input data, so let's configure a "local" firehose to read the example netflow data we saved earlier: +Now let's define our input source, which is specified in an `ioConfig` object. Each task type has its own type of `ioConfig`. To read input data, we need to specify an `inputSource`. The the example netflow data we saved earlier needs to be read from a local file, which is configured below: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367085110
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/first/StringAggregatorUtils.java
##
@@ -87,7 +87,7 @@ public static void writePair(
if (pair.rhs != null) {
mutationBuffer.position(position + Long.BYTES + Integer.BYTES);
- mutationBuffer.limit(maxStringBytes);
+ mutationBuffer.limit(position + Long.BYTES + Integer.BYTES +
maxStringBytes);
Review comment:
This is to fix an existing bug i found. The limit method for buffer sets it
at absolute position. The correct limit should be maxStringBytes after where we
are writing the String to the buffer (which is position + Long.BYTES +
Integer.BYTES). Hence, limit should be position + Long.BYTES + Integer.BYTES +
maxStringBytes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #9155: Tutorials use new ingestion spec where possible
jihoonson merged pull request #9155: Tutorials use new ingestion spec where possible URL: https://github.com/apache/druid/pull/9155 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (4716e0b -> 85a3d41)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 4716e0b Fix concurrency of ComplexMetrics.java (#9134) add 85a3d41 Tutorials use new ingestion spec where possible (#9155) No new revisions were added by this update. Summary of changes: docs/development/extensions-core/mysql.md | 6 +- docs/development/extensions-core/postgresql.md | 6 +- docs/tutorials/tutorial-batch.md | 69 ++--- docs/tutorials/tutorial-compaction.md | 4 +- docs/tutorials/tutorial-ingestion-spec.md | 342 + docs/tutorials/tutorial-rollup.md | 35 +-- docs/tutorials/tutorial-transform-spec.md | 35 +-- .../tutorial/compaction-day-granularity.json | 2 +- .../quickstart/tutorial/compaction-init-index.json | 69 ++--- .../tutorial/compaction-keep-granularity.json | 2 +- examples/quickstart/tutorial/deletion-index.json | 69 ++--- examples/quickstart/tutorial/retention-index.json | 69 ++--- examples/quickstart/tutorial/rollup-index.json | 35 +-- examples/quickstart/tutorial/transform-index.json | 35 +-- .../quickstart/tutorial/updates-append-index2.json | 33 +- .../quickstart/tutorial/updates-init-index.json| 33 +- .../tutorial/updates-overwrite-index.json | 33 +- examples/quickstart/tutorial/wikipedia-index.json | 69 ++--- 18 files changed, 427 insertions(+), 519 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9171: Doc update for the new input source and the new input format
suneet-s commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367137986
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
Review comment:
Do I need to add the s3 extension for this support or is it bundled with the
hdfs extension somehow?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9171: Doc update for the new input source and the new input format
suneet-s commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367138675
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
Review comment:
Sorry, please ignore I see that the hadoop-aws module needs to be added -
mentioned below
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9171: Doc update for the new input source and the new input format
suneet-s commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367141503
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
-The HDFS extension can also be used for GCS as deep storage.
+To use the AWS S3 as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
|Property|Possible Values|Description|Default|
||---|---|---|
-|`druid.storage.type`|hdfs||Must be set.|
-|`druid.storage.storageDirectory`||gs://bucket/example/directory|Must be set.|
+|`druid.storage.type`|hdfs| |Must be set.|
+|`druid.storage.storageDirectory`|s3a://bucket/example/directory or
s3n://bucket/example/directory|Path to the deep storage|Must be set.|
-All services that need to access GCS need to have the [GCS connector
jar](https://cloud.google.com/hadoop/google-cloud-storage-connector#manualinstallation)
in their class path. One option is to place this jar in /lib/ and
/extensions/druid-hdfs-storage/
+You also need to include the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html),
especially the `hadoop-aws.jar` in the Druid classpath.
+Run the below command to install the `hadoop-aws.jar` file under
`${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes.
-Tested with Druid 0.9.0, Hadoop 2.7.2 and gcs-connector jar 1.4.4-hadoop2.
-
-
+```bash
+java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps
-h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}";
+cp
${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
${DRUID_HOME}/extensions/druid-hdfs-storage/
+```
-## Native batch ingestion
+Finally, you need to add the below properties in the `core-site.xml`.
+For more configurations, see the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html).
+
+```xml
+
+ fs.s3a.impl
+ org.apache.hadoop.fs.s3a.S3AFileSystem
+ The implementation class of the S3A Filesystem
+
+
+
+ fs.AbstractFileSystem.s3a.impl
+ org.apache.hadoop.fs.s3a.S3A
+ The implementation class of the S3A
AbstractFileSystem.
+
+
+
+ fs.s3a.access.key
+ AWS access key ID. Omit for IAM role-based or provider-based
authentication.
+ your access key
+
+
+
+ fs.s3a.secret.key
+ AWS secret key. Omit for IAM role-based or provider-based
authentication.
+ your secret key
+
+```
-This firehose ingests events from a predefined list of files from a Hadoop
filesystem.
-This firehose is _splittable_ and can be used by [native parallel index
tasks](../../ingestion/native-batch.md#parallel-task).
-Since each split represents an HDFS file, each worker task of `index_parallel`
will read an object.
+ Configuration for Google Cloud Storage
-Sample spec:
+To use the Google cloud Storage as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
-```json
-"firehose" : {
-"type" : "hdfs",
-"paths": "/foo/bar,/foo/baz"
-}
+|Property|Possible Values|Description|Default|
+||---|---|---|
+|`druid.storage.type`|hdfs||Must be set.|
+|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the
deep storage|Must be set.|
+
+All services that need to access GCS need to have the [GCS connector
jar](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md)
in their class path.
+One option is to place this jar in `${DRUID_HOME}/lib/` and
`${DRUID_HOME}/extensions/druid-hdfs-storage/`.
+
+Finally, you need to add the below properties in the `core-site.xml`.
+For mo
[GitHub] [druid] suneet-s commented on a change in pull request #9171: Doc update for the new input source and the new input format
suneet-s commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367139554
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
-The HDFS extension can also be used for GCS as deep storage.
+To use the AWS S3 as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
|Property|Possible Values|Description|Default|
||---|---|---|
-|`druid.storage.type`|hdfs||Must be set.|
-|`druid.storage.storageDirectory`||gs://bucket/example/directory|Must be set.|
+|`druid.storage.type`|hdfs| |Must be set.|
+|`druid.storage.storageDirectory`|s3a://bucket/example/directory or
s3n://bucket/example/directory|Path to the deep storage|Must be set.|
-All services that need to access GCS need to have the [GCS connector
jar](https://cloud.google.com/hadoop/google-cloud-storage-connector#manualinstallation)
in their class path. One option is to place this jar in /lib/ and
/extensions/druid-hdfs-storage/
+You also need to include the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html),
especially the `hadoop-aws.jar` in the Druid classpath.
+Run the below command to install the `hadoop-aws.jar` file under
`${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes.
-Tested with Druid 0.9.0, Hadoop 2.7.2 and gcs-connector jar 1.4.4-hadoop2.
-
-
+```bash
+java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps
-h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}";
+cp
${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
${DRUID_HOME}/extensions/druid-hdfs-storage/
+```
-## Native batch ingestion
+Finally, you need to add the below properties in the `core-site.xml`.
+For more configurations, see the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html).
+
+```xml
+
+ fs.s3a.impl
+ org.apache.hadoop.fs.s3a.S3AFileSystem
+ The implementation class of the S3A Filesystem
+
+
+
+ fs.AbstractFileSystem.s3a.impl
+ org.apache.hadoop.fs.s3a.S3A
+ The implementation class of the S3A
AbstractFileSystem.
+
+
+
+ fs.s3a.access.key
+ AWS access key ID. Omit for IAM role-based or provider-based
authentication.
+ your access key
+
+
+
+ fs.s3a.secret.key
+ AWS secret key. Omit for IAM role-based or provider-based
authentication.
+ your secret key
+
+```
-This firehose ingests events from a predefined list of files from a Hadoop
filesystem.
-This firehose is _splittable_ and can be used by [native parallel index
tasks](../../ingestion/native-batch.md#parallel-task).
-Since each split represents an HDFS file, each worker task of `index_parallel`
will read an object.
+ Configuration for Google Cloud Storage
-Sample spec:
+To use the Google cloud Storage as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
-```json
-"firehose" : {
-"type" : "hdfs",
-"paths": "/foo/bar,/foo/baz"
-}
+|Property|Possible Values|Description|Default|
+||---|---|---|
+|`druid.storage.type`|hdfs||Must be set.|
+|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the
deep storage|Must be set.|
+
+All services that need to access GCS need to have the [GCS connector
jar](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md)
in their class path.
+One option is to place this jar in `${DRUID_HOME}/lib/` and
`${DRUID_HOME}/extensions/druid-hdfs-storage/`.
+
+Finally, you need to add the below properties in the `core-site.xml`.
+For mo
[GitHub] [druid] suneet-s commented on a change in pull request #9171: Doc update for the new input source and the new input format
suneet-s commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367140667
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
-The HDFS extension can also be used for GCS as deep storage.
+To use the AWS S3 as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
|Property|Possible Values|Description|Default|
||---|---|---|
-|`druid.storage.type`|hdfs||Must be set.|
-|`druid.storage.storageDirectory`||gs://bucket/example/directory|Must be set.|
+|`druid.storage.type`|hdfs| |Must be set.|
+|`druid.storage.storageDirectory`|s3a://bucket/example/directory or
s3n://bucket/example/directory|Path to the deep storage|Must be set.|
-All services that need to access GCS need to have the [GCS connector
jar](https://cloud.google.com/hadoop/google-cloud-storage-connector#manualinstallation)
in their class path. One option is to place this jar in /lib/ and
/extensions/druid-hdfs-storage/
+You also need to include the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html),
especially the `hadoop-aws.jar` in the Druid classpath.
+Run the below command to install the `hadoop-aws.jar` file under
`${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes.
-Tested with Druid 0.9.0, Hadoop 2.7.2 and gcs-connector jar 1.4.4-hadoop2.
-
-
+```bash
+java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps
-h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}";
+cp
${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
${DRUID_HOME}/extensions/druid-hdfs-storage/
+```
-## Native batch ingestion
+Finally, you need to add the below properties in the `core-site.xml`.
+For more configurations, see the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html).
+
+```xml
+
+ fs.s3a.impl
+ org.apache.hadoop.fs.s3a.S3AFileSystem
+ The implementation class of the S3A Filesystem
+
+
+
+ fs.AbstractFileSystem.s3a.impl
+ org.apache.hadoop.fs.s3a.S3A
+ The implementation class of the S3A
AbstractFileSystem.
+
+
+
+ fs.s3a.access.key
+ AWS access key ID. Omit for IAM role-based or provider-based
authentication.
+ your access key
+
+
+
+ fs.s3a.secret.key
+ AWS secret key. Omit for IAM role-based or provider-based
authentication.
+ your secret key
+
+```
-This firehose ingests events from a predefined list of files from a Hadoop
filesystem.
-This firehose is _splittable_ and can be used by [native parallel index
tasks](../../ingestion/native-batch.md#parallel-task).
-Since each split represents an HDFS file, each worker task of `index_parallel`
will read an object.
+ Configuration for Google Cloud Storage
-Sample spec:
+To use the Google cloud Storage as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
-```json
-"firehose" : {
-"type" : "hdfs",
-"paths": "/foo/bar,/foo/baz"
-}
+|Property|Possible Values|Description|Default|
+||---|---|---|
+|`druid.storage.type`|hdfs||Must be set.|
+|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the
deep storage|Must be set.|
+
+All services that need to access GCS need to have the [GCS connector
jar](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md)
in their class path.
+One option is to place this jar in `${DRUID_HOME}/lib/` and
`${DRUID_HOME}/extensions/druid-hdfs-storage/`.
+
+Finally, you need to add the below properties in the `core-site.xml`.
+For mo
[GitHub] [druid] jon-wei commented on a change in pull request #9187: Implement ANY aggregator
jon-wei commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367126090
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
Review comment:
I would also go with primitive double and a boolean found, we can avoid the
boxing and I don't think using one Double object would save memory compared to
double+boolean
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367110861 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java ## @@ -121,6 +121,11 @@ public static final byte MEAN_CACHE_TYPE_ID = 0x41; + public static final byte LONG_ANY_CACHE_TYPE_ID = 0x42; + public static final byte DOUBLE_ANY_CACHE_TYPE_ID = 0x42; Review comment: yes these need to be different values This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367120449 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.query.aggregation.any; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.google.common.base.Preconditions; +import org.apache.druid.java.util.common.StringUtils; +import org.apache.druid.query.aggregation.Aggregator; +import org.apache.druid.query.aggregation.AggregatorFactory; +import org.apache.druid.query.aggregation.AggregatorUtil; +import org.apache.druid.query.aggregation.BufferAggregator; +import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory; +import org.apache.druid.segment.ColumnSelectorFactory; +import org.apache.druid.segment.ColumnValueSelector; +import org.apache.druid.segment.column.ColumnHolder; + +import javax.annotation.Nullable; +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.Objects; + +public class DoubleAnyAggregatorFactory extends NullableNumericAggregatorFactory Review comment: I think this could be `SimpleDoubleAggregatorFactory` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY
aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367119977
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
Review comment:
Since this is created by `NullableNumericAggregatorFactory` it can be a
primitive. `NullableNumericAggregator` and `NullableNumericBufferAggregator`
will initialize to a null value, so `aggregate` will never be called unless you
encounter a not null value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY
aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367116049
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
+
+ public DoubleAnyAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null && !valueSelector.isNull()) {
Review comment:
`valueSelector.isNull()` will never be true since this the factory is
`NullableNumericAggregatorFactory`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367154652 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java ## @@ -121,6 +121,11 @@ public static final byte MEAN_CACHE_TYPE_ID = 0x41; + public static final byte LONG_ANY_CACHE_TYPE_ID = 0x42; + public static final byte DOUBLE_ANY_CACHE_TYPE_ID = 0x42; Review comment: Copy+Paste and forgot to change the value. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367154903 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java ## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.query.aggregation.any; + +import org.apache.druid.query.aggregation.Aggregator; +import org.apache.druid.segment.BaseDoubleColumnValueSelector; + +public class DoubleAnyAggregator implements Aggregator Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm commented on a change in pull request #8881: New `OBJECTARRAY` implementation for `SqlQuery.resultFormat`
gianm commented on a change in pull request #8881: New `OBJECTARRAY`
implementation for `SqlQuery.resultFormat`
URL: https://github.com/apache/druid/pull/8881#discussion_r367156664
##
File path: sql/src/main/java/org/apache/druid/sql/http/ObjectArrayWriter.java
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.http;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.druid.sql.http.ResultFormat.Writer;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.List;
+
+public class ObjectArrayWriter implements Writer
+{
+
+ private final ObjectWriter objectWriter;
+ private final JsonGenerator jsonGenerator;
+ private final OutputStream outputStream;
+
+ public ObjectArrayWriter(ObjectWriter objectWriter)
+ {
+this.objectWriter = objectWriter;
+this.jsonGenerator = objectWriter.getJsonGenerator();
+this.outputStream = objectWriter.getOutputStream();
+ }
+
+ @Override
+ public void writeResponseStart() throws IOException
+ {
+jsonGenerator.writeStartObject();
+jsonGenerator.writeFieldName("data");
+
+objectWriter.writeResponseStart();
+ }
+
+ @Override
+ public void writeHeader(List columnNames) throws IOException
+ {
+objectWriter.writeHeader(columnNames);
+ }
+
+ @Override
+ public void writeRowStart() throws IOException
+ {
+objectWriter.writeRowStart();
+ }
+
+ @Override
+ public void writeRowField(String name, Object value) throws IOException
+ {
+objectWriter.writeRowField(name, value);
+ }
+
+ @Override
+ public void writeRowEnd() throws IOException
+ {
+objectWriter.writeRowEnd();
+ }
+
+ @Override
+ public void writeResponseEnd() throws IOException
+ {
+objectWriter.writeResponseEnd(false);
Review comment:
Instead of adding the new `ObjectWriter#writeResponseEnd(boolean flush)`
method, IMO it makes more sense to simply call
`jsonGenerator.writeEndArray();` here. It reduces coupling between this class
and ObjectWriter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] gianm commented on a change in pull request #8881: New `OBJECTARRAY` implementation for `SqlQuery.resultFormat`
gianm commented on a change in pull request #8881: New `OBJECTARRAY`
implementation for `SqlQuery.resultFormat`
URL: https://github.com/apache/druid/pull/8881#discussion_r367155396
##
File path: sql/src/main/java/org/apache/druid/sql/http/ResultFormat.java
##
@@ -100,6 +100,20 @@ public Writer createFormatter(final OutputStream
outputStream, final ObjectMappe
{
return new ObjectLinesWriter(outputStream, jsonMapper);
}
+ },
+
+ OBJECTARRAY {
Review comment:
This should be added to the doc page `docs/querying/sql.md`. I don't think
the name is very clear though. Maybe "wrappedObject" would be clearer in
relation to the other ones?
- object: array of objects
- array: array of arrays
- wrappedObject: object containing an array of objects
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367157480
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367157886
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
+
+ public DoubleAnyAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null && !valueSelector.isNull()) {
+ foundValue = valueSelector.getDouble();
+}
+ }
+
+ @Override
+ public Object get()
+ {
+return foundValue;
+ }
+
+ @Override
+ public float getFloat()
+ {
+return foundValue.floatValue();
+ }
+
+ @Override
+ public long getLong()
+ {
+return foundValue.longValue();
+ }
+
+ @Override
+ public double getDouble()
+ {
+return foundValue;
+ }
+
+ @Override
+ public void close()
+ {
+
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367158326
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] jihoonson commented on a change in pull request #9171: Doc update for the new input source and the new input format
jihoonson commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367158366
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
-The HDFS extension can also be used for GCS as deep storage.
+To use the AWS S3 as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
|Property|Possible Values|Description|Default|
||---|---|---|
-|`druid.storage.type`|hdfs||Must be set.|
-|`druid.storage.storageDirectory`||gs://bucket/example/directory|Must be set.|
+|`druid.storage.type`|hdfs| |Must be set.|
+|`druid.storage.storageDirectory`|s3a://bucket/example/directory or
s3n://bucket/example/directory|Path to the deep storage|Must be set.|
-All services that need to access GCS need to have the [GCS connector
jar](https://cloud.google.com/hadoop/google-cloud-storage-connector#manualinstallation)
in their class path. One option is to place this jar in /lib/ and
/extensions/druid-hdfs-storage/
+You also need to include the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html),
especially the `hadoop-aws.jar` in the Druid classpath.
+Run the below command to install the `hadoop-aws.jar` file under
`${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes.
-Tested with Druid 0.9.0, Hadoop 2.7.2 and gcs-connector jar 1.4.4-hadoop2.
-
-
+```bash
+java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps
-h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}";
+cp
${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
${DRUID_HOME}/extensions/druid-hdfs-storage/
+```
-## Native batch ingestion
+Finally, you need to add the below properties in the `core-site.xml`.
+For more configurations, see the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html).
+
+```xml
+
+ fs.s3a.impl
+ org.apache.hadoop.fs.s3a.S3AFileSystem
+ The implementation class of the S3A Filesystem
+
+
+
+ fs.AbstractFileSystem.s3a.impl
+ org.apache.hadoop.fs.s3a.S3A
+ The implementation class of the S3A
AbstractFileSystem.
+
+
+
+ fs.s3a.access.key
+ AWS access key ID. Omit for IAM role-based or provider-based
authentication.
+ your access key
+
+
+
+ fs.s3a.secret.key
+ AWS secret key. Omit for IAM role-based or provider-based
authentication.
+ your secret key
+
+```
-This firehose ingests events from a predefined list of files from a Hadoop
filesystem.
-This firehose is _splittable_ and can be used by [native parallel index
tasks](../../ingestion/native-batch.md#parallel-task).
-Since each split represents an HDFS file, each worker task of `index_parallel`
will read an object.
+ Configuration for Google Cloud Storage
-Sample spec:
+To use the Google cloud Storage as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
-```json
-"firehose" : {
-"type" : "hdfs",
-"paths": "/foo/bar,/foo/baz"
-}
+|Property|Possible Values|Description|Default|
+||---|---|---|
+|`druid.storage.type`|hdfs||Must be set.|
+|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the
deep storage|Must be set.|
+
+All services that need to access GCS need to have the [GCS connector
jar](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md)
in their class path.
+One option is to place this jar in `${DRUID_HOME}/lib/` and
`${DRUID_HOME}/extensions/druid-hdfs-storage/`.
+
+Finally, you need to add the below properties in the `core-site.xml`.
+For m
[GitHub] [druid] jihoonson commented on a change in pull request #9171: Doc update for the new input source and the new input format
jihoonson commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367158379
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
-The HDFS extension can also be used for GCS as deep storage.
+To use the AWS S3 as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
|Property|Possible Values|Description|Default|
||---|---|---|
-|`druid.storage.type`|hdfs||Must be set.|
-|`druid.storage.storageDirectory`||gs://bucket/example/directory|Must be set.|
+|`druid.storage.type`|hdfs| |Must be set.|
+|`druid.storage.storageDirectory`|s3a://bucket/example/directory or
s3n://bucket/example/directory|Path to the deep storage|Must be set.|
-All services that need to access GCS need to have the [GCS connector
jar](https://cloud.google.com/hadoop/google-cloud-storage-connector#manualinstallation)
in their class path. One option is to place this jar in /lib/ and
/extensions/druid-hdfs-storage/
+You also need to include the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html),
especially the `hadoop-aws.jar` in the Druid classpath.
+Run the below command to install the `hadoop-aws.jar` file under
`${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes.
-Tested with Druid 0.9.0, Hadoop 2.7.2 and gcs-connector jar 1.4.4-hadoop2.
-
-
+```bash
+java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps
-h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}";
+cp
${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
${DRUID_HOME}/extensions/druid-hdfs-storage/
+```
-## Native batch ingestion
+Finally, you need to add the below properties in the `core-site.xml`.
+For more configurations, see the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html).
+
+```xml
+
+ fs.s3a.impl
+ org.apache.hadoop.fs.s3a.S3AFileSystem
+ The implementation class of the S3A Filesystem
+
+
+
+ fs.AbstractFileSystem.s3a.impl
+ org.apache.hadoop.fs.s3a.S3A
+ The implementation class of the S3A
AbstractFileSystem.
+
+
+
+ fs.s3a.access.key
+ AWS access key ID. Omit for IAM role-based or provider-based
authentication.
+ your access key
+
+
+
+ fs.s3a.secret.key
+ AWS secret key. Omit for IAM role-based or provider-based
authentication.
+ your secret key
+
+```
-This firehose ingests events from a predefined list of files from a Hadoop
filesystem.
-This firehose is _splittable_ and can be used by [native parallel index
tasks](../../ingestion/native-batch.md#parallel-task).
-Since each split represents an HDFS file, each worker task of `index_parallel`
will read an object.
+ Configuration for Google Cloud Storage
-Sample spec:
+To use the Google cloud Storage as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
-```json
-"firehose" : {
-"type" : "hdfs",
-"paths": "/foo/bar,/foo/baz"
-}
+|Property|Possible Values|Description|Default|
+||---|---|---|
+|`druid.storage.type`|hdfs||Must be set.|
+|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the
deep storage|Must be set.|
+
+All services that need to access GCS need to have the [GCS connector
jar](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md)
in their class path.
+One option is to place this jar in `${DRUID_HOME}/lib/` and
`${DRUID_HOME}/extensions/druid-hdfs-storage/`.
+
+Finally, you need to add the below properties in the `core-site.xml`.
+For m
[GitHub] [druid] jihoonson commented on a change in pull request #9171: Doc update for the new input source and the new input format
jihoonson commented on a change in pull request #9171: Doc update for the new
input source and the new input format
URL: https://github.com/apache/druid/pull/9171#discussion_r367158343
##
File path: docs/development/extensions-core/hdfs.md
##
@@ -36,49 +36,105 @@ To use this Apache Druid extension, make sure to
[include](../../development/ext
|`druid.hadoop.security.kerberos.principal`|`dr...@example.com`| Principal
user name |empty|
|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.headlessUser.keytab`|Path
to keytab file|empty|
-If you are using the Hadoop indexer, set your output directory to be a
location on Hadoop and it will work.
+Besides the above settings, you also need to include all Hadoop configuration
files (such as `core-site.xml`, `hdfs-site.xml`)
+in the Druid classpath. One way to do this is copying all those files under
`${DRUID_HOME}/conf/_common`.
+
+If you are using the Hadoop ingestion, set your output directory to be a
location on Hadoop and it will work.
If you want to eagerly authenticate against a secured hadoop/hdfs cluster you
must set `druid.hadoop.security.kerberos.principal` and
`druid.hadoop.security.kerberos.keytab`, this is an alternative to the cron job
method that runs `kinit` command periodically.
-### Configuration for Google Cloud Storage
+### Configuration for Cloud Storage
+
+You can also use the AWS S3 or the Google Cloud Storage as the deep storage
via HDFS.
+
+ Configuration for AWS S3
-The HDFS extension can also be used for GCS as deep storage.
+To use the AWS S3 as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
|Property|Possible Values|Description|Default|
||---|---|---|
-|`druid.storage.type`|hdfs||Must be set.|
-|`druid.storage.storageDirectory`||gs://bucket/example/directory|Must be set.|
+|`druid.storage.type`|hdfs| |Must be set.|
+|`druid.storage.storageDirectory`|s3a://bucket/example/directory or
s3n://bucket/example/directory|Path to the deep storage|Must be set.|
-All services that need to access GCS need to have the [GCS connector
jar](https://cloud.google.com/hadoop/google-cloud-storage-connector#manualinstallation)
in their class path. One option is to place this jar in /lib/ and
/extensions/druid-hdfs-storage/
+You also need to include the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html),
especially the `hadoop-aws.jar` in the Druid classpath.
+Run the below command to install the `hadoop-aws.jar` file under
`${DRUID_HOME}/extensions/druid-hdfs-storage` in all nodes.
-Tested with Druid 0.9.0, Hadoop 2.7.2 and gcs-connector jar 1.4.4-hadoop2.
-
-
+```bash
+java -classpath "${DRUID_HOME}lib/*" org.apache.druid.cli.Main tools pull-deps
-h "org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}";
+cp
${DRUID_HOME}/hadoop-dependencies/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar
${DRUID_HOME}/extensions/druid-hdfs-storage/
+```
-## Native batch ingestion
+Finally, you need to add the below properties in the `core-site.xml`.
+For more configurations, see the [Hadoop AWS
module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html).
+
+```xml
+
+ fs.s3a.impl
+ org.apache.hadoop.fs.s3a.S3AFileSystem
+ The implementation class of the S3A Filesystem
+
+
+
+ fs.AbstractFileSystem.s3a.impl
+ org.apache.hadoop.fs.s3a.S3A
+ The implementation class of the S3A
AbstractFileSystem.
+
+
+
+ fs.s3a.access.key
+ AWS access key ID. Omit for IAM role-based or provider-based
authentication.
+ your access key
+
+
+
+ fs.s3a.secret.key
+ AWS secret key. Omit for IAM role-based or provider-based
authentication.
+ your secret key
+
+```
-This firehose ingests events from a predefined list of files from a Hadoop
filesystem.
-This firehose is _splittable_ and can be used by [native parallel index
tasks](../../ingestion/native-batch.md#parallel-task).
-Since each split represents an HDFS file, each worker task of `index_parallel`
will read an object.
+ Configuration for Google Cloud Storage
-Sample spec:
+To use the Google cloud Storage as the deep storage, you need to configure
`druid.storage.storageDirectory` properly.
-```json
-"firehose" : {
-"type" : "hdfs",
-"paths": "/foo/bar,/foo/baz"
-}
+|Property|Possible Values|Description|Default|
+||---|---|---|
+|`druid.storage.type`|hdfs||Must be set.|
+|`druid.storage.storageDirectory`|gs://bucket/example/directory|Path to the
deep storage|Must be set.|
+
+All services that need to access GCS need to have the [GCS connector
jar](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md)
in their class path.
+One option is to place this jar in `${DRUID_HOME}/lib/` and
`${DRUID_HOME}/extensions/druid-hdfs-storage/`.
+
+Finally, you need to add the below properties in the `core-site.xml`.
+For m
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367158997
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367159577
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For ad
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367160604
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
Review comment:
getCombiningFactory are combining between segments
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367160539
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
Review comment:
There's javadoc on AggregatorFactory class. But i think it's like reusing
input output fields
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9111: Add HashJoinSegment, a virtual segment for joins.
jihoonson commented on issue #9111: Add HashJoinSegment, a virtual segment for joins. URL: https://github.com/apache/druid/pull/9111#issuecomment-574905423 > Removed "getSegmentIdentifier" method from StorageAdapter. It was not being used. Added "Release Notes" label since it's marked as a public API. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367161045
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
Review comment:
Not sure I understand the question. This is for sorting the aggregated
result. I think it does not really matters
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367162064
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
Review comment:
Not sure if the storeDoubleAsFloat is even needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367163512
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.putDouble(position, NULL_VALUE);
+ }
+
+ @Override
+ public void aggregate(ByteBuffer buf, int position)
+ {
+if (buf.getDouble(position) == NULL_VALUE && !valueSelector.isNull()) {
+ buf.putDouble(position, valueSelector.getDouble());
+}
+ }
+
+ @Override
+ public Object get(ByteBuffer buf, int position)
+ {
+return buf.getDouble(position);
+ }
+
+ @Override
+ public float getFloat(ByteBuffer buf, int position)
+ {
+return (float) buf.getDouble(position);
+ }
+
+ @Override
+ public long getLong(ByteBuffer buf, int position)
+ {
+return (long) buf.getDouble(position);
+ }
+
+ @Override
+ public double getDouble(ByteBuffer buf, int position)
+ {
+return buf.getDouble(position);
+ }
+
+ @Override
+ public void close()
+ {
+
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367165160
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367165128
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367165057
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
+
+ @Override
+ protected ColumnValueSelector selector(ColumnSelectorFactory metricFactory)
+ {
+return metricFactory.makeColumnValueSelector(fieldName);
+ }
+
+ @Override
+ protected Aggregator factorize(ColumnSelectorFactory metricFactory,
ColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator factorizeBuffered(ColumnSelectorFactory
metricFactory, ColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ public Comparator getComparator()
+ {
+return DoubleAnyAggregatorFactory.VALUE_COMPARATOR;
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName));
+ }
+
+ @Override
+ public Object deserialize(Object object)
+ {
+return object;
+ }
+
+ @Override
+ @Nullable
+ public Object finalizeComputation(@Nullable Object object)
+ {
+return object;
+ }
+
+ @Override
+ @JsonProperty
+ public String getName()
+ {
+return name;
+ }
+
+ @JsonProperty
+ public String getFieldName()
+ {
+return fieldName;
+ }
+
+ @Override
+ public List requiredFields()
+ {
+return Arrays.asList(fieldName);
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8(fieldName);
+
+return ByteBuffer.allocate(1 + fieldNameBytes.length)
+ .put(AggregatorUtil.DOUBLE_ANY_CACHE_TYPE_ID)
+ .put(fieldNameBytes)
+ .array();
+ }
+
+ @Override
+ public String getTypeName()
+ {
+if (storeDoubleAsFloat) {
+ return "float";
+}
+return "double";
+ }
+
+ @Override
+ public int getMaxIntermediateSize()
+
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367166074
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends
NullableNumericAggregatorFactory
+{
+ private static final Comparator VALUE_COMPARATOR =
Comparator.nullsFirst(
+ Comparator.comparingDouble(Number::doubleValue)
+ );
+
+ private final String fieldName;
+ private final String name;
+ private final boolean storeDoubleAsFloat;
+
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName
+ )
+ {
+Preconditions.checkNotNull(name, "Must have a valid, non-null aggregator
name");
+Preconditions.checkNotNull(fieldName, "Must have a valid, non-null
fieldName");
+this.name = name;
+this.fieldName = fieldName;
+this.storeDoubleAsFloat = ColumnHolder.storeDoubleAsFloat();
+ }
Review comment:
Actually gonna remove storeDoubleAsFloat
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367167409
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final double NULL_VALUE = 0;
Review comment:
Added byte to indicate whether or not the value has been found. We actually
do not get null due to the NullableNumericAggregatorFactory/
NullableNumericAggregator stuff
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367167598
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/first/StringAggregatorUtils.java
##
@@ -87,7 +87,7 @@ public static void writePair(
if (pair.rhs != null) {
mutationBuffer.position(position + Long.BYTES + Integer.BYTES);
- mutationBuffer.limit(maxStringBytes);
+ mutationBuffer.limit(position + Long.BYTES + Integer.BYTES +
maxStringBytes);
Review comment:
Removed this. Will fix as part of separate PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367168182
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregator.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+public class DoubleAnyAggregator implements Aggregator
+{
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ private Double foundValue;
+
+ public DoubleAnyAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null && !valueSelector.isNull()) {
Review comment:
Removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9187: Implement ANY aggregator
maytasm3 commented on issue #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#issuecomment-574914486 Note: ANY on Double/Float/Long column with the useDefaultValueForNull=true will **not** prefer non-null values over default value for null (i.e. 0) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367188578 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.query.aggregation.any; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.google.common.base.Preconditions; +import org.apache.druid.java.util.common.StringUtils; +import org.apache.druid.query.aggregation.Aggregator; +import org.apache.druid.query.aggregation.AggregatorFactory; +import org.apache.druid.query.aggregation.AggregatorUtil; +import org.apache.druid.query.aggregation.BufferAggregator; +import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory; +import org.apache.druid.segment.ColumnSelectorFactory; +import org.apache.druid.segment.ColumnValueSelector; +import org.apache.druid.segment.column.ColumnHolder; + +import javax.annotation.Nullable; +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.Objects; + +public class DoubleAnyAggregatorFactory extends NullableNumericAggregatorFactory Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9187: Implement ANY aggregator
jon-wei commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367204738
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This Aggregator is created by the {@link DoubleAnyAggregatorFactory} which
extends from
+ * {@link NullableNumericAggregatorFactory}. If null needs to be handle, then
{@link NullableNumericAggregatorFactory}
+ * will wrap this aggregator in {@link NullableNumericAggregator} and can
handle all null in that class.
+ * Hence, no null will ever be pass into this aggregator from the
valueSelector.
+ */
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final byte BYTE_FLAG_IS_NOT_SET = 0;
+ private static final byte BYTE_FLAG_IS_SET = 1;
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.put(position, BYTE_FLAG_IS_NOT_SET);
+buf.putDouble(position + Byte.BYTES, NULL_VALUE);
+ }
+
+ @Override
+ public void aggregate(ByteBuffer buf, int position)
+ {
+if (buf.get(position) == BYTE_FLAG_IS_NOT_SET) {
+ buf.putDouble(position + Byte.BYTES, valueSelector.getDouble());
+ buf.put(position, BYTE_FLAG_IS_SET);
Review comment:
nit: suggest setting the IS_SET byte first before the double value, since
that's the order they appear in the buffer
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9187: Implement ANY aggregator
jon-wei commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367208078 ## File path: docs/querying/sql.md ## @@ -203,6 +203,10 @@ Only the COUNT aggregation can accept DISTINCT. |`EARLIEST(expr, maxBytesPerString)`|Like `EARLIEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| |`LATEST(expr)`|Returns the latest non-null value of `expr`, which must be numeric. If `expr` comes from a relation with a timestamp column (like a Druid datasource) then "latest" is the value last encountered with the maximum overall timestamp of all values being aggregated. If `expr` does not come from a relation with a timestamp, then it is simply the last value encountered.| |`LATEST(expr, maxBytesPerString)`|Like `LATEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| +|`ANY_VALUE(expr)`|Returns any value of `expr`, which must be numeric. If `druid.generic.useDefaultValueForNull=true` this can returns the default value for null and does not prefer "non-null" values over the default value for null. If `druid.generic.useDefaultValueForNull=false`, then this will returns any non-null value of `expr`| Review comment: Can you also add entries for the new aggregators under `docs/querying/aggregations.md`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9187: Implement ANY aggregator
jon-wei commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367207961 ## File path: docs/querying/sql.md ## @@ -203,6 +203,10 @@ Only the COUNT aggregation can accept DISTINCT. |`EARLIEST(expr, maxBytesPerString)`|Like `EARLIEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| |`LATEST(expr)`|Returns the latest non-null value of `expr`, which must be numeric. If `expr` comes from a relation with a timestamp column (like a Druid datasource) then "latest" is the value last encountered with the maximum overall timestamp of all values being aggregated. If `expr` does not come from a relation with a timestamp, then it is simply the last value encountered.| |`LATEST(expr, maxBytesPerString)`|Like `LATEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| +|`ANY_VALUE(expr)`|Returns any value of `expr`, which must be numeric. If `druid.generic.useDefaultValueForNull=true` this can returns the default value for null and does not prefer "non-null" values over the default value for null. If `druid.generic.useDefaultValueForNull=false`, then this will returns any non-null value of `expr`| +|`ANY_VALUE(expr, maxBytesPerString)`|Like `ANY_VALUE(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| Review comment: This should mention that the default maxBytesPerString is 1024 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9187: Implement ANY aggregator
jon-wei commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367204550
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.DoubleMaxAggregator;
+import org.apache.druid.query.aggregation.DoubleMaxBufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.aggregation.SimpleDoubleAggregatorFactory;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends SimpleDoubleAggregatorFactory
+{
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName,
+ @JsonProperty("expression") @Nullable String expression,
+ @JacksonInject ExprMacroTable macroTable
+ )
+ {
+super(macroTable, name, fieldName, expression);
+ }
+
+ public DoubleAnyAggregatorFactory(String name, String fieldName)
+ {
+this(name, fieldName, null, ExprMacroTable.nil());
+ }
+
+ @Override
+ protected double nullValue()
+ {
+return Double.NaN;
+ }
+
+ @Override
+ protected Aggregator buildAggregator(BaseDoubleColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator
buildBufferAggregator(BaseDoubleColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name, null, macroTable);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName, expression, macroTable));
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8WithNullToEmpty(fieldName);
Review comment:
This method and similar could use `CacheKeyBuilder` instead
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9187: Implement ANY aggregator
jon-wei commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367207795 ## File path: docs/querying/sql.md ## @@ -203,6 +203,10 @@ Only the COUNT aggregation can accept DISTINCT. |`EARLIEST(expr, maxBytesPerString)`|Like `EARLIEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| |`LATEST(expr)`|Returns the latest non-null value of `expr`, which must be numeric. If `expr` comes from a relation with a timestamp column (like a Druid datasource) then "latest" is the value last encountered with the maximum overall timestamp of all values being aggregated. If `expr` does not come from a relation with a timestamp, then it is simply the last value encountered.| |`LATEST(expr, maxBytesPerString)`|Like `LATEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| +|`ANY_VALUE(expr)`|Returns any value of `expr`, which must be numeric. If `druid.generic.useDefaultValueForNull=true` this can returns the default value for null and does not prefer "non-null" values over the default value for null. If `druid.generic.useDefaultValueForNull=false`, then this will returns any non-null value of `expr`| Review comment: "this can returns the default" -> "this can return the default", similarly for "then this will returns" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9111: Add HashJoinSegment, a virtual segment for joins.
clintropolis commented on a change in pull request #9111: Add HashJoinSegment, a virtual segment for joins. URL: https://github.com/apache/druid/pull/9111#discussion_r367203499 ## File path: processing/src/main/java/org/apache/druid/segment/join/PossiblyNullColumnValueSelector.java ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.segment.join; + +import org.apache.druid.common.config.NullHandling; +import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector; +import org.apache.druid.segment.ColumnValueSelector; + +import javax.annotation.Nullable; +import java.util.function.BooleanSupplier; + +public class PossiblyNullColumnValueSelector implements ColumnValueSelector Review comment: this would probably be worth distinguishing functionality/intent from `BaseNullableColumnValueSelector` through javadocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9111: Add HashJoinSegment, a virtual segment for joins.
clintropolis commented on a change in pull request #9111: Add HashJoinSegment,
a virtual segment for joins.
URL: https://github.com/apache/druid/pull/9111#discussion_r367208666
##
File path:
processing/src/main/java/org/apache/druid/segment/join/table/IndexedTableColumnValueSelector.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.join.table;
+
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnValueSelector;
+
+import javax.annotation.Nullable;
+import java.util.function.IntSupplier;
+
+public class IndexedTableColumnValueSelector implements
ColumnValueSelector
+{
+ private final IntSupplier currentRow;
+ private final IndexedTable.Reader columnReader;
+
+ IndexedTableColumnValueSelector(IndexedTable table, IntSupplier currentRow,
int columnNumber)
+ {
+this.currentRow = currentRow;
+this.columnReader = table.columnReader(columnNumber);
+ }
+
+ @Override
+ public double getDouble()
+ {
+final int rowNum = currentRow.getAsInt();
+
+if (rowNum != -1) {
+ final Object value = columnReader.read(currentRow.getAsInt());
+
+ if (value instanceof Number) {
+return ((Number) value).doubleValue();
+ }
+}
+
+// Otherwise this shouldn't have been called (due to isNull returning
true).
+assert NullHandling.replaceWithDefault();
+return NullHandling.defaultDoubleValue();
Review comment:
Will this and other primitive get methods cause a null pointer exception if
it does happen to get called in production-ish environment and sql compatible
null handling is enabled because the assert will not be executed and
`NullHandling.defaultDoubleValue()` will return `null`. Should this return the
0 value instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky opened a new pull request #9195: Web console: fix refresh button in segments view
vogievetsky opened a new pull request #9195: Web console: fix refresh button in segments view URL: https://github.com/apache/druid/pull/9195 - Fix refresh button - Fix next page in no sql mode - Fix invalid query being generated if there are no segments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367232554 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/first/StringAggregatorUtils.java ## @@ -29,7 +29,7 @@ import java.nio.ByteBuffer; import java.nio.charset.StandardCharsets; -public class StringFirstLastUtils +public class StringAggregatorUtils Review comment: If you decide to end up moving `chop` to `StringUtils`, please revert this rename This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367233250 ## File path: sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java ## @@ -1297,6 +1301,46 @@ public void testLatestAggregators() throws Exception ); } + // This test the on-heap version of the AnyAggregator (Double/Float/Long/String) + @Test + public void testAnyAggregator() throws Exception Review comment: nit: It would probably be worth adding an additional test that tests numeric columns agains `druid.numfoo` table since it contains numeric columns that have null values when run in sql compatible null mode, and also tests for ordering by each 'any' aggregator. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY
aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367230483
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This Aggregator is created by the {@link DoubleAnyAggregatorFactory} which
extends from
+ * {@link NullableNumericAggregatorFactory}. If null needs to be handle, then
{@link NullableNumericAggregatorFactory}
+ * will wrap this aggregator in {@link NullableNumericAggregator} and can
handle all null in that class.
+ * Hence, no null will ever be pass into this aggregator from the
valueSelector.
+ */
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final byte BYTE_FLAG_IS_NOT_SET = 0;
+ private static final byte BYTE_FLAG_IS_SET = 1;
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.put(position, BYTE_FLAG_IS_NOT_SET);
+buf.putDouble(position + Byte.BYTES, NULL_VALUE);
Review comment:
instead of `NULL_VALUE` maybe use `NullHandling.ZERO_DOUBLE` or like just 0
since this is the only place this is used
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY
aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367231030
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This Aggregator is created by the {@link DoubleAnyAggregatorFactory} which
extends from
+ * {@link NullableNumericAggregatorFactory}. If null needs to be handle, then
{@link NullableNumericAggregatorFactory}
+ * will wrap this aggregator in {@link NullableNumericAggregator} and can
handle all null in that class.
+ * Hence, no null will ever be pass into this aggregator from the
valueSelector.
+ */
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final byte BYTE_FLAG_IS_NOT_SET = 0;
+ private static final byte BYTE_FLAG_IS_SET = 1;
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.put(position, BYTE_FLAG_IS_NOT_SET);
+buf.putDouble(position + Byte.BYTES, NULL_VALUE);
+ }
+
+ @Override
+ public void aggregate(ByteBuffer buf, int position)
+ {
+if (buf.get(position) == BYTE_FLAG_IS_NOT_SET) {
+ buf.putDouble(position + Byte.BYTES, valueSelector.getDouble());
+ buf.put(position, BYTE_FLAG_IS_SET);
Review comment:
I agree on put ordering
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9187: Implement ANY aggregator
clintropolis commented on a change in pull request #9187: Implement ANY
aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367232247
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/StringAnyAggregator.java
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.first.StringAggregatorUtils;
+import org.apache.druid.segment.BaseObjectColumnValueSelector;
+import org.apache.druid.segment.DimensionHandlerUtils;
+
+public class StringAnyAggregator implements Aggregator
+{
+ private final BaseObjectColumnValueSelector valueSelector;
+ private final int maxStringBytes;
+
+ private String foundValue;
+
+ public StringAnyAggregator(BaseObjectColumnValueSelector valueSelector, int
maxStringBytes)
+ {
+this.valueSelector = valueSelector;
+this.maxStringBytes = maxStringBytes;
+this.foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null) {
+ final Object object = valueSelector.getObject();
+ if (object != null) {
+foundValue = DimensionHandlerUtils.convertObjectToString(object);
+if (foundValue != null && foundValue.length() > maxStringBytes) {
+ foundValue = foundValue.substring(0, maxStringBytes);
+}
+ }
+}
+ }
+
+ @Override
+ public Object get()
+ {
+return StringAggregatorUtils.chop(foundValue, maxStringBytes);
Review comment:
nit: It is probably worth pushing `chop` down into `StringUtils` rather than
renaming and widening the usage of `StringAggregatorUtils`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9181: Speed up String first/last aggregators when folding isn't needed.
clintropolis commented on a change in pull request #9181: Speed up String
first/last aggregators when folding isn't needed.
URL: https://github.com/apache/druid/pull/9181#discussion_r367234568
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/first/StringFirstLastUtils.java
##
@@ -33,23 +36,63 @@
{
private static final int NULL_VALUE = -1;
+ /**
+ * Shorten "s" to "maxBytes" chars. Fast and loose because these are *chars*
not *bytes*. Use
+ * {@link #chop(String, int)} for slower, but accurate chopping.
+ */
+ @Nullable
+ public static String fastLooseChop(@Nullable final String s, final int
maxBytes)
+ {
+if (s == null || s.length() <= maxBytes) {
+ return s;
+} else {
+ return s.substring(0, maxBytes);
+}
+ }
+
+ /**
+ * Shorten "s" to what could fit in "maxBytes" bytes as UTF-8.
+ */
@Nullable
public static String chop(@Nullable final String s, final int maxBytes)
{
if (s == null) {
return null;
} else {
- // Shorten firstValue to what could fit in maxBytes as UTF-8.
final byte[] bytes = new byte[maxBytes];
final int len = StringUtils.toUtf8WithLimit(s, ByteBuffer.wrap(bytes));
return new String(bytes, 0, len, StandardCharsets.UTF_8);
}
}
+ /**
+ * Returns whether a given value selector *might* contain
SerializablePairLongString objects.
+ */
+ public static boolean selectorNeedsFoldCheck(
+ final BaseObjectColumnValueSelector valueSelector,
+ @Nullable final ColumnCapabilities valueSelectorCapabilities
+ )
+ {
+if (valueSelectorCapabilities != null &&
valueSelectorCapabilities.getType() != ValueType.COMPLEX) {
+ // Known, non-complex type.
+ return false;
+}
+
+if (valueSelector instanceof NilColumnValueSelector) {
+ // Nil column, definitely no SerializablePairLongStrings.
+ return false;
+}
+
+// Check if the reported class could possibly be
SerializablePairLongString.
+final Class clazz = valueSelector.classOfObject();
+return clazz.isAssignableFrom(SerializablePairLongString.class)
Review comment:
Apologies, I was wrong, I think this line + comment just isn't super
intuitive on first glance that it is a catch-all that any ‘unknown’ selector
types will also be true (since clazz will be Object.class in that case). Maybe
expanding on the comment would make it require less thought?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #9101: Link javaOpts to middlemanager runtime.properties docs
clintropolis merged pull request #9101: Link javaOpts to middlemanager runtime.properties docs URL: https://github.com/apache/druid/pull/9101 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (85a3d41 -> 92ac22d)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 85a3d41 Tutorials use new ingestion spec where possible (#9155) add 92ac22d Link javaOpts to middlemanager runtime.properties docs (#9101) No new revisions were added by this update. Summary of changes: docs/development/extensions-core/s3.md | 2 +- docs/operations/other-hadoop.md| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367253148
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyAggregatorFactory.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.AggregatorUtil;
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.DoubleMaxAggregator;
+import org.apache.druid.query.aggregation.DoubleMaxBufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.aggregation.SimpleDoubleAggregatorFactory;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+
+public class DoubleAnyAggregatorFactory extends SimpleDoubleAggregatorFactory
+{
+ @JsonCreator
+ public DoubleAnyAggregatorFactory(
+ @JsonProperty("name") String name,
+ @JsonProperty("fieldName") final String fieldName,
+ @JsonProperty("expression") @Nullable String expression,
+ @JacksonInject ExprMacroTable macroTable
+ )
+ {
+super(macroTable, name, fieldName, expression);
+ }
+
+ public DoubleAnyAggregatorFactory(String name, String fieldName)
+ {
+this(name, fieldName, null, ExprMacroTable.nil());
+ }
+
+ @Override
+ protected double nullValue()
+ {
+return Double.NaN;
+ }
+
+ @Override
+ protected Aggregator buildAggregator(BaseDoubleColumnValueSelector selector)
+ {
+return new DoubleAnyAggregator(selector);
+ }
+
+ @Override
+ protected BufferAggregator
buildBufferAggregator(BaseDoubleColumnValueSelector selector)
+ {
+return new DoubleAnyBufferAggregator(selector);
+ }
+
+ @Override
+ @Nullable
+ public Object combine(@Nullable Object lhs, @Nullable Object rhs)
+ {
+if (lhs != null) {
+ return lhs;
+} else {
+ return rhs;
+}
+ }
+
+ @Override
+ public AggregatorFactory getCombiningFactory()
+ {
+return new DoubleAnyAggregatorFactory(name, name, null, macroTable);
+ }
+
+ @Override
+ public List getRequiredColumns()
+ {
+return Collections.singletonList(new DoubleAnyAggregatorFactory(fieldName,
fieldName, expression, macroTable));
+ }
+
+ @Override
+ public byte[] getCacheKey()
+ {
+byte[] fieldNameBytes = StringUtils.toUtf8WithNullToEmpty(fieldName);
Review comment:
Done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367255431
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This Aggregator is created by the {@link DoubleAnyAggregatorFactory} which
extends from
+ * {@link NullableNumericAggregatorFactory}. If null needs to be handle, then
{@link NullableNumericAggregatorFactory}
+ * will wrap this aggregator in {@link NullableNumericAggregator} and can
handle all null in that class.
+ * Hence, no null will ever be pass into this aggregator from the
valueSelector.
+ */
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final byte BYTE_FLAG_IS_NOT_SET = 0;
+ private static final byte BYTE_FLAG_IS_SET = 1;
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.put(position, BYTE_FLAG_IS_NOT_SET);
+buf.putDouble(position + Byte.BYTES, NULL_VALUE);
+ }
+
+ @Override
+ public void aggregate(ByteBuffer buf, int position)
+ {
+if (buf.get(position) == BYTE_FLAG_IS_NOT_SET) {
+ buf.putDouble(position + Byte.BYTES, valueSelector.getDouble());
+ buf.put(position, BYTE_FLAG_IS_SET);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367255751 ## File path: docs/querying/sql.md ## @@ -203,6 +203,10 @@ Only the COUNT aggregation can accept DISTINCT. |`EARLIEST(expr, maxBytesPerString)`|Like `EARLIEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| |`LATEST(expr)`|Returns the latest non-null value of `expr`, which must be numeric. If `expr` comes from a relation with a timestamp column (like a Druid datasource) then "latest" is the value last encountered with the maximum overall timestamp of all values being aggregated. If `expr` does not come from a relation with a timestamp, then it is simply the last value encountered.| |`LATEST(expr, maxBytesPerString)`|Like `LATEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| +|`ANY_VALUE(expr)`|Returns any value of `expr`, which must be numeric. If `druid.generic.useDefaultValueForNull=true` this can returns the default value for null and does not prefer "non-null" values over the default value for null. If `druid.generic.useDefaultValueForNull=false`, then this will returns any non-null value of `expr`| Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367255974 ## File path: docs/querying/sql.md ## @@ -203,6 +203,10 @@ Only the COUNT aggregation can accept DISTINCT. |`EARLIEST(expr, maxBytesPerString)`|Like `EARLIEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| |`LATEST(expr)`|Returns the latest non-null value of `expr`, which must be numeric. If `expr` comes from a relation with a timestamp column (like a Druid datasource) then "latest" is the value last encountered with the maximum overall timestamp of all values being aggregated. If `expr` does not come from a relation with a timestamp, then it is simply the last value encountered.| |`LATEST(expr, maxBytesPerString)`|Like `LATEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| +|`ANY_VALUE(expr)`|Returns any value of `expr`, which must be numeric. If `druid.generic.useDefaultValueForNull=true` this can returns the default value for null and does not prefer "non-null" values over the default value for null. If `druid.generic.useDefaultValueForNull=false`, then this will returns any non-null value of `expr`| +|`ANY_VALUE(expr, maxBytesPerString)`|Like `ANY_VALUE(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| Review comment: What do you mean default? You need to always pass the value. There is no default value for maxBytesPerString This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367257549 ## File path: docs/querying/sql.md ## @@ -203,6 +203,10 @@ Only the COUNT aggregation can accept DISTINCT. |`EARLIEST(expr, maxBytesPerString)`|Like `EARLIEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| |`LATEST(expr)`|Returns the latest non-null value of `expr`, which must be numeric. If `expr` comes from a relation with a timestamp column (like a Druid datasource) then "latest" is the value last encountered with the maximum overall timestamp of all values being aggregated. If `expr` does not come from a relation with a timestamp, then it is simply the last value encountered.| |`LATEST(expr, maxBytesPerString)`|Like `LATEST(expr)`, but for strings. The `maxBytesPerString` parameter determines how much aggregation space to allocate per string. Strings longer than this limit will be truncated. This parameter should be set as low as possible, since high values will lead to wasted memory.| +|`ANY_VALUE(expr)`|Returns any value of `expr`, which must be numeric. If `druid.generic.useDefaultValueForNull=true` this can returns the default value for null and does not prefer "non-null" values over the default value for null. If `druid.generic.useDefaultValueForNull=false`, then this will returns any non-null value of `expr`| Review comment: Done. Btw I saw filterNullValues for stringLast and stringFirst. Is that still true? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367260857
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/DoubleAnyBufferAggregator.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.BufferAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregator;
+import org.apache.druid.query.aggregation.NullableNumericAggregatorFactory;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.BaseDoubleColumnValueSelector;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This Aggregator is created by the {@link DoubleAnyAggregatorFactory} which
extends from
+ * {@link NullableNumericAggregatorFactory}. If null needs to be handle, then
{@link NullableNumericAggregatorFactory}
+ * will wrap this aggregator in {@link NullableNumericAggregator} and can
handle all null in that class.
+ * Hence, no null will ever be pass into this aggregator from the
valueSelector.
+ */
+public class DoubleAnyBufferAggregator implements BufferAggregator
+{
+ private static final byte BYTE_FLAG_IS_NOT_SET = 0;
+ private static final byte BYTE_FLAG_IS_SET = 1;
+ private static final double NULL_VALUE = 0;
+ private final BaseDoubleColumnValueSelector valueSelector;
+
+ public DoubleAnyBufferAggregator(BaseDoubleColumnValueSelector valueSelector)
+ {
+this.valueSelector = valueSelector;
+ }
+
+ @Override
+ public void init(ByteBuffer buf, int position)
+ {
+buf.put(position, BYTE_FLAG_IS_NOT_SET);
+buf.putDouble(position + Byte.BYTES, NULL_VALUE);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367263109 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/first/StringAggregatorUtils.java ## @@ -29,7 +29,7 @@ import java.nio.ByteBuffer; import java.nio.charset.StandardCharsets; -public class StringFirstLastUtils +public class StringAggregatorUtils Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
URL: https://github.com/apache/druid/pull/9187#discussion_r367263080
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/any/StringAnyAggregator.java
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.any;
+
+import org.apache.druid.query.aggregation.Aggregator;
+import org.apache.druid.query.aggregation.first.StringAggregatorUtils;
+import org.apache.druid.segment.BaseObjectColumnValueSelector;
+import org.apache.druid.segment.DimensionHandlerUtils;
+
+public class StringAnyAggregator implements Aggregator
+{
+ private final BaseObjectColumnValueSelector valueSelector;
+ private final int maxStringBytes;
+
+ private String foundValue;
+
+ public StringAnyAggregator(BaseObjectColumnValueSelector valueSelector, int
maxStringBytes)
+ {
+this.valueSelector = valueSelector;
+this.maxStringBytes = maxStringBytes;
+this.foundValue = null;
+ }
+
+ @Override
+ public void aggregate()
+ {
+if (foundValue == null) {
+ final Object object = valueSelector.getObject();
+ if (object != null) {
+foundValue = DimensionHandlerUtils.convertObjectToString(object);
+if (foundValue != null && foundValue.length() > maxStringBytes) {
+ foundValue = foundValue.substring(0, maxStringBytes);
+}
+ }
+}
+ }
+
+ @Override
+ public Object get()
+ {
+return StringAggregatorUtils.chop(foundValue, maxStringBytes);
Review comment:
sounds good to me. Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on a change in pull request #9187: Implement ANY aggregator
maytasm3 commented on a change in pull request #9187: Implement ANY aggregator URL: https://github.com/apache/druid/pull/9187#discussion_r367263397 ## File path: sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java ## @@ -1297,6 +1301,46 @@ public void testLatestAggregators() throws Exception ); } + // This test the on-heap version of the AnyAggregator (Double/Float/Long/String) + @Test + public void testAnyAggregator() throws Exception Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
