[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981316#comment-15981316
]
ASF GitHub Bot commented on RYA-260:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-rya/pull/156
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979259#comment-15979259
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112761421
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979251#comment-15979251
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112760188
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979246#comment-15979246
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112738370
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979217#comment-15979217
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on the issue:
https://github.com/apache/incubator-rya/pull/156
I've addressed everything.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979216#comment-15979216
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112754163
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloDeletePCJ.java
---
@@ -118,14 +116,14 @@ private void stopUpdatingPCJ(final String
fluoAppName, final String pcjId) {
// Connect to the Fluo application that is updating this
instance's PCJs.
final AccumuloConnectionDetails cd =
super.getAccumuloConnectionDetails();
-final FluoClient fluoClient = new FluoClientFactory().connect(
+try(final FluoClient fluoClient = new FluoClientFactory().connect(
cd.getUsername(),
new String(cd.getPassword()),
cd.getInstanceName(),
cd.getZookeepers(),
-fluoAppName);
-
-// Delete the PCJ from the Fluo App.
-new DeletePcj(1000).deletePcj(fluoClient, pcjId);
--- End diff --
Done.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979215#comment-15979215
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112754079
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.api/src/main/java/org/apache/rya/indexing/pcj/fluo/api/CreatePcj.java
---
@@ -82,6 +82,7 @@
*/
@DefaultAnnotation(NonNull.class)
public class CreatePcj {
+private static final Logger log = Logger.getLogger(CreatePcj.class);
--- End diff --
Ok, I'm not going to change anything in this review then.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979110#comment-15979110
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112740494
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/AccumuloPcjSerializer.java
---
@@ -112,24 +109,6 @@ public BindingSet convert(byte[] bindingSetBytes,
VariableOrder varOrder) throws
}
}
-/**
- * Checks to see if the names of all the {@link Binding}s in the
{@link BindingSet}
- * are a subset of the variables names in {@link VariableOrder}.
- *
- * @param bindingSet - The binding set whose Bindings will be
inspected. (not null)
- * @param varOrder - The names of the bindings that may appear in the
BindingSet. (not null)
- * @throws IllegalArgumentException Indicates the names of the
bindings are
- * not a subset of the variable order.
- */
-private static void checkBindingsSubsetOfVarOrder(BindingSet
bindingSet, VariableOrder varOrder) throws IllegalArgumentException {
-checkNotNull(bindingSet);
-checkNotNull(varOrder);
-
-Set bindingNames = bindingSet.getBindingNames();
-List varNames = varOrder.getVariableOrders();
-checkArgument(varNames.containsAll(bindingNames), "The BindingSet
contains a Binding whose name is not part of the VariableOrder.");
-}
-
--- End diff --
Done.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979109#comment-15979109
]
ASF GitHub Bot commented on RYA-260:
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112740422
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.api/src/main/java/org/apache/rya/indexing/pcj/fluo/api/CreatePcj.java
---
@@ -82,6 +82,7 @@
*/
@DefaultAnnotation(NonNull.class)
public class CreatePcj {
+private static final Logger log = Logger.getLogger(CreatePcj.class);
--- End diff --
not sure, but we need to pick one and stick to it.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979107#comment-15979107
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112739832
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/AccumuloPcjSerializer.java
---
@@ -112,24 +109,6 @@ public BindingSet convert(byte[] bindingSetBytes,
VariableOrder varOrder) throws
}
}
-/**
- * Checks to see if the names of all the {@link Binding}s in the
{@link BindingSet}
- * are a subset of the variables names in {@link VariableOrder}.
- *
- * @param bindingSet - The binding set whose Bindings will be
inspected. (not null)
- * @param varOrder - The names of the bindings that may appear in the
BindingSet. (not null)
- * @throws IllegalArgumentException Indicates the names of the
bindings are
- * not a subset of the variable order.
- */
-private static void checkBindingsSubsetOfVarOrder(BindingSet
bindingSet, VariableOrder varOrder) throws IllegalArgumentException {
-checkNotNull(bindingSet);
-checkNotNull(varOrder);
-
-Set bindingNames = bindingSet.getBindingNames();
-List varNames = varOrder.getVariableOrders();
-checkArgument(varNames.containsAll(bindingNames), "The BindingSet
contains a Binding whose name is not part of the VariableOrder.");
-}
-
--- End diff --
Actually, never mind. It's isn't needed anymore. You're allowed to omit a
binding for a variable that is in the variable order because we need that for
optional support.
So I'm going to remove this check and just add another unit test.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979093#comment-15979093
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112737434
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/BindingSetStringConverter.java
---
@@ -58,7 +56,13 @@
@Override
public String convert(final BindingSet bindingSet, final VariableOrder
varOrder) {
-checkBindingsSubsetOfVarOrder(bindingSet, varOrder);
+requireNonNull(bindingSet);
+requireNonNull(varOrder);
+
+// If the binding set is empty, just return empty string.
+if(bindingSet.getBindingNames().isEmpty()) {
+return "";
+}
--- End diff --
I don't think I need this change anymore, so I'm removing it.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979065#comment-15979065
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112732124
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/AccumuloPcjSerializer.java
---
@@ -112,24 +109,6 @@ public BindingSet convert(byte[] bindingSetBytes,
VariableOrder varOrder) throws
}
}
-/**
- * Checks to see if the names of all the {@link Binding}s in the
{@link BindingSet}
- * are a subset of the variables names in {@link VariableOrder}.
- *
- * @param bindingSet - The binding set whose Bindings will be
inspected. (not null)
- * @param varOrder - The names of the bindings that may appear in the
BindingSet. (not null)
- * @throws IllegalArgumentException Indicates the names of the
bindings are
- * not a subset of the variable order.
- */
-private static void checkBindingsSubsetOfVarOrder(BindingSet
bindingSet, VariableOrder varOrder) throws IllegalArgumentException {
-checkNotNull(bindingSet);
-checkNotNull(varOrder);
-
-Set bindingNames = bindingSet.getBindingNames();
-List varNames = varOrder.getVariableOrders();
-checkArgument(varNames.containsAll(bindingNames), "The BindingSet
contains a Binding whose name is not part of the VariableOrder.");
-}
-
--- End diff --
Though I think the logic of this method is actually backwards. It should be
checking to ensure the bindingNames contain all of the var names, not the other
way around. I could put this method back and fix the logic.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979058#comment-15979058
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112731946
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/AccumuloPcjSerializer.java
---
@@ -112,24 +109,6 @@ public BindingSet convert(byte[] bindingSetBytes,
VariableOrder varOrder) throws
}
}
-/**
- * Checks to see if the names of all the {@link Binding}s in the
{@link BindingSet}
- * are a subset of the variables names in {@link VariableOrder}.
- *
- * @param bindingSet - The binding set whose Bindings will be
inspected. (not null)
- * @param varOrder - The names of the bindings that may appear in the
BindingSet. (not null)
- * @throws IllegalArgumentException Indicates the names of the
bindings are
- * not a subset of the variable order.
- */
-private static void checkBindingsSubsetOfVarOrder(BindingSet
bindingSet, VariableOrder varOrder) throws IllegalArgumentException {
-checkNotNull(bindingSet);
-checkNotNull(varOrder);
-
-Set bindingNames = bindingSet.getBindingNames();
-List varNames = varOrder.getVariableOrders();
-checkArgument(varNames.containsAll(bindingNames), "The BindingSet
contains a Binding whose name is not part of the VariableOrder.");
-}
-
--- End diff --
I don't think it's needed. If a BindingSet has some names that aren't part
of the VariableOrder, we can just omit those binidngs from the string that is
created by the serializer.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979044#comment-15979044
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112729828
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.api/src/main/java/org/apache/rya/indexing/pcj/fluo/api/CreatePcj.java
---
@@ -82,6 +82,7 @@
*/
@DefaultAnnotation(NonNull.class)
public class CreatePcj {
+private static final Logger log = Logger.getLogger(CreatePcj.class);
--- End diff --
Which logging framework are we using? I can update my code at the very
least to use that. Is it SLF4J?
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979045#comment-15979045
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112729952
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/VisibilityBindingSetSerde.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+
+import org.apache.fluo.api.data.Bytes;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Serializes and deserializes a {@link VisibilityBindingSet} to and from
{@link Bytes} objects.
+ */
+@DefaultAnnotation(NonNull.class)
+public class VisibilityBindingSetSerde {
--- End diff --
I stole the naming convention from Apache Hive.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979036#comment-15979036
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112729142
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloDeletePCJ.java
---
@@ -118,14 +116,14 @@ private void stopUpdatingPCJ(final String
fluoAppName, final String pcjId) {
// Connect to the Fluo application that is updating this
instance's PCJs.
final AccumuloConnectionDetails cd =
super.getAccumuloConnectionDetails();
-final FluoClient fluoClient = new FluoClientFactory().connect(
+try(final FluoClient fluoClient = new FluoClientFactory().connect(
cd.getUsername(),
new String(cd.getPassword()),
cd.getInstanceName(),
cd.getZookeepers(),
-fluoAppName);
-
-// Delete the PCJ from the Fluo App.
-new DeletePcj(1000).deletePcj(fluoClient, pcjId);
--- End diff --
I'll go do that.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979031#comment-15979031
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112728756
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/QueryResultUpdater.java
---
@@ -53,32 +54,46 @@
* @param tx - The transaction all Fluo queries will use. (not null)
* @param childBindingSet - A binding set that the query's child node
has emmitted. (not null)
* @param queryMetadata - The metadata of the Query whose results will
be updated. (not null)
+ * @throws Exception A problem caused the update to fail.
*/
public void updateQueryResults(
final TransactionBase tx,
final VisibilityBindingSet childBindingSet,
-final QueryMetadata queryMetadata) {
+final QueryMetadata queryMetadata) throws Exception {
checkNotNull(tx);
checkNotNull(childBindingSet);
checkNotNull(queryMetadata);
+log.trace(
+"Transaction ID: " + tx.getStartTimestamp() + "\n" +
+"Join Node ID: " + queryMetadata.getNodeId() + "\n" +
+"Child Node ID: " + queryMetadata.getChildNodeId() + "\n" +
+"Child Binding Set:\n" + childBindingSet + "\n");
+
// Create the query's Binding Set from the child node's binding
set.
final VariableOrder queryVarOrder =
queryMetadata.getVariableOrder();
+final BindingSet queryBindingSet =
BindingSetUtil.keepBindings(queryVarOrder, childBindingSet);
-final MapBindingSet queryBindingSet = new MapBindingSet();
-for(final String bindingName : queryVarOrder) {
-if(childBindingSet.hasBinding(bindingName)) {
-final Binding binding =
childBindingSet.getBinding(bindingName);
-queryBindingSet.addBinding(binding);
-}
+// Create the Row Key for the result. If the child node groups
results, then the key must only contain the Group By variables.
+final Bytes resultRow;
+
+final String childNodeId = queryMetadata.getChildNodeId();
+final boolean isGrouped = childNodeId.startsWith(
IncrementalUpdateConstants.AGGREGATION_PREFIX );
+if(isGrouped) {
--- End diff --
Ok, I don't know if this is related, but here was the initial motivation.
We're not going to write on top of the existing query result every time if we
don't use the group by arguments as the Row Key for the result. So if we're
doing the Max function and the first binding set that is written isn't always
the max, then you'll have at least two binding sets. We are not guaranteed that
the order of the binding set notification will match the order that the binding
sets were written, so it's possible for the older result to be exported after
the newer result. Which is wrong.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979006#comment-15979006
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112726456
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/QueryResultUpdater.java
---
@@ -53,32 +54,46 @@
* @param tx - The transaction all Fluo queries will use. (not null)
* @param childBindingSet - A binding set that the query's child node
has emmitted. (not null)
* @param queryMetadata - The metadata of the Query whose results will
be updated. (not null)
+ * @throws Exception A problem caused the update to fail.
*/
public void updateQueryResults(
final TransactionBase tx,
final VisibilityBindingSet childBindingSet,
-final QueryMetadata queryMetadata) {
+final QueryMetadata queryMetadata) throws Exception {
checkNotNull(tx);
checkNotNull(childBindingSet);
checkNotNull(queryMetadata);
+log.trace(
+"Transaction ID: " + tx.getStartTimestamp() + "\n" +
+"Join Node ID: " + queryMetadata.getNodeId() + "\n" +
+"Child Node ID: " + queryMetadata.getChildNodeId() + "\n" +
+"Child Binding Set:\n" + childBindingSet + "\n");
+
// Create the query's Binding Set from the child node's binding
set.
final VariableOrder queryVarOrder =
queryMetadata.getVariableOrder();
+final BindingSet queryBindingSet =
BindingSetUtil.keepBindings(queryVarOrder, childBindingSet);
-final MapBindingSet queryBindingSet = new MapBindingSet();
-for(final String bindingName : queryVarOrder) {
-if(childBindingSet.hasBinding(bindingName)) {
-final Binding binding =
childBindingSet.getBinding(bindingName);
-queryBindingSet.addBinding(binding);
-}
+// Create the Row Key for the result. If the child node groups
results, then the key must only contain the Group By variables.
+final Bytes resultRow;
+
+final String childNodeId = queryMetadata.getChildNodeId();
+final boolean isGrouped = childNodeId.startsWith(
IncrementalUpdateConstants.AGGREGATION_PREFIX );
+if(isGrouped) {
--- End diff --
If I remove that code, all of my aggregation integration tests fail.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978997#comment-15978997
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112725587
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/QueryResultUpdater.java
---
@@ -53,32 +54,46 @@
* @param tx - The transaction all Fluo queries will use. (not null)
* @param childBindingSet - A binding set that the query's child node
has emmitted. (not null)
* @param queryMetadata - The metadata of the Query whose results will
be updated. (not null)
+ * @throws Exception A problem caused the update to fail.
*/
public void updateQueryResults(
final TransactionBase tx,
final VisibilityBindingSet childBindingSet,
-final QueryMetadata queryMetadata) {
+final QueryMetadata queryMetadata) throws Exception {
checkNotNull(tx);
checkNotNull(childBindingSet);
checkNotNull(queryMetadata);
+log.trace(
+"Transaction ID: " + tx.getStartTimestamp() + "\n" +
+"Join Node ID: " + queryMetadata.getNodeId() + "\n" +
+"Child Node ID: " + queryMetadata.getChildNodeId() + "\n" +
+"Child Binding Set:\n" + childBindingSet + "\n");
+
// Create the query's Binding Set from the child node's binding
set.
final VariableOrder queryVarOrder =
queryMetadata.getVariableOrder();
+final BindingSet queryBindingSet =
BindingSetUtil.keepBindings(queryVarOrder, childBindingSet);
-final MapBindingSet queryBindingSet = new MapBindingSet();
-for(final String bindingName : queryVarOrder) {
-if(childBindingSet.hasBinding(bindingName)) {
-final Binding binding =
childBindingSet.getBinding(bindingName);
-queryBindingSet.addBinding(binding);
-}
+// Create the Row Key for the result. If the child node groups
results, then the key must only contain the Group By variables.
+final Bytes resultRow;
+
+final String childNodeId = queryMetadata.getChildNodeId();
+final boolean isGrouped = childNodeId.startsWith(
IncrementalUpdateConstants.AGGREGATION_PREFIX );
+if(isGrouped) {
--- End diff --
I'm actually not sure. Let me check that.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978983#comment-15978983
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112723126
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978981#comment-15978981
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112722927
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
--- End diff --
Done.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978978#comment-15978978
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112722449
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978975#comment-15978975
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112722051
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/VisibilityBindingSetSerde.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+
+import org.apache.fluo.api.data.Bytes;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Serializes and deserializes a {@link VisibilityBindingSet} to and from
{@link Bytes} objects.
+ */
+@DefaultAnnotation(NonNull.class)
+public class VisibilityBindingSetSerde {
+
+/**
+ * Serializes a {@link VisibilityBindingSet} into a {@link Bytes}
object.
+ *
+ * @param bindingSet - The binding set that will be serialized. (not
null)
+ * @return The serialized object.
+ * @throws Exception A problem was encountered while serializing the
object.
+ */
+public Bytes serialize(final VisibilityBindingSet bindingSet) throws
Exception {
+requireNonNull(bindingSet);
+
--- End diff --
Yea. There's no reason to write our own serialization for the Value
sections, which just hold bytes.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978965#comment-15978965
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112720900
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/PcjTables.java
---
@@ -231,9 +240,10 @@ public PcjMetadata getPcjMetadata(
checkNotNull(accumuloConn);
checkNotNull(pcjTableName);
+Scanner scanner = null;
try {
--- End diff --
Apparently that interface doesn't extends AutoCloseable.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978958#comment-15978958
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112719552
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/utils/VisibilitySimplifier.java
---
@@ -20,26 +20,48 @@
import static java.util.Objects.requireNonNull;
-import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
-import edu.umd.cs.findbugs.annotations.NonNull;
-
import org.apache.accumulo.core.security.ColumnVisibility;
import com.google.common.base.Charsets;
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
/**
* Simplifies Accumulo visibility expressions.
*/
@DefaultAnnotation(NonNull.class)
public class VisibilitySimplifier {
/**
+ * Unions two visibility equations and then simplifies the result.
+ *
+ * @param vis1 - The first visibility equation that will be unioned.
(not null)
+ * @param vis2 - The other visibility equation that will be unioned.
(not null)
+ * @return A simplified form of the unioned visibility equations.
+ */
+public static String unionAndSimplify(final String vis1, final String
vis2) {
+requireNonNull(vis1);
+requireNonNull(vis2);
+
+if(vis1.isEmpty()) {
+return vis2;
+}
+
+if(vis2.isEmpty()) {
+return vis1;
+}
+
+return simplify("(" + vis1 + ")&(" + vis2 + ")");
+}
+
--- End diff --
The whitespace is as I intended, so I'm going to leave it as is.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978955#comment-15978955
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112719303
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloCreatePCJ.java
---
@@ -139,16 +138,15 @@ private void updateFluoApp(final String ryaInstance,
final String fluoAppName, f
// Connect to the Fluo application that is updating this
instance's PCJs.
final AccumuloConnectionDetails cd =
super.getAccumuloConnectionDetails();
-final FluoClient fluoClient = new FluoClientFactory().connect(
+try(final FluoClient fluoClient = new FluoClientFactory().connect(
cd.getUsername(),
new String(cd.getPassword()),
cd.getInstanceName(),
cd.getZookeepers(),
-fluoAppName);
-
-// Initialize the PCJ within the Fluo application.
-final org.apache.rya.indexing.pcj.fluo.api.CreatePcj fluoCreatePcj
= new org.apache.rya.indexing.pcj.fluo.api.CreatePcj();
-fluoCreatePcj.withRyaIntegration(pcjId, pcjStorage, fluoClient,
getConnector(), ryaInstance);
+fluoAppName);) {
+// Initialize the PCJ within the Fluo application.
+final org.apache.rya.indexing.pcj.fluo.api.CreatePcj
fluoCreatePcj = new org.apache.rya.indexing.pcj.fluo.api.CreatePcj();
--- End diff --
No. The class name CreatePcj is used twice in the project. This class
implements one, which it imports, so this one can't be.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978950#comment-15978950
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112718946
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/instance/AccumuloRyaInstanceDetailsRepository.java
---
@@ -89,12 +89,17 @@ public AccumuloRyaInstanceDetailsRepository(final
Connector connector, final Str
@Override
public boolean isInitialized() throws RyaDetailsRepositoryException {
+Scanner scanner = null;
try {
--- End diff --
Actually, never mind. The Accumulo Scanner interface doesn't implement
AutoCloseable.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977550#comment-15977550
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112537048
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
--- End diff --
make both SerDe or Serde
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
>
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977553#comment-15977553
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112535097
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/PcjTables.java
---
@@ -231,9 +240,10 @@ public PcjMetadata getPcjMetadata(
checkNotNull(accumuloConn);
checkNotNull(pcjTableName);
+Scanner scanner = null;
try {
--- End diff --
Try with resources?
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977549#comment-15977549
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112533438
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/AccumuloPcjSerializer.java
---
@@ -112,24 +109,6 @@ public BindingSet convert(byte[] bindingSetBytes,
VariableOrder varOrder) throws
}
}
-/**
- * Checks to see if the names of all the {@link Binding}s in the
{@link BindingSet}
- * are a subset of the variables names in {@link VariableOrder}.
- *
- * @param bindingSet - The binding set whose Bindings will be
inspected. (not null)
- * @param varOrder - The names of the bindings that may appear in the
BindingSet. (not null)
- * @throws IllegalArgumentException Indicates the names of the
bindings are
- * not a subset of the variable order.
- */
-private static void checkBindingsSubsetOfVarOrder(BindingSet
bindingSet, VariableOrder varOrder) throws IllegalArgumentException {
-checkNotNull(bindingSet);
-checkNotNull(varOrder);
-
-Set bindingNames = bindingSet.getBindingNames();
-List varNames = varOrder.getVariableOrders();
-checkArgument(varNames.containsAll(bindingNames), "The BindingSet
contains a Binding whose name is not part of the VariableOrder.");
-}
-
--- End diff --
Why do away with this check?
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977543#comment-15977543
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112541163
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/VisibilityBindingSetSerde.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+
+import org.apache.fluo.api.data.Bytes;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Serializes and deserializes a {@link VisibilityBindingSet} to and from
{@link Bytes} objects.
+ */
+@DefaultAnnotation(NonNull.class)
+public class VisibilityBindingSetSerde {
+
+/**
+ * Serializes a {@link VisibilityBindingSet} into a {@link Bytes}
object.
+ *
+ * @param bindingSet - The binding set that will be serialized. (not
null)
+ * @return The serialized object.
+ * @throws Exception A problem was encountered while serializing the
object.
+ */
+public Bytes serialize(final VisibilityBindingSet bindingSet) throws
Exception {
+requireNonNull(bindingSet);
+
--- End diff --
Was your motivation for this class to cut down on the amount of String
manipulation in the app? I guess there is really no need to do String
serialization with a VariableOrder if we are creating something this is not
ordered.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977546#comment-15977546
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112555177
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977542#comment-15977542
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112536064
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.api/src/main/java/org/apache/rya/indexing/pcj/fluo/api/CreatePcj.java
---
@@ -82,6 +82,7 @@
*/
@DefaultAnnotation(NonNull.class)
public class CreatePcj {
+private static final Logger log = Logger.getLogger(CreatePcj.class);
--- End diff --
I second this.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977547#comment-15977547
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112531966
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/utils/VisibilitySimplifier.java
---
@@ -20,26 +20,48 @@
import static java.util.Objects.requireNonNull;
-import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
-import edu.umd.cs.findbugs.annotations.NonNull;
-
import org.apache.accumulo.core.security.ColumnVisibility;
import com.google.common.base.Charsets;
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
/**
* Simplifies Accumulo visibility expressions.
*/
@DefaultAnnotation(NonNull.class)
public class VisibilitySimplifier {
/**
+ * Unions two visibility equations and then simplifies the result.
+ *
+ * @param vis1 - The first visibility equation that will be unioned.
(not null)
+ * @param vis2 - The other visibility equation that will be unioned.
(not null)
+ * @return A simplified form of the unioned visibility equations.
+ */
+public static String unionAndSimplify(final String vis1, final String
vis2) {
+requireNonNull(vis1);
+requireNonNull(vis2);
+
+if(vis1.isEmpty()) {
+return vis2;
+}
+
+if(vis2.isEmpty()) {
+return vis1;
+}
+
+return simplify("(" + vis1 + ")&(" + vis2 + ")");
+}
+
--- End diff --
Nit. Seems like there's a lot of white space in this method.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977551#comment-15977551
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112551390
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloDeletePCJ.java
---
@@ -118,14 +116,14 @@ private void stopUpdatingPCJ(final String
fluoAppName, final String pcjId) {
// Connect to the Fluo application that is updating this
instance's PCJs.
final AccumuloConnectionDetails cd =
super.getAccumuloConnectionDetails();
-final FluoClient fluoClient = new FluoClientFactory().connect(
+try(final FluoClient fluoClient = new FluoClientFactory().connect(
cd.getUsername(),
new String(cd.getPassword()),
cd.getInstanceName(),
cd.getZookeepers(),
-fluoAppName);
-
-// Delete the PCJ from the Fluo App.
-new DeletePcj(1000).deletePcj(fluoClient, pcjId);
--- End diff --
It appears that you did not update DeletePcj to delete your aggregation
results/metadata.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977541#comment-15977541
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112560160
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/QueryResultUpdater.java
---
@@ -53,32 +54,46 @@
* @param tx - The transaction all Fluo queries will use. (not null)
* @param childBindingSet - A binding set that the query's child node
has emmitted. (not null)
* @param queryMetadata - The metadata of the Query whose results will
be updated. (not null)
+ * @throws Exception A problem caused the update to fail.
*/
public void updateQueryResults(
final TransactionBase tx,
final VisibilityBindingSet childBindingSet,
-final QueryMetadata queryMetadata) {
+final QueryMetadata queryMetadata) throws Exception {
checkNotNull(tx);
checkNotNull(childBindingSet);
checkNotNull(queryMetadata);
+log.trace(
+"Transaction ID: " + tx.getStartTimestamp() + "\n" +
+"Join Node ID: " + queryMetadata.getNodeId() + "\n" +
+"Child Node ID: " + queryMetadata.getChildNodeId() + "\n" +
+"Child Binding Set:\n" + childBindingSet + "\n");
+
// Create the query's Binding Set from the child node's binding
set.
final VariableOrder queryVarOrder =
queryMetadata.getVariableOrder();
+final BindingSet queryBindingSet =
BindingSetUtil.keepBindings(queryVarOrder, childBindingSet);
-final MapBindingSet queryBindingSet = new MapBindingSet();
-for(final String bindingName : queryVarOrder) {
-if(childBindingSet.hasBinding(bindingName)) {
-final Binding binding =
childBindingSet.getBinding(bindingName);
-queryBindingSet.addBinding(binding);
-}
+// Create the Row Key for the result. If the child node groups
results, then the key must only contain the Group By variables.
+final Bytes resultRow;
+
+final String childNodeId = queryMetadata.getChildNodeId();
+final boolean isGrouped = childNodeId.startsWith(
IncrementalUpdateConstants.AGGREGATION_PREFIX );
+if(isGrouped) {
--- End diff --
Is this necessary? Can the names of the Projection variables actually
differ from the variables that appear in the group by clause? I thought that
the query parser prevented the projection variables from being different than
the group by/ aggregation variables.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977545#comment-15977545
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112538109
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/AggregationResultUpdater.java
---
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.utils.VisibilitySimplifier;
+import org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationElement;
+import
org.apache.rya.indexing.pcj.fluo.app.query.AggregationMetadata.AggregationType;
+import org.apache.rya.indexing.pcj.fluo.app.query.FluoQueryColumns;
+import org.apache.rya.indexing.pcj.fluo.app.util.RowKeyUtil;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Value;
+import org.openrdf.model.datatypes.XMLDatatypeUtil;
+import org.openrdf.model.impl.DecimalLiteralImpl;
+import org.openrdf.model.impl.IntegerLiteralImpl;
+import org.openrdf.query.algebra.MathExpr.MathOp;
+import org.openrdf.query.algebra.evaluation.ValueExprEvaluationException;
+import org.openrdf.query.algebra.evaluation.util.MathUtil;
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Updates the results of an Aggregate node when its child has added a new
Binding Set to its results.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AggregationResultUpdater {
+private static final Logger log =
Logger.getLogger(AggregationResultUpdater.class);
+
+private static final AggregationStateSerDe AGG_STATE_SERDE = new
ObjectSerializationAggregationStateSerDe();
+private static final VisibilityBindingSetSerde BS_SERDE = new
VisibilityBindingSetSerde();
+
+private static final ImmutableMap FUNCTIONS;
+static {
+final ImmutableMap.Builder
builder = ImmutableMap.builder();
+builder.put(AggregationType.COUNT, new CountFunction());
+builder.put(AggregationType.SUM, new SumFunction());
+builder.put(AggregationType.AVERAGE, new AverageFunction());
+builder.put(AggregationType.MIN, new MinFunction());
+builder.put(AggregationType.MAX, new MaxFunction());
+FUNCTIONS = builder.build();
+}
+
+/**
+ * Updates the results of an Aggregation node where its child has
emitted a new Binding Set.
+ *
+ * @param tx - The transaction all Fluo queries will use. (not null)
+ * @oaram childRow - The Row Key of the child Binding Set that
changed. (not null)
+
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977544#comment-15977544
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112531584
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/instance/AccumuloRyaInstanceDetailsRepository.java
---
@@ -89,12 +89,17 @@ public AccumuloRyaInstanceDetailsRepository(final
Connector connector, final Str
@Override
public boolean isInitialized() throws RyaDetailsRepositoryException {
+Scanner scanner = null;
try {
--- End diff --
That would certainly be cleaner.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977552#comment-15977552
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112546328
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/observers/TripleObserver.java
---
@@ -62,85 +66,113 @@ public ObservedColumn getObservedColumn() {
@Override
public void process(final TransactionBase tx, final Bytes brow, final
Column column) {
-//get string representation of triple
-String row = brow.toString();
-final String triple = IncUpdateDAO.getTripleString(brow);
-String visibility = tx.gets(row, FluoQueryColumns.TRIPLES, "");
-
-//get variable metadata for all SP in table
-RowScanner rscanner =
tx.scanner().over(Span.prefix(SP_PREFIX)).fetch(FluoQueryColumns.STATEMENT_PATTERN_VARIABLE_ORDER).byRow().build();
-
+// Get string representation of triple.
+final RyaStatement ryaStatement =
IncUpdateDAO.deserializeTriple(brow);
+log.trace(
+"Transaction ID: " + tx.getStartTimestamp() + "\n" +
+"Rya Statement: " + ryaStatement + "\n");
-//see if triple matches conditions of any of the SP
+final String triple = IncUpdateDAO.getTripleString(ryaStatement);
-for (ColumnScanner colScanner : rscanner) {
+// Iterate over each of the Statement Patterns that are being
matched against.
+final RowScanner spScanner = tx.scanner()
+.over(Span.prefix(SP_PREFIX))
+
+// Only fetch rows that have the pattern in them. There
will only be a single row with a pattern per SP.
+.fetch(FluoQueryColumns.STATEMENT_PATTERN_PATTERN)
+.byRow()
+.build();
+
+//see if triple matches conditions of any of the SP
+for (final ColumnScanner colScanner : spScanner) {
+// Get the Statement Pattern's node id.
final String spID = colScanner.getsRow();
-final StatementPatternMetadata spMetadata =
QUERY_DAO.readStatementPatternMetadata(tx, spID);
+// Fetch its metadata.
+final StatementPatternMetadata spMetadata =
QUERY_METADATA_DAO.readStatementPatternMetadata(tx, spID);
+
+// Attempt to match the triple against the pattern.
final String pattern = spMetadata.getStatementPattern();
-
-for (ColumnValue cv : colScanner) {
-final String varOrders = cv.getsValue();
-final VariableOrder varOrder = new
VariableOrder(varOrders);
-final String bindingSetString = getBindingSet(triple,
pattern, varOrders);
-
-//Statement matches to a binding set
-if(bindingSetString.length() != 0) {
-final VisibilityBindingSet bindingSet = new
VisibilityBindingSet(
-CONVERTER.convert(bindingSetString, varOrder),
-visibility);
-final String valueString =
CONVERTER.convert(bindingSet, varOrder);
-tx.set(spID + NODEID_BS_DELIM + bindingSetString,
FluoQueryColumns.STATEMENT_PATTERN_BINDING_SET, valueString);
+final VariableOrder varOrder = spMetadata.getVariableOrder();
+final String bindingSetString = getBindingSet(triple, pattern,
varOrder);
+
+// Statement matches to a binding set.
+if(bindingSetString.length() != 0) {
+// Fetch the triple's visibility label.
+final String visibility = tx.gets(brow.toString(),
FluoQueryColumns.TRIPLES, "");
+
+// Create the Row ID for the emitted binding set. It does
not contain visibilities.
+final String row = spID + NODEID_BS_DELIM +
bindingSetString;
+final Bytes rowBytes = Bytes.of(
row.getBytes(Charsets.UTF_8) );
+
+// If this is a new Binding Set, then emit it.
+if(tx.get(rowBytes,
FluoQueryColumns.STATEMENT_PATTERN_BINDING_SET) == null) {
--- End diff --
This is a worthwhile optimization.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977548#comment-15977548
]
ASF GitHub Bot commented on RYA-260:
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112534616
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/BindingSetStringConverter.java
---
@@ -58,7 +56,13 @@
@Override
public String convert(final BindingSet bindingSet, final VariableOrder
varOrder) {
-checkBindingsSubsetOfVarOrder(bindingSet, varOrder);
+requireNonNull(bindingSet);
+requireNonNull(varOrder);
+
+// If the binding set is empty, just return empty string.
+if(bindingSet.getBindingNames().isEmpty()) {
+return "";
+}
--- End diff --
Seems like it would be better either throw an exception or to let the user
know that the bindingSetString was only being built using a subset of the
variables in varOrder in the event that the variables in varOrder aren't the
same as the variables in the BindingSet.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973919#comment-15973919
]
ASF GitHub Bot commented on RYA-260:
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112089139
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloCreatePCJ.java
---
@@ -139,16 +138,15 @@ private void updateFluoApp(final String ryaInstance,
final String fluoAppName, f
// Connect to the Fluo application that is updating this
instance's PCJs.
final AccumuloConnectionDetails cd =
super.getAccumuloConnectionDetails();
-final FluoClient fluoClient = new FluoClientFactory().connect(
+try(final FluoClient fluoClient = new FluoClientFactory().connect(
cd.getUsername(),
new String(cd.getPassword()),
cd.getInstanceName(),
cd.getZookeepers(),
-fluoAppName);
-
-// Initialize the PCJ within the Fluo application.
-final org.apache.rya.indexing.pcj.fluo.api.CreatePcj fluoCreatePcj
= new org.apache.rya.indexing.pcj.fluo.api.CreatePcj();
-fluoCreatePcj.withRyaIntegration(pcjId, pcjStorage, fluoClient,
getConnector(), ryaInstance);
+fluoAppName);) {
+// Initialize the PCJ within the Fluo application.
+final org.apache.rya.indexing.pcj.fluo.api.CreatePcj
fluoCreatePcj = new org.apache.rya.indexing.pcj.fluo.api.CreatePcj();
--- End diff --
can you import CreatePcj?
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973921#comment-15973921
]
ASF GitHub Bot commented on RYA-260:
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112101036
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/VisibilityBindingSetSerde.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.fluo.app;
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+
+import org.apache.fluo.api.data.Bytes;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Serializes and deserializes a {@link VisibilityBindingSet} to and from
{@link Bytes} objects.
+ */
+@DefaultAnnotation(NonNull.class)
+public class VisibilityBindingSetSerde {
--- End diff --
interesting name
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973920#comment-15973920
]
ASF GitHub Bot commented on RYA-260:
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112089607
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.api/src/main/java/org/apache/rya/indexing/pcj/fluo/api/CreatePcj.java
---
@@ -82,6 +82,7 @@
*/
@DefaultAnnotation(NonNull.class)
public class CreatePcj {
+private static final Logger log = Logger.getLogger(CreatePcj.class);
--- End diff --
not really a review comment, but i think it might behoove(sp?) us to open a
ticket to make our logging uniform.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973791#comment-15973791
]
ASF GitHub Bot commented on RYA-260:
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/156#discussion_r112094335
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/instance/AccumuloRyaInstanceDetailsRepository.java
---
@@ -89,12 +89,17 @@ public AccumuloRyaInstanceDetailsRepository(final
Connector connector, final Str
@Override
public boolean isInitialized() throws RyaDetailsRepositoryException {
+Scanner scanner = null;
try {
--- End diff --
Both of these could be try with resources.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (RYA-260) Add Aggregation support for Fluo/PCJ app
[
https://issues.apache.org/jira/browse/RYA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973552#comment-15973552
]
ASF GitHub Bot commented on RYA-260:
GitHub user kchilton2 opened a pull request:
https://github.com/apache/incubator-rya/pull/156
RYA-260 Fluo SPARQL Processing Aggregation Support
## Description
>What Changed?
Added Aggregation support to the Fluo PCJ application. It is a known
problem that the Accumulo PCJ Index does not properly handle aggregation output
exporting. That should be covered by a new ticket. Also fixed a bunch of
resource leaks and changed up how the Fluo integration tests were implemented
so that they use the Fluo recipe for Accumulo exporting integration tests.
### Links
[Jira](https://issues.apache.org/jira/browse/RYA-260)
### Checklist
- [ ] Code Review
- [ ] Squash Commits
People To Reivew
@meiercaleb
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kchilton2/incubator-rya RYA-260-fixed
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rya/pull/156.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #156
commit 278f6d92fcfe1c68a388b8aa1fcd7ca2134bcfaa
Author: Kevin Chilton
Date: 2017-04-07T19:57:57Z
RYA-260 Fluo PCJ application has had Aggregation support added to it. Also
fixed a bunch of resource leaks that were causing integration tests to fail.
> Add Aggregation support for Fluo/PCJ app
>
>
> Key: RYA-260
> URL: https://issues.apache.org/jira/browse/RYA-260
> Project: Rya
> Issue Type: New Feature
>Reporter: Andrew Smith
>Assignee: Kevin Chilton
>
> A user must be able to submit a PCJ query that contains the following
> aggregation functions from SPARQL:
> * Sum
> * Count
> * Average
> * Min
> * Max
> This task does not include any aggregations that appear within a GroupBy
> clause. We only need to support queries that have the aggregation within the
> SELECT section.
> For example, the following query should be processed:
> {code}
> SELECT (avg(?price) as ?averagePrice)
> {
> urn:BookA urn:price ?price.
> }
> {code}
> And the following query should not be processed because it requires a group
> by:
> {code}
> SELECT ?title (avg(?price) as ?averagePrice)
> {
> ?title urn:price ?price.
> }
> GROUP BY ?title
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
