date:20170203

[jira] [Commented] (RYA-241) Add Geowave support

2017-02-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/RYA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852140#comment-15852140
 ] 

ASF GitHub Bot commented on RYA-241:


Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/139#discussion_r99424572
  
--- Diff: pom.xml ---
@@ -80,6 +80,7 @@ under the License.
 0.9.2 
 
 1.3.0-m1 
+0.9.4-SNAPSHOT 
--- End diff --

Can you back this down to a released version (e.g. 0.9.3)?


> Add Geowave support
> ---
>
> Key: RYA-241
> URL: https://issues.apache.org/jira/browse/RYA-241
> Project: Rya
>  Issue Type: New Feature
>Reporter: Eric White
>Assignee: Eric White
>
> Add in Geowave as an alternative to Geomesa



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RYA-241) Add Geowave support

2017-02-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/RYA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852141#comment-15852141
 ] 

ASF GitHub Bot commented on RYA-241:


Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/139#discussion_r99417564
  
--- Diff: 
extras/rya.geoindexing/src/main/java/org/apache/rya/indexing/accumulo/geo/GeoWaveGeoIndexer.java
 ---
@@ -0,0 +1,661 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.accumulo.geo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Instance;
+import org.apache.accumulo.core.client.mock.MockInstance;
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.experimental.AbstractAccumuloIndexer;
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.GeoIndexer;
+import org.apache.rya.indexing.Md5Hash;
+import org.apache.rya.indexing.StatementConstraints;
+import org.apache.rya.indexing.StatementSerializer;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.geotools.data.DataStore;
+import org.geotools.data.DataUtilities;
+import org.geotools.data.FeatureSource;
+import org.geotools.data.FeatureStore;
+import org.geotools.factory.CommonFactoryFinder;
+import org.geotools.factory.Hints;
+import org.geotools.feature.DefaultFeatureCollection;
+import org.geotools.feature.SchemaException;
+import org.geotools.feature.simple.SimpleFeatureBuilder;
+import org.geotools.filter.text.cql2.CQLException;
+import org.geotools.filter.text.ecql.ECQL;
+import org.opengis.feature.simple.SimpleFeature;
+import org.opengis.feature.simple.SimpleFeatureType;
+import org.opengis.filter.Filter;
+import org.opengis.filter.FilterFactory;
+import org.opengis.filter.identity.Identifier;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.query.QueryEvaluationException;
+
+import com.vividsolutions.jts.geom.Geometry;
+import com.vividsolutions.jts.io.ParseException;
+
+import info.aduna.iteration.CloseableIteration;
+import mil.nga.giat.geowave.adapter.vector.FeatureDataAdapter;
+import mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveGTDataStore;
+import 
mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveGTDataStoreFactory;
+import mil.nga.giat.geowave.adapter.vector.plugin.GeoWavePluginException;
+import mil.nga.giat.geowave.adapter.vector.query.cql.CQLQuery;
+import 
mil.nga.giat.geowave.core.geotime.ingest.SpatialDimensionalityTypeProvider;
+import mil.nga.giat.geowave.core.store.CloseableIterator;
+import mil.nga.giat.geowave.core.store.StoreFactoryFamilySpi;
+import mil.nga.giat.geowave.core.store.index.PrimaryIndex;
+import mil.nga.giat.geowave.core.store.memory.MemoryStoreFactoryFamily;
+import mil.nga.giat.geowave.core.store.query.EverythingQuery;
+import mil.nga.giat.geowave.core.store.query.QueryOptions;
+import mil.nga.giat.geowave.datastore.accumulo.AccumuloDataStore;
+import mil.nga.giat.geowave.datastore.accumulo.AccumuloStoreFactoryFamily;
+
+/**
+ * A {@link

[jira] [Commented] (RYA-241) Add Geowave support

2017-02-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/RYA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852139#comment-15852139
 ] 

ASF GitHub Bot commented on RYA-241:


Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/139#discussion_r99424446
  
--- Diff: extras/indexingExample/src/main/assembly/assembly.xml ---
@@ -33,6 +33,7 @@ under the License.
 
 
org.apache.rya:rya.indexing:*:accumulo-server
 
org.locationtech.geomesa:geomesa-accumulo-distributed-runtime:*
+
mil.nga.giat:geowave-datastore-accumulo:*
--- End diff --

I think the dependency is something like `geowave-deploy-0.9.3-accumulo`?  
I can't find this dependency on their maven repository, though


> Add Geowave support
> ---
>
> Key: RYA-241
> URL: https://issues.apache.org/jira/browse/RYA-241
> Project: Rya
>  Issue Type: New Feature
>Reporter: Eric White
>Assignee: Eric White
>
> Add in Geowave as an alternative to Geomesa



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

2017-02-03 Thread amihalik

Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/139#discussion_r99417564
  
--- Diff: 
extras/rya.geoindexing/src/main/java/org/apache/rya/indexing/accumulo/geo/GeoWaveGeoIndexer.java
 ---
@@ -0,0 +1,661 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.accumulo.geo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Instance;
+import org.apache.accumulo.core.client.mock.MockInstance;
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.experimental.AbstractAccumuloIndexer;
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.GeoIndexer;
+import org.apache.rya.indexing.Md5Hash;
+import org.apache.rya.indexing.StatementConstraints;
+import org.apache.rya.indexing.StatementSerializer;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.geotools.data.DataStore;
+import org.geotools.data.DataUtilities;
+import org.geotools.data.FeatureSource;
+import org.geotools.data.FeatureStore;
+import org.geotools.factory.CommonFactoryFinder;
+import org.geotools.factory.Hints;
+import org.geotools.feature.DefaultFeatureCollection;
+import org.geotools.feature.SchemaException;
+import org.geotools.feature.simple.SimpleFeatureBuilder;
+import org.geotools.filter.text.cql2.CQLException;
+import org.geotools.filter.text.ecql.ECQL;
+import org.opengis.feature.simple.SimpleFeature;
+import org.opengis.feature.simple.SimpleFeatureType;
+import org.opengis.filter.Filter;
+import org.opengis.filter.FilterFactory;
+import org.opengis.filter.identity.Identifier;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.query.QueryEvaluationException;
+
+import com.vividsolutions.jts.geom.Geometry;
+import com.vividsolutions.jts.io.ParseException;
+
+import info.aduna.iteration.CloseableIteration;
+import mil.nga.giat.geowave.adapter.vector.FeatureDataAdapter;
+import mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveGTDataStore;
+import 
mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveGTDataStoreFactory;
+import mil.nga.giat.geowave.adapter.vector.plugin.GeoWavePluginException;
+import mil.nga.giat.geowave.adapter.vector.query.cql.CQLQuery;
+import 
mil.nga.giat.geowave.core.geotime.ingest.SpatialDimensionalityTypeProvider;
+import mil.nga.giat.geowave.core.store.CloseableIterator;
+import mil.nga.giat.geowave.core.store.StoreFactoryFamilySpi;
+import mil.nga.giat.geowave.core.store.index.PrimaryIndex;
+import mil.nga.giat.geowave.core.store.memory.MemoryStoreFactoryFamily;
+import mil.nga.giat.geowave.core.store.query.EverythingQuery;
+import mil.nga.giat.geowave.core.store.query.QueryOptions;
+import mil.nga.giat.geowave.datastore.accumulo.AccumuloDataStore;
+import mil.nga.giat.geowave.datastore.accumulo.AccumuloStoreFactoryFamily;
+
+/**
+ * A {@link GeoIndexer} wrapper around a GeoWave {@link 
AccumuloDataStore}. This class configures and connects to the Datastore, 
creates the
+ * RDF Feature Type, and interacts with the Datastore.
+ * 
+ * Specifically, this class

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

2017-02-03 Thread amihalik

Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/139#discussion_r99424446
  
--- Diff: extras/indexingExample/src/main/assembly/assembly.xml ---
@@ -33,6 +33,7 @@ under the License.
 
 
org.apache.rya:rya.indexing:*:accumulo-server
 
org.locationtech.geomesa:geomesa-accumulo-distributed-runtime:*
+
mil.nga.giat:geowave-datastore-accumulo:*
--- End diff --

I think the dependency is something like `geowave-deploy-0.9.3-accumulo`?  
I can't find this dependency on their maven repository, though


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

2017-02-03 Thread amihalik

Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/139#discussion_r99424572
  
--- Diff: pom.xml ---
@@ -80,6 +80,7 @@ under the License.
 0.9.2 
 
 1.3.0-m1 
+0.9.4-SNAPSHOT 
--- End diff --

Can you back this down to a released version (e.g. 0.9.3)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-241) Add Geowave support

2017-02-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/RYA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852054#comment-15852054
 ] 

ASF GitHub Bot commented on RYA-241:


GitHub user ejwhite922 opened a pull request:

https://github.com/apache/incubator-rya/pull/139

RYA-241 Adding GeoWave indexer to rya.geoindexing.

## Description
Added GeoWave indexer to rya.geoindexing.  This is a configurable option 
that can be used in place of GeoMesa.

### Tests
Unit Tests

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-241)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Review
@pujav65 
@amihalik 
@isper3at 
@DLotts

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ejwhite922/incubator-rya RYA-241_Geowave

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #139


commit 208b1efb1b62d1877a439cf3f2379313085b13a9
Author: eric.white 
Date:   2017-01-27T15:46:08Z

RYA-241 Adding GeoWave indexer to rya.geoindexing.  This is a configurable 
option that can be used in place of GeoMesa.




> Add Geowave support
> ---
>
> Key: RYA-241
> URL: https://issues.apache.org/jira/browse/RYA-241
> Project: Rya
>  Issue Type: New Feature
>Reporter: Eric White
>Assignee: Eric White
>
> Add in Geowave as an alternative to Geomesa



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

2017-02-03 Thread ejwhite922

GitHub user ejwhite922 opened a pull request:

https://github.com/apache/incubator-rya/pull/139

RYA-241 Adding GeoWave indexer to rya.geoindexing.

## Description
Added GeoWave indexer to rya.geoindexing.  This is a configurable option 
that can be used in place of GeoMesa.

### Tests
Unit Tests

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-241)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Review
@pujav65 
@amihalik 
@isper3at 
@DLotts

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ejwhite922/incubator-rya RYA-241_Geowave

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #139


commit 208b1efb1b62d1877a439cf3f2379313085b13a9
Author: eric.white 
Date:   2017-01-27T15:46:08Z

RYA-241 Adding GeoWave indexer to rya.geoindexing.  This is a configurable 
option that can be used in place of GeoMesa.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (RYA-246) Separate Fluo query result exporters, Rya PCJ intermingled kafka triggers

2017-02-03 Thread David W. Lotts (JIRA)


 [ 
https://issues.apache.org/jira/browse/RYA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David W. Lotts updated RYA-246:
---
Description: 
Currently the Fluo query observer framework (FQF) has a great plugin-in-ish way 
of doing exports.  (See field: {{exporters}} in class: 
{{org.apache.rya.indexing.pcj.fluo.app.observers.QueryResultObserver}} )  
Exports are query results and the action taken when they occur.  Currently 
there are two.  Despite this awesome framework, there are three interminglings 
that can become issues in a production system.  Feel free to make sub tasks for 
each of these three tasks:

1. Create PCJ independently of alerts/triggers
At query register time (with FQF), optionally create PCJ's and optionally 
create alerts (with Alerting) preferably atomically, so if it fails in the 
middle it does not leave dangling references and garbage -- currently there is 
one method that does everything: CreatePCJ.withRyaInteraction()

2. Enable exporters per query -- currently all queries will be exported on all 
globally enabled exporters.  When the query is registered (with FQF) specify 
the exporters that should be used.

3. Separate export observer queues per exporter -- Currently retrying a failed 
export observer will re-export all export types, even if some already 
succeeded, causing duplicate triggers.


Also there is a need to refine the terms for java identifiers (for example: 
fields, classes) and documentation.  Please document here a decision about what 
we name different things.  The term PCJ are the index tables used by Rya to 
optimize it's queries.  But sometimes it is used to describe the whole Fluo 
framework.  Please define a unique term for the following.  The starred term 
will be used here, but feel free to rename them:

- Rya query optimization PCJ  (ex: the Blabla exporter)  PCJ*
- Kafka trigger alert notifier   (ex: the Foofoo exporter)  Alerting*
- Fluo query observer framework  (ex: results from the Barbar)  FQF*

Unique identifers:

- FQF uses QueryID
- Alerting uses TopicID 
- PCJ uses PcjId

Note that currently TopicID is assigned the same as QueryID, but that badly 
presupposes queryID's cannot be grouped onto one Topic.

  was:
Currently the Fluo query observer framework (FQF) has a great plugin-in-ish way 
of doing exports.  (See field: {{exporters}} in class: 
{{org.apache.rya.indexing.pcj.fluo.app.observers.QueryResultObserver}} )  
Exports are query results and the action taken when they occur.  Currently 
there are two.  Despite this awsome framework, there are three interminglings 
that can become issues in a production system:

1. Create PCJ independently of alerts/triggers
At query register time (with FQF), optionally create PCJ's and optionally 
create alerts (with Alerting) preferably atomically, so if it fails in the 
middle it does not leave dangling references and garbage -- currently there is 
one method that does everything: CreatePCJ.withRyaInteraction()

2. Enable exporters per query -- currently all queries will be exported on all 
globally enabled exporters

3. Separate export observer queues per exporter -- Currently retrying a failed 
export observer will re-export all export types, even if some already 
succeeded, causing duplicate triggers.


Also there is a need to refine the terms for java identifiers (for example: 
fields, classes) and documentation.  Please document here a decision about what 
we name different things.  The term PCJ are the index tables used by Rya to 
optimize it's queries.  But sometimes it is used to describe the whole Fluo 
framework.  Please define a unique term for the following.  The starred term 
will be used here, but feel free to rename them:

- Rya query optimization PCJ  (ex: the Blabla exporter)  PCJ*
- Kafka trigger alert notifier   (ex: the Foofoo exporter)  Alerting*
- Fluo query observer framework  (ex: results from the Barbar)  FQF*

Unique identifers:

- FQF uses QueryID
- Alerting uses TopicID 
- PCJ uses PcjId

Note that currently TopicID is assigned the same as QueryID, but that badly 
presupposes queryID's cannot be grouped onto one Topic.


> Separate Fluo query result exporters, Rya PCJ intermingled kafka triggers
> -
>
> Key: RYA-246
> URL: https://issues.apache.org/jira/browse/RYA-246
> Project: Rya
>  Issue Type: Improvement
>  Components: dao
> Environment: Fluo, Kafka, PCJ, Accumulo
>Reporter: David W. Lotts
>
> Currently the Fluo query observer framework (FQF) has a great plugin-in-ish 
> way of doing exports.  (See field: {{exporters}} in class: 
> {{org.apache.rya.indexing.pcj.fluo.app.observers.QueryResultObserver}} )  
> Exports are query results and the action taken when they occur.  Currently 
> there are two.  Despite this awesome framework, there are three 
>

[jira] [Commented] (RYA-241) Add Geowave support

[jira] [Commented] (RYA-241) Add Geowave support

[jira] [Commented] (RYA-241) Add Geowave support

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

[jira] [Commented] (RYA-241) Add Geowave support

[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...

[jira] [Updated] (RYA-246) Separate Fluo query result exporters, Rya PCJ intermingled kafka triggers

9 matches

Site Navigation

Mail list logo

Footer information