[GitHub] incubator-rya issue #299: RYA-500: Make RdfFileInputTool to accept multiple ...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/299 Asfbot build ---
[GitHub] incubator-rya pull request #248: RYA-356 Added a Twill App for running the p...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/248#discussion_r148799979
--- Diff:
extras/periodic.notification/api/src/main/java/org/apache/rya/periodic/notification/api/PeriodicNotificationClient.java
---
@@ -36,29 +36,26 @@
* Adds a new notification to be registered with the {@link
NotificationCoordinatorExecutor}
* @param notification - notification to be added
*/
-public void addNotification(PeriodicNotification notification);
--- End diff --
tldr: I think we should pull the PR as is, but don't perform "style
changes" and "feature changes" in a single PR in the future.
I didn't know that the modifier was redundant. I'd keep in the "public"
modifiers to be consistent with the code base, but I'm fine with the change
since it is consistent within the sub module.
Overall, I think there are two key things to keep in mind:
1. Style changes may turn off reviewers from looking at your "feature
change" PR. Style changes typically make your code commit a lot larger than
necessary and muddy important "feature changes" with a lot of style changes.
2. Defining a community style and tools to evaluate and enforce this style
is very important. Consistent style across the code base will lower the
barrier of entry for new developers, will build a larger community, etc. I
would enjoy reviewing and pulling those changes in. But I'd be equally turned
off if those style changes included "feature changes".
---
[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/236 Asfbot build ---
[GitHub] incubator-rya issue #241: RYA-333 Added "-P coverage" to generate code cover...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/241 I tried creating a build on b.a.o with the coverage profile enabled and "record jacoco report" plug-in enabled. You can check on it's progress here: https://builds.apache.org/job/incubator-rya-master-with-optionals-with-coverage/ I'll roll these changes into the master builds once this PR is pulled in. ---
[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/236 @ejwhite922 There's a couple issues with the batch writer and it doesn't have to do with exceptions being caught. (1) the QueueFullChecker thread should not be a daemon thread "The Java Virtual Machine exits when the only threads running are all daemon threads" means that the JVM will exit while flushing to mongo. (2) It doesn't seem like the secondary indexer (i.e. geoindexer) is flushed when I close down the sail layer. Can you double check that? Also, it seems like the JVM takes a very long time to exit after I call sailrepository.shutDown(). I think has to do with the BatchWriter as well. ---
[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/236 The more I look at this, the more Iâm confident that this has to more with the mongo libs and not anything that we wrote for Rya. Iâll try to put together an example tomorrow. ---
[GitHub] incubator-rya issue #236: RYA-393 MongoDB Batch Writer Duplicate Key Excepti...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/236 I'm going to hold off on merging this for a minute. This exception talk has me thinking there are other exceptions that we are not processing correctly. brb ---
[GitHub] incubator-rya pull request #219: Creating inference Examples
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/219#discussion_r137566563
--- Diff: extras/indexingExample/src/main/java/InferenceExamples.java ---
@@ -0,0 +1,592 @@
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.commons.lang.Validate;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.ConsoleAppender;
+import org.apache.log4j.Level;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.log4j.PatternLayout;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.apache.rya.indexing.mongodb.MongoIndexingConfiguration;
+import
org.apache.rya.indexing.mongodb.MongoIndexingConfiguration.MongoDBIndexingConfigBuilder;
+import org.apache.rya.mongodb.MockMongoFactory;
+import org.apache.rya.mongodb.MongoConnectorFactory;
+import org.apache.rya.rdftriplestore.RdfCloudTripleStore;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.apache.zookeeper.ClientCnxn;
+import org.openrdf.model.Namespace;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.model.vocabulary.RDFS;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.QueryResultHandlerException;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResultHandler;
+import org.openrdf.query.TupleQueryResultHandlerException;
+import org.openrdf.query.Update;
+import org.openrdf.query.UpdateExecutionException;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.RepositoryResult;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+
+import com.mongodb.MongoClient;
+import com.mongodb.ServerAddress;
+
+
+//
+//See notes in inferenceExamples_readme.txt
+//
+
+public class InferenceExamples {
+ private static final Logger log =
Logger.getLogger(InferenceExamples.class);
+
+ private static final boolean IS_DETAILED_LOGGING_ENABLED = false;
+
+ //
+ // Connection configuration parameters
+ //
+
+ private static final boolean PRINT_QUERIES = true;
+ private static final String MONGO_DB = "rya";
+ private static final String MONGO_COLL_PREFIX = "rya_";
+ private static final boolean USE_EMBEDDED_MONGO = true;
+ private static final String MONGO_INSTANCE_URL = "localhost";
+ private static final String MONGO_INSTANCE_PORT = "27017";
+ private static final String MongoUserName="usern";
+ private static final String MongoUserPassword="passwd";
+
+ public static void setupLogging() {
+ final Logger rootLogger = LogManager.getRootLogger();
+ rootLogger.setLevel(Level.OFF);
+ final ConsoleAppender ca = (ConsoleAppender)
rootLogger.getAppender("stdout");
--- End diff --
Do you still need this if you've created the config file?
---
[GitHub] incubator-rya pull request #219: Creating inference Examples
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/219#discussion_r137567339
--- Diff: extras/indexingExample/src/main/java/InferenceExamples.java ---
@@ -0,0 +1,592 @@
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.commons.lang.Validate;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.ConsoleAppender;
+import org.apache.log4j.Level;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.log4j.PatternLayout;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.apache.rya.indexing.mongodb.MongoIndexingConfiguration;
+import
org.apache.rya.indexing.mongodb.MongoIndexingConfiguration.MongoDBIndexingConfigBuilder;
+import org.apache.rya.mongodb.MockMongoFactory;
+import org.apache.rya.mongodb.MongoConnectorFactory;
+import org.apache.rya.rdftriplestore.RdfCloudTripleStore;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.apache.zookeeper.ClientCnxn;
+import org.openrdf.model.Namespace;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.model.vocabulary.RDFS;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.QueryResultHandlerException;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResultHandler;
+import org.openrdf.query.TupleQueryResultHandlerException;
+import org.openrdf.query.Update;
+import org.openrdf.query.UpdateExecutionException;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.RepositoryResult;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+
+import com.mongodb.MongoClient;
+import com.mongodb.ServerAddress;
+
+
+//
+//See notes in inferenceExamples_readme.txt
+//
+
+public class InferenceExamples {
+ private static final Logger log =
Logger.getLogger(InferenceExamples.class);
+
+ private static final boolean IS_DETAILED_LOGGING_ENABLED = false;
+
+ //
+ // Connection configuration parameters
+ //
+
+ private static final boolean PRINT_QUERIES = true;
+ private static final String MONGO_DB = "rya";
+ private static final String MONGO_COLL_PREFIX = "rya_";
+ private static final boolean USE_EMBEDDED_MONGO = true;
+ private static final String MONGO_INSTANCE_URL = "localhost";
+ private static final String MONGO_INSTANCE_PORT = "27017";
+ private static final String MongoUserName="usern";
+ private static final String MongoUserPassword="passwd";
+
+ public static void setupLogging() {
+ final Logger rootLogger = LogManager.getRootLogger();
+ rootLogger.setLevel(Level.OFF);
+ final ConsoleAppender ca = (ConsoleAppender)
rootLogger.getAppender("stdout");
+ ca.setLayout(new PatternLayout("%d{MMM dd HH:mm:ss} %5p
[%t] (%F:%L) - %m%n"));
+ rootLogger.setLevel(Level.INFO);
+ // Filter out noisy messages from the following classes.
+ Logger.getLogger(ClientCnxn.class).setLevel(Level.OFF);
+ Logger.getLogger(MockMongoFactory.class).setLevel(Level.OFF);
+ }
+
+ public static void main(final String[] args) throws Exception {
+ if (IS_DETAILED_LOGGING_ENABLED) {
+ setupLogging();
+ }
+ final Configuration conf = getConf();
+ conf.setBoolean(ConfigUtils.DISPLAY_QUERY_PLAN, PRINT_QUERIES);
+
+ SailRepository repository = null;
+ SailRepositoryConnection conn = null;
+ try {
+ log.info("Connecting to Indexing Sail Repository.");
+ final Sail sail = RyaSailFactory.getInstance(conf);
+ repository = new SailRepository(sail);
+ conn = repository.getConnection();
+
+
+ final long start = System.currentTimeMillis();
+
+ testInfer(conn, sail);
+ testPropertyChainInference(conn, sail);
+ testPropertyChainInferenceAlt
[GitHub] incubator-rya pull request #219: Creating inference Examples
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/219#discussion_r137566475 --- Diff: extras/indexingExample/src/main/java/InferenceExamples.java --- @@ -0,0 +1,592 @@ +import java.io.IOException; +import java.util.List; + +import org.apache.commons.lang.Validate; +import org.apache.hadoop.conf.Configuration; +import org.apache.log4j.ConsoleAppender; +import org.apache.log4j.Level; +import org.apache.log4j.LogManager; +import org.apache.log4j.Logger; +import org.apache.log4j.PatternLayout; +import org.apache.rya.indexing.accumulo.ConfigUtils; +import org.apache.rya.indexing.mongodb.MongoIndexingConfiguration; +import org.apache.rya.indexing.mongodb.MongoIndexingConfiguration.MongoDBIndexingConfigBuilder; +import org.apache.rya.mongodb.MockMongoFactory; +import org.apache.rya.mongodb.MongoConnectorFactory; +import org.apache.rya.rdftriplestore.RdfCloudTripleStore; +import org.apache.rya.rdftriplestore.inference.InferenceEngineException; +import org.apache.rya.sail.config.RyaSailFactory; +import org.apache.zookeeper.ClientCnxn; +import org.openrdf.model.Namespace; +import org.openrdf.model.URI; +import org.openrdf.model.ValueFactory; +import org.openrdf.model.vocabulary.OWL; +import org.openrdf.model.vocabulary.RDF; +import org.openrdf.model.vocabulary.RDFS; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.QueryResultHandlerException; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResultHandler; +import org.openrdf.query.TupleQueryResultHandlerException; +import org.openrdf.query.Update; +import org.openrdf.query.UpdateExecutionException; +import org.openrdf.repository.RepositoryException; +import org.openrdf.repository.RepositoryResult; +import org.openrdf.repository.sail.SailRepository; +import org.openrdf.repository.sail.SailRepositoryConnection; +import org.openrdf.sail.Sail; + +import com.mongodb.MongoClient; +import com.mongodb.ServerAddress; + + +// +//See notes in inferenceExamples_readme.txt --- End diff -- Remove this reference (since the file is gone) and document the purpose of the examples here. ---
[GitHub] incubator-rya pull request #213: RYA-246-Query-Export-Strategy
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/213#discussion_r134795805
--- Diff:
common/rya.api/src/main/java/org/apache/rya/api/client/CreatePCJ.java ---
@@ -27,8 +29,28 @@
@DefaultAnnotation(NonNull.class)
public interface CreatePCJ {
+public static enum QueryType{Construct, Projection, Periodic};
+public static enum ExportStrategy{Rya, Kafka, NoOpExport};
+
+
+/**
+ * Designate a new PCJ that will be maintained by the target instance
of Rya.
+ * Results will be exported according to the specified export
strategies.
+ *
+ * @param instanceName - Indicates which Rya instance will create and
maintain
+ * the PCJ. (not null)
+ * @param sparql - The SPARQL query that will be maintained. (not null)
+ * @param strategies - The export strategies used to export results
for this query
+ * @return The ID that was assigned to this newly created PCJ.
+ * @throws InstanceDoesNotExistException No instance of Rya exists for
the provided name.
+ * @throws RyaClientException Something caused the command to fail.
+ */
+public String createPCJ(final String instanceName, String sparql,
Set strategies) throws InstanceDoesNotExistException,
RyaClientException;
--- End diff --
I can export a result both to Rya and Kafka?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #204: RYA-337 Adding batch queries to MongoDB. Cl...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/204#discussion_r133984224
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/iter/RyaStatementBindingSetCursorIterator.java
---
@@ -44,20 +48,21 @@
public class RyaStatementBindingSetCursorIterator implements
CloseableIteration<Entry<RyaStatement, BindingSet>, RyaDAOException> {
private static final Logger log =
Logger.getLogger(RyaStatementBindingSetCursorIterator.class);
+
+private static final int QUERY_BATCH_SIZE = 50;
--- End diff --
Empirical testing... To support the "show me the first 10" or "first 100"
type queries. I noticed that if we left this at 1000, then the "show me the
first 10" or "first 100" type queries were very slow. However, the "show me
all" (where all = 30k), we equally as quick using 50 or 1000.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #204: RYA-337 Adding batch queries to MongoDB. Cl...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/204#discussion_r133983577
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -140,72 +121,35 @@ public MongoDBRdfConfiguration getConf() {
public CloseableIteration<RyaStatement, RyaDAOException> batchQuery(
final Collection stmts, MongoDBRdfConfiguration
conf)
throws RyaDAOException {
-if (conf == null) {
-conf = configuration;
-}
-final Long maxResults = conf.getLimit();
-final Set queries = new HashSet();
-
-try {
-for (final RyaStatement stmt : stmts) {
-queries.add( strategy.getQuery(stmt));
- }
+final Map<RyaStatement, BindingSet> queries = new HashMap<>();
-// TODO not sure what to do about regex ranges?
-final RyaStatementCursorIterator iterator = new
RyaStatementCursorIterator(getCollection(conf), queries,
-strategy, configuration.getAuthorizations());
-
-if (maxResults != null) {
-iterator.setMaxResults(maxResults);
-}
-return iterator;
-} catch (final Exception e) {
-throw new RyaDAOException(e);
+for (final RyaStatement stmt : stmts) {
+queries.put(stmt, new MapBindingSet());
}
+return new
RyaStatementCursorIterator(queryWithBindingSet(queries.entrySet(), conf));
}
+
@Override
public CloseableIterable query(final RyaQuery ryaQuery)
throws RyaDAOException {
-final Set queries = new HashSet();
-
-try {
-queries.add( strategy.getQuery(ryaQuery));
-
-// TODO not sure what to do about regex ranges?
-// TODO this is gross
-final RyaStatementCursorIterable iterator = new
RyaStatementCursorIterable(
-new NonCloseableRyaStatementCursorIterator(new
RyaStatementCursorIterator(getCollection(getConf()),
-queries, strategy,
configuration.getAuthorizations(;
-
-return iterator;
-} catch (final Exception e) {
-throw new RyaDAOException(e);
-}
+return query(new
BatchRyaQuery(Collections.singleton(ryaQuery.getQuery(;
}
+
@Override
public CloseableIterable query(final BatchRyaQuery
batchRyaQuery)
throws RyaDAOException {
- try {
- final Set queries = new HashSet();
-for (final RyaStatement statement :
batchRyaQuery.getQueries()){
-queries.add( strategy.getQuery(statement));
+final Map<RyaStatement, BindingSet> queries = new HashMap<>();
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #204: RYA-337 Adding batch queries to MongoDB. Cl...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/204#discussion_r133983396
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -140,72 +121,35 @@ public MongoDBRdfConfiguration getConf() {
public CloseableIteration<RyaStatement, RyaDAOException> batchQuery(
final Collection stmts, MongoDBRdfConfiguration
conf)
throws RyaDAOException {
-if (conf == null) {
-conf = configuration;
-}
-final Long maxResults = conf.getLimit();
-final Set queries = new HashSet();
-
-try {
-for (final RyaStatement stmt : stmts) {
-queries.add( strategy.getQuery(stmt));
- }
+final Map<RyaStatement, BindingSet> queries = new HashMap<>();
-// TODO not sure what to do about regex ranges?
-final RyaStatementCursorIterator iterator = new
RyaStatementCursorIterator(getCollection(conf), queries,
-strategy, configuration.getAuthorizations());
-
-if (maxResults != null) {
-iterator.setMaxResults(maxResults);
-}
-return iterator;
-} catch (final Exception e) {
-throw new RyaDAOException(e);
+for (final RyaStatement stmt : stmts) {
+queries.put(stmt, new MapBindingSet());
}
+return new
RyaStatementCursorIterator(queryWithBindingSet(queries.entrySet(), conf));
}
+
@Override
public CloseableIterable query(final RyaQuery ryaQuery)
throws RyaDAOException {
-final Set queries = new HashSet();
-
-try {
-queries.add( strategy.getQuery(ryaQuery));
-
-// TODO not sure what to do about regex ranges?
-// TODO this is gross
-final RyaStatementCursorIterable iterator = new
RyaStatementCursorIterable(
-new NonCloseableRyaStatementCursorIterator(new
RyaStatementCursorIterator(getCollection(getConf()),
-queries, strategy,
configuration.getAuthorizations(;
-
-return iterator;
-} catch (final Exception e) {
-throw new RyaDAOException(e);
-}
+return query(new
BatchRyaQuery(Collections.singleton(ryaQuery.getQuery(;
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #204: RYA-337 Adding batch queries to MongoDB. Cl...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/204#discussion_r133983248
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -111,25 +97,20 @@ public MongoDBRdfConfiguration getConf() {
if (conf == null) {
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #204: RYA-337 Adding batch queries to MongoDB. Cl...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/204#discussion_r133983122
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -86,22 +84,10 @@ public MongoDBRdfConfiguration getConf() {
public CloseableIteration<RyaStatement, RyaDAOException> query(
final RyaStatement stmt, MongoDBRdfConfiguration conf)
throws RyaDAOException {
-if (conf == null) {
-conf = configuration;
-}
-final Long maxResults = conf.getLimit();
-final Set queries = new HashSet();
-final DBObject query = strategy.getQuery(stmt);
-queries.add(query);
-final MongoDatabase db =
mongoClient.getDatabase(conf.getMongoDBName());
-final MongoCollection collection =
db.getCollection(conf.getTriplesCollectionName());
-final RyaStatementCursorIterator iterator = new
RyaStatementCursorIterator(collection, queries, strategy,
-conf.getAuthorizations());
-
-if (maxResults != null) {
-iterator.setMaxResults(maxResults);
-}
-return iterator;
+Entry<RyaStatement, BindingSet> entry = new
AbstractMap.SimpleEntry<>(stmt, new MapBindingSet());
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #204: RYA-337 Adding batch queries to MongoDB. Cl...
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/204 RYA-337 Adding batch queries to MongoDB. Closes #204 ## Description Added a batch query mechanism to MongoDB DAO and simplified MongoDBQueryEngine ### Tests Ran unit tests ### Links [Jira](https://issues.apache.org/jira/browse/RYA-337) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Review @isper3at @pujav65 @meiercaleb You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya RYA-337 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/204.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #204 commit d19ed3c7fb4ca71bc3c75e1aafc77c4b89c76270 Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-08-08T15:17:37Z RYA-337 Adding batch queries to MongoDB. Closes #204 Additionally, simplifying MongoDBQueryEngine --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #203: Fixes a regression with QueriesBenchmarkConfReader...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/203 Thanks. I'll merge it now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #199: RYA-316 Long OBJ string
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/199 @isper3at thanks! Merging now... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #199: RYA-316 Long OBJ string
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/199 @pujav65 totally agree. That's why I added the "please get some metrics" portion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/199#discussion_r132245115
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java
---
@@ -64,14 +68,14 @@
@Override
public void createIndices(final DBCollection coll){
BasicDBObject doc = new BasicDBObject();
-doc.put(SUBJECT, 1);
-doc.put(PREDICATE, 1);
+doc.put(SUBJECT_HASH, 1);
+doc.put(PREDICATE_HASH, 1);
coll.createIndex(doc);
--- End diff --
@pujav65 thanks.
@isper3at clearly a bug. please add OBJECT_HASH, OBJECT_TYPE_HASH to the
first index.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/199#discussion_r132242206 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java --- @@ -53,8 +54,11 @@ public static final String OBJECT_TYPE_VALUE = XMLSchema.ANYURI.stringValue(); public static final String CONTEXT = "context"; public static final String PREDICATE = "predicate"; -public static final String OBJECT = "object"; +public static final String PREDICATE_HASH = "predicate_hash"; +public static final String OBJECT = "object_original"; --- End diff -- @pujav65 I'm concerned about index size. please hash everything. If you want another ticket for "please hash everything" I'm fine with that, but let's knock that out while @isper3at is cleaning this stuff up. Key thing with mongo is to get the index to fit in memory, so lets do that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/199#discussion_r132235261
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java
---
@@ -64,14 +68,14 @@
@Override
public void createIndices(final DBCollection coll){
BasicDBObject doc = new BasicDBObject();
-doc.put(SUBJECT, 1);
-doc.put(PREDICATE, 1);
+doc.put(SUBJECT_HASH, 1);
+doc.put(PREDICATE_HASH, 1);
coll.createIndex(doc);
-doc = new BasicDBObject(PREDICATE, 1);
-doc.put(OBJECT, 1);
+doc = new BasicDBObject(PREDICATE_HASH, 1);
+doc.put(OBJECT_HASH, 1);
doc.put(OBJECT_TYPE, 1);
coll.createIndex(doc);
-doc = new BasicDBObject(OBJECT, 1);
+doc = new BasicDBObject(OBJECT_HASH, 1);
doc.put(OBJECT_TYPE, 1);
doc.put(SUBJECT, 1);
--- End diff --
SUBJECT_HASH
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/199#discussion_r132235567
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java
---
@@ -64,14 +68,14 @@
@Override
public void createIndices(final DBCollection coll){
BasicDBObject doc = new BasicDBObject();
-doc.put(SUBJECT, 1);
-doc.put(PREDICATE, 1);
+doc.put(SUBJECT_HASH, 1);
+doc.put(PREDICATE_HASH, 1);
coll.createIndex(doc);
--- End diff --
@pujav65 Looking over this index creation code... this seems like a bug...
where's the SPO index? I think this first index should be SUBJECT_HASH,
PREDICATE_HASH, OBJECT_HASH, OBJECT_TYPE_HASH
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/199#discussion_r132235041 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java --- @@ -53,8 +54,11 @@ public static final String OBJECT_TYPE_VALUE = XMLSchema.ANYURI.stringValue(); public static final String CONTEXT = "context"; public static final String PREDICATE = "predicate"; -public static final String OBJECT = "object"; +public static final String PREDICATE_HASH = "predicate_hash"; +public static final String OBJECT = "object_original"; --- End diff -- yep, might as well hash context and object type as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/199#discussion_r132191684 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java --- @@ -53,8 +54,11 @@ public static final String OBJECT_TYPE_VALUE = XMLSchema.ANYURI.stringValue(); public static final String CONTEXT = "context"; public static final String PREDICATE = "predicate"; -public static final String OBJECT = "object"; +public static final String PREDICATE_HASH = "predicate_hash"; +public static final String OBJECT = "object_original"; --- End diff -- Can you change this to just "object" or change "context" "predicate" "subject" to "xxx_original" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #199: RYA-316 Long OBJ string
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/199#discussion_r132192953
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java
---
@@ -85,14 +89,14 @@ public DBObject getQuery(final RyaStatement stmt) {
final RyaURI context = stmt.getContext();
final BasicDBObject query = new BasicDBObject();
if (subject != null){
-query.append(SUBJECT, subject.getData());
+query.append(SUBJECT_HASH,
DigestUtils.sha256Hex(subject.getData()));
--- End diff --
Can we store/query in binary (32 bytes) vs hex string (64 bytes)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #199: RYA-316 Long OBJ string
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/199 @isper3at Since this was so straightforward, can you go ahead and hash the S and O as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #196: RYA-313
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/196#discussion_r131464691
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -118,7 +124,8 @@ public MongoDBRdfConfiguration getConf() {
}
// TODO not sure what to do about regex ranges?
-final RyaStatementBindingSetCursorIterator iterator = new
RyaStatementBindingSetCursorIterator(coll, rangeMap, strategy,
conf.getAuthorizations());
+final RyaStatementBindingSetCursorIterator iterator = new
RyaStatementBindingSetCursorIterator(
+getCollection(conf), rangeMap, strategy,
conf.getAuthorizations());
--- End diff --
Since you call getCollection(conf), I believe that coll is now an unused
field. Can you remove it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #196: RYA-313
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/196 Looks good @isper3at. I'm going to test this out and wait for one more before I merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #194: RYA-325 Renamed rya.console to rya.shell.
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/194#discussion_r131287677 --- Diff: extras/pom.xml --- @@ -34,7 +34,7 @@ under the License. rya.prospector rya.manual rya.periodic.service -rya.console +shell --- End diff -- okay, I'm fine with this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #181: Rya-307 MongoDB Rya DAO Batch Writer
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/181 @isper3at Thanks for taking a look at it. I'm testing it out now before I merge it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #194: RYA-325 Renamed rya.console to rya.shell.
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/194#discussion_r131240139 --- Diff: extras/pom.xml --- @@ -34,7 +34,7 @@ under the License. rya.prospector rya.manual rya.periodic.service -rya.console +shell --- End diff -- agree with Andrew. Why not put this is rya.shell? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #181: Rya-307 MongoDB Rya DAO Batch Writer
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/181 @pujav65 sounds good. We've been testing out #186 and that's about to be merged. Hold off on merging #181 until #186 (and perhaps #182) is on master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #182: RYA-324, RYA-272 Geo refactoring and examples
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/182 I know that we planned to commit this PR to master first, but #186 is ready to go. I looked at this and #186. It turns out that we can commit #186 into master, then commit this PR, and we won't have any merge issues. Since #186 is ready now, I'm going to commit it, and then assist you with any issues you have merging this PR. --Aaron --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #190: RYA-330 Cleaned up build testing and added profile...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/190 Rebuilding on my local machine with this latest update. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/188 @jdasch `ERROR: Maven JVM terminated unexpectedly with exit code 137`. The build servers are dead to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #187: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/187 @pujav65 The link you sent pointed to a failed build (the grey bulb means it failed). Also, you're right, that configuration doesn't build optional. I kicked of a job earlier at https://builds.apache.org/job/incubator-rya-fork-with-optionals/23/ That build failed as well. Can you look at the Fluo integration tests and figure out why they are killing the build? I've been working on it for most of today and I'm not getting anywhere. I'm tempted to disable those tests, run them individually, and push a commit to apache master if that all works out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #186: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/186 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #186: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/186 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130093053
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java
---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements
ExecuteSparqlQuery {
+private static final Logger log =
Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+private final InstanceExists instanceExists;
+
+/**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used
to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of
Accumulo
+ * that hosts Rya instance. (not null)
+ */
+public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails
connectionDetails, final Connector connector) {
+super(connectionDetails, connector);
+instanceExists = new AccumuloInstanceExists(connectionDetails,
connector);
+}
+
+
+@Override
+public String executeSparqlQuery(final String ryaInstanceName, final
String sparqlQuery)
+throws InstanceDoesNotExistException, RyaClientException {
+requireNonNull(ryaInstanceName);
+requireNonNull(sparqlQuery);
+
+// Ensure the Rya Instance exists.
+if(!instanceExists.exists(ryaInstanceName)) {
+throw new InstanceDoesNotExistException(String.format("There
is no Rya instance named '%s'.", ryaInstanceName));
+}
+
+
+Sail sail = null;
+SailRepository sailRepo = null;
+SailRepositoryConnection sailRepoConn = null;
+
+try {
+// Get a Sail object that is connected to the Rya instance.
+final AccumuloRdfConfiguration ryaConf =
getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+sail = RyaSailFactory.getInstance(ryaConf);
+
+// Load the file.
+sailRepo = new SailRepository(sail);
+sailRepoConn = sailRepo.getConnection(
[GitHub] incubator-rya issue #186: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/186 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r12333
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java
---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements
ExecuteSparqlQuery {
+private static final Logger log =
Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+private final InstanceExists instanceExists;
+
+/**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used
to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of
Accumulo
+ * that hosts Rya instance. (not null)
+ */
+public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails
connectionDetails, final Connector connector) {
+super(connectionDetails, connector);
+instanceExists = new AccumuloInstanceExists(connectionDetails,
connector);
+}
+
+
+@Override
+public String executeSparqlQuery(final String ryaInstanceName, final
String sparqlQuery)
+throws InstanceDoesNotExistException, RyaClientException {
+requireNonNull(ryaInstanceName);
+requireNonNull(sparqlQuery);
+
+// Ensure the Rya Instance exists.
+if(!instanceExists.exists(ryaInstanceName)) {
+throw new InstanceDoesNotExistException(String.format("There
is no Rya instance named '%s'.", ryaInstanceName));
+}
+
+
+Sail sail = null;
+SailRepository sailRepo = null;
+SailRepositoryConnection sailRepoConn = null;
+
+try {
+// Get a Sail object that is connected to the Rya instance.
+final AccumuloRdfConfiguration ryaConf =
getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+sail = RyaSailFactory.getInstance(ryaConf);
+
+// Load the file.
+sailRepo = new SailRepository(sail);
+sailRepoConn = sailRepo.getConnection(
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129998856
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java
---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
--- End diff --
ExecuteSparqlQuery?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r12058
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java
---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements
ExecuteSparqlQuery {
+private static final Logger log =
Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+private final InstanceExists instanceExists;
+
+/**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used
to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of
Accumulo
+ * that hosts Rya instance. (not null)
+ */
+public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails
connectionDetails, final Connector connector) {
+super(connectionDetails, connector);
+instanceExists = new AccumuloInstanceExists(connectionDetails,
connector);
+}
+
+
+@Override
+public String executeSparqlQuery(final String ryaInstanceName, final
String sparqlQuery)
+throws InstanceDoesNotExistException, RyaClientException {
+requireNonNull(ryaInstanceName);
+requireNonNull(sparqlQuery);
+
+// Ensure the Rya Instance exists.
+if(!instanceExists.exists(ryaInstanceName)) {
+throw new InstanceDoesNotExistException(String.format("There
is no Rya instance named '%s'.", ryaInstanceName));
+}
+
+
+Sail sail = null;
+SailRepository sailRepo = null;
+SailRepositoryConnection sailRepoConn = null;
+
+try {
+// Get a Sail object that is connected to the Rya instance.
+final AccumuloRdfConfiguration ryaConf =
getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+sail = RyaSailFactory.getInstance(ryaConf);
+
+// Load the file.
--- End diff --
?
---
If your project is set up for it, you can reply to this email and have your
[GitHub] incubator-rya issue #183: RYA-323 Enabling Multi Var Geo Functions. Closes #...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/183 @pujav65 yeah puja hold off on merging any of these PRs until we get the build sorted. we'll test out @isper3at's #187 and #186 PRs first, then determine if need this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #187: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/187 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #187: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/187 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #187: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/187 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #187: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/187 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #187: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/187 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #187: Rya 317 Added Near function for geo mongodb
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/187 Rya 317 Added Near function for geo mongodb (Note this is a continuation of PR #186. This PR includes additional commits that will be squashed) ## Description >What Changed? Changed GeoTupleSet to no long always call GeoWithin Create the Geo Indexer properly Add GeoNear function for MongoDB and an unsupported exception for accumulo. ### Tests >Coverage? Full Near() coverage added test to ensure delete works on GEO ### Links [Jira](https://issues.apache.org/jira/browse/RYA-317) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Reivew @isper3at @meiercaleb You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya RYA-317 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/187.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #187 commit 168769a5bec213173f47701ebadda842c11c1092 Author: isper3at <smith...@gmail.com> Date: 2017-07-25T20:15:38Z RYA-317 Added Mongo Near support Support for mongo added. fixed so no longer always calls sfWithin properly enabled Geo indexer in mongo commit 0ca7597ff4d0c85a7f233bb1746cf401a81f6725 Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-07-27T14:54:57Z Build Fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #186: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/186 @isper3at also: bonus for the use of "spheroid complement disc" :1st_place_medal: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #186: Rya 317 Added Near function for geo mongodb
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/186 @isper3at this commit looks good. I don't have any comments on my initial pass. I'm building and testing this locally right now to see if I have any issues with it. @meiercaleb @isper3at note that this branch has drifted from master and therefore won't build. I cherry-picked the relevant commit (912a1bd), created amihalik:RYA-317, and I'm building it on Jenkins [here](https://builds.apache.org/job/incubator-rya-fork-with-optionals/20/) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #177: RYA-280-Periodic Query Service
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/177#discussion_r129669294
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/PeriodicQueryStorageMetadata.java
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage;
+
+import java.util.Objects;
+
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Metadata for a given PeriodicQueryStorage table.
+ */
+public class PeriodicQueryStorageMetadata {
+
+private String sparql;
+private VariableOrder varOrder;
+
+/**
+ * Create a PeriodicQueryStorageMetadat object
--- End diff --
typo: ...Metadat**a**
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #177: RYA-280-Periodic Query Service
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/177#discussion_r129670570 --- Diff: extras/rya.indexing.pcj/src/test/java/org/apache/rya/indexing/pcj/storage/accumulo/accumulo/AccumuloPeriodicQueryResultStorageIT.java --- @@ -0,0 +1,271 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.accumulo.accumulo; --- End diff -- accumulo.accumulo? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #177: RYA-280-Periodic Query Service
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/177#discussion_r129668480
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/accumulo/PeriodicQueryTableNameFactory.java
---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import org.apache.rya.indexing.pcj.storage.PeriodicQueryResultStorage;
+
+/**
+ * Class for creating the names of {@link PeriodicQueryResultStorage}
tables.
+ *
+ */
+public class PeriodicQueryTableNameFactory {
+
+public static final String PeriodicTableSuffix = "PERIODIC_QUERY_";
--- End diff --
"midfix" :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #179: RYA-309 Added delete support to indexers
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/179 @meiercaleb is good to go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #179: RYA-309 Added delete support to indexers
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/179 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #183: RYA-323 Enabling Multi Var Geo Functions. Closes #...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/183 @isper3at I can't wait to see your PR We might still need my PR to do a multi-var geo:equals. Although I wouldn't doubt that you fixed that as well. :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #183: RYA-323 Enabling Multi Var Geo Functions. Closes #...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/183 @isper3at Can you look at this too. This PR is required to enable one of the main use cases for the geo:nearby function that you're working on --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #183: RYA-323 Enabling Multi Var Geo Functions. C...
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/183 RYA-323 Enabling Multi Var Geo Functions. Closes #183 ## Description >What Changed? [Brief Description of what changed] ### Tests >Coverage? [Description of what tests were written] ### Links [Jira](https://issues.apache.org/jira/browse/RYA-323) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Reivew @meiercaleb You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya GEO-MULTI-VAR Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/183.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #183 commit 80d98670873bbe422f3cefaec27752d2bbd3cd89 Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-07-25T17:43:03Z RYA-323 Enabling Multi Var Geo Functions. Closes #183 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #181: Rya-307 MongoDB Rya DAO Batch Writer
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/181#discussion_r129044540
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/AbstractMongoIndexer.java
---
@@ -68,15 +73,28 @@
protected T storageStrategy;
+private MongoDbBatchWriter mongoDbBatchWriter;
+
protected void initCore() {
dbName = conf.get(MongoDBRdfConfiguration.MONGO_DB_NAME);
db = this.mongoClient.getDB(dbName);
-collection =
db.getCollection(conf.get(MongoDBRdfConfiguration.MONGO_COLLECTION_PREFIX,
"rya") + getCollectionName());
+final String collectionName =
conf.get(MongoDBRdfConfiguration.MONGO_COLLECTION_PREFIX, "rya") +
getCollectionName();
--- End diff --
Can you modify the AbstactMongoIndexer so that it and the DAO share the
same batch writer?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #181: Rya-307 MongoDB Rya DAO Batch Writer
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/181 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #179: RYA-309 Added delete support to indexers
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/179 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #179: RYA-309 Added delete support to indexers
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/179#discussion_r128336316 --- Diff: extras/rya.geoindexing/src/test/java/org/apache/rya/indexing/mongo/MongoGeoIT.java --- @@ -0,0 +1,135 @@ +package org.apache.rya.indexing.mongo; --- End diff -- RAT does not approve! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #179: RYA-309 Added delete support to indexers
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/179#discussion_r128336188
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRyaDAO.java ---
@@ -93,7 +93,7 @@ public MongoDBRyaDAO(final MongoDBRdfConfiguration conf,
final MongoClient mongo
@Override
public void setConf(final MongoDBRdfConfiguration conf) {
this.conf = conf;
-this.auths = conf.getAuthorizations();
+auths = conf.getAuthorizations();
--- End diff --
where did `this` go?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #179: RYA-309 Added delete support to indexers
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/179#discussion_r128335780
--- Diff:
extras/rya.geoindexing/src/test/java/org/apache/rya/indexing/mongo/MongoGeoIT.java
---
@@ -0,0 +1,135 @@
+package org.apache.rya.indexing.mongo;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.GeoConstants;
+import org.apache.rya.indexing.GeoRyaSailFactory;
+import org.apache.rya.indexing.OptionalConfigUtils;
+import org.apache.rya.indexing.TemporalInstant;
+import org.apache.rya.indexing.TemporalInstantRfc3339;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.apache.rya.indexing.mongodb.MongoIndexingConfiguration;
+import org.apache.rya.mongodb.MockMongoFactory;
+import org.junit.Before;
+import org.junit.Test;
+import org.openrdf.model.Resource;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.StatementImpl;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.model.vocabulary.RDFS;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.Update;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+
+import com.mongodb.MongoClient;
+import com.vividsolutions.jts.geom.Coordinate;
+import com.vividsolutions.jts.geom.Geometry;
+import com.vividsolutions.jts.geom.GeometryFactory;
+
+public class MongoGeoIT {
+private MongoClient client;
+private Sail sail;
+private SailRepositoryConnection conn;
+
+@Before
+public void before() throws Exception {
+final MongoIndexingConfiguration indxrConf =
MongoIndexingConfiguration.builder()
+.setMongoCollectionPrefix("rya_")
+.setMongoDBName("indexerTests")
+.setUseMongoFreetextIndex(true)
+.setUseMongoTemporalIndex(true)
+.setMongoFreeTextPredicates(RDFS.LABEL.stringValue())
+.setMongoTemporalPredicates("Property:atTime")
+.build();
+
+client = MockMongoFactory.newFactory().newMongoClient();
+indxrConf.setBoolean(OptionalConfigUtils.USE_GEO, true);
+indxrConf.set(ConfigUtils.GEO_PREDICATES_LIST,
"http://www.opengis.net/ont/geosparql#asWKT;);
+indxrConf.setBoolean(ConfigUtils.USE_MONGO, true);
+indxrConf.setMongoClient(client);
+
+sail = GeoRyaSailFactory.getInstance(indxrConf);
+conn = new SailRepository(sail).getConnection();
+conn.begin();
+}
+
+@Test
+public void deleteTest() throws Exception {
+populateRya();
+
+assertEquals(8,
client.getDatabase("indexerTests").getCollection("rya__triples").count());
+assertEquals(4,
client.getDatabase("indexerTests").getCollection("rya_rya_geo").count());
+assertEquals(1,
client.getDatabase("indexerTests").getCollection("rya_rya_temporal").count());
+assertEquals(2,
client.getDatabase("indexerTests").getCollection("rya_rya_freetext").count());
+
+//free text -- remove one from many
+String delete = "DELETE DATA \n" //
+ + "{\n"
+ + " <http://www.w3.org/2000/01/rdf-schema#label>
\"Alice Palace Hose\" "
+ + "}";
+Update update = conn.prepareUpdate(QueryLanguage.SPARQL, delete);
+update.execute();
+
+// temporal -- remove one from one
+delete = "DELETE DATA \n" //
+ + "{\n"
+ + "\"0001-02-03T04:05:06Z\" "
+ + "}";
+
--- End diff --
Can you add a test for geo?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #176: Rya 268
Github user amihalik closed the pull request at: https://github.com/apache/incubator-rya/pull/176 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #178: RYA-315 Fixing issue with Core Indices on MongoDB
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/178 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #175: Rya 268 Adding in Geo Predicate Filter List
Github user amihalik closed the pull request at: https://github.com/apache/incubator-rya/pull/175 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #161: RYA-273-Construct Query Support
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/161#discussion_r123325067
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/query/ConstructQueryMetadata.java
---
@@ -0,0 +1,192 @@
+package org.apache.rya.indexing.pcj.fluo.app.query;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.rya.indexing.pcj.fluo.app.ConstructGraph;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.openrdf.query.BindingSet;
+
+import com.google.common.base.Objects;
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Metadata object used to store metadata for Construct Query Nodes found
in
+ * SPARQL queries.
+ *
+ */
+public class ConstructQueryMetadata extends CommonNodeMetadata {
+
+private String childNodeId;
+private ConstructGraph graph;
+private String sparql;
+
+/**
+ * Creates ConstructQueryMetadata object from the provided metadata
arguments.
+ * @param nodeId - id for the ConstructQueryNode
+ * @param childNodeId - id for the child of the ConstructQueryNode
+ * @param graph - {@link ConstructGraph} used to project {@link
BindingSet}s onto sets of statement representing construct graph
+ * @param sparql - SPARQL query containing construct graph
+ */
+public ConstructQueryMetadata(String nodeId, String childNodeId,
ConstructGraph graph, String sparql) {
+super(nodeId, new VariableOrder("subject", "predicate", "object"));
+Preconditions.checkNotNull(childNodeId);
+Preconditions.checkNotNull(graph);
+Preconditions.checkNotNull(sparql);
+this.childNodeId = childNodeId;
+this.graph = graph;
+this.sparql = sparql;
+}
+
+/**
+ * @return sparql query string representing this construct query
+ */
+public String getSparql() {
+return sparql;
+}
+
+/**
+ * @return The node whose results are projected onto the given
+ * {@link ConstructGraph}.
+ */
+public String getChildNodeId() {
+return childNodeId;
+}
+
+/**
+ * @return The ConstructGraph used to form statement {@link
BindingSet}s for
+ * this Construct Query
+ */
+public ConstructGraph getConstructGraph() {
+return graph;
+}
+
+@Override
+public int hashCode() {
+return Objects.hashCode(super.getNodeId(),
super.getVariableOrder(), childNodeId, graph, sparql);
+}
+
+@Override
+public boolean equals(final Object o) {
+if (this == o) {
+return true;
+}
+
+if (o instanceof ConstructQueryMetadata) {
+ConstructQueryMetadata queryMetadata =
(ConstructQueryMetadata) o;
+if (super.equals(queryMetadata)) {
+return new EqualsBuilder().append(childNodeId,
queryMetadata.childNodeId).append(graph, queryMetadata.graph)
+.append(sparql, queryMetadata.sparql).isEquals();
+}
+return false;
+}
+return false;
+}
+
+@Override
+public String toString() {
+return new StringBuilder().append("Construct Query Metadata
{\n").append("Node ID: " + super.getNodeId() + "\n")
--- End diff --
Consider Objects.ToStringHelper
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
[GitHub] incubator-rya pull request #161: RYA-273-Construct Query Support
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/161#discussion_r123325785
--- Diff:
extras/rya.pcj.fluo/pcj.fluo.app/src/main/java/org/apache/rya/indexing/pcj/fluo/app/query/ConstructQueryMetadata.java
---
@@ -0,0 +1,192 @@
+package org.apache.rya.indexing.pcj.fluo.app.query;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.rya.indexing.pcj.fluo.app.ConstructGraph;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.openrdf.query.BindingSet;
+
+import com.google.common.base.Objects;
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Metadata object used to store metadata for Construct Query Nodes found
in
+ * SPARQL queries.
+ *
+ */
+public class ConstructQueryMetadata extends CommonNodeMetadata {
+
+private String childNodeId;
+private ConstructGraph graph;
+private String sparql;
+
+/**
+ * Creates ConstructQueryMetadata object from the provided metadata
arguments.
+ * @param nodeId - id for the ConstructQueryNode
+ * @param childNodeId - id for the child of the ConstructQueryNode
+ * @param graph - {@link ConstructGraph} used to project {@link
BindingSet}s onto sets of statement representing construct graph
+ * @param sparql - SPARQL query containing construct graph
+ */
+public ConstructQueryMetadata(String nodeId, String childNodeId,
ConstructGraph graph, String sparql) {
+super(nodeId, new VariableOrder("subject", "predicate", "object"));
+Preconditions.checkNotNull(childNodeId);
+Preconditions.checkNotNull(graph);
+Preconditions.checkNotNull(sparql);
+this.childNodeId = childNodeId;
+this.graph = graph;
+this.sparql = sparql;
+}
+
+/**
+ * @return sparql query string representing this construct query
+ */
+public String getSparql() {
+return sparql;
+}
+
+/**
+ * @return The node whose results are projected onto the given
+ * {@link ConstructGraph}.
+ */
+public String getChildNodeId() {
+return childNodeId;
+}
+
+/**
+ * @return The ConstructGraph used to form statement {@link
BindingSet}s for
+ * this Construct Query
+ */
+public ConstructGraph getConstructGraph() {
+return graph;
+}
+
+@Override
+public int hashCode() {
+return Objects.hashCode(super.getNodeId(),
super.getVariableOrder(), childNodeId, graph, sparql);
+}
+
+@Override
+public boolean equals(final Object o) {
+if (this == o) {
+return true;
+}
+
+if (o instanceof ConstructQueryMetadata) {
+ConstructQueryMetadata queryMetadata =
(ConstructQueryMetadata) o;
+if (super.equals(queryMetadata)) {
+return new EqualsBuilder().append(childNodeId,
queryMetadata.childNodeId).append(graph, queryMetadata.graph)
+.append(sparql, queryMetadata.sparql).isEquals();
+}
+return false;
+}
+return false;
+}
+
+@Override
+public String toString() {
+return new StringBuilder().append("Construct Query Metadata
{\n").append("Node ID: " + super.getNodeId() + "\n")
--- End diff --
no need to implement this, however.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
ena
[GitHub] incubator-rya issue #138: Rya 236 - GeoTemporal Indexing
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/138 Wow. These failures aren't really failures. You just so happened to get the same flaky machine (H22) three times in a row. I rebuilt your changes [here ](https://builds.apache.org/job/incubator-rya-fork/2/) and [here](https://builds.apache.org/job/incubator-rya-fork-with-optionals/6/) and locally. Everything built and ran just fine (excluding the known test failures in rya.merge). I'm going to merge these changes into master. I'll squash your last three commits into a single commit. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #171: RYA-288: corrected path to launch PcjAdminClient
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/171 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #171: RYA-288: corrected path to launch PcjAdminClient
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/171 afsbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #166: RYA-268 Adding Geoindexing Profile to Web p...
GitHub user amihalik reopened a pull request: https://github.com/apache/incubator-rya/pull/166 RYA-268 Adding Geoindexing Profile to Web project ## Description Added a geoindexing profile to the web project ### Tests no new tests ### Links [Jira RYA-268](https://issues.apache.org/jira/browse/RYA-268) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Review @meiercaleb @isper3at You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya RYA-268 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/166.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #166 commit 6a3923c73da658006ea57c0e83b1ea48f114bbfd Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-06-02T19:52:30Z RYA-268 Adding Geoindexing Profile to Web project --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #166: RYA-268 Adding Geoindexing Profile to Web project
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/166 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #166: RYA-268 Adding Geoindexing Profile to Web p...
Github user amihalik closed the pull request at: https://github.com/apache/incubator-rya/pull/166 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #170: RYA-284: updated to support the use of bloom filte...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/170 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #170: RYA-284: updated to support the use of bloom filte...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/170 I'm seeing some new tests errors and failures when I try to build this PR on my machine. I'll have asfbot build this on b.a.o. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/169#discussion_r120970204
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/query/AccumuloRyaQueryEngine.java
---
@@ -374,56 +387,64 @@ protected String getData(RyaType ryaType) {
final TripleRowRegex tripleRowRegex =
strategy.buildRegex(regexSubject, regexPredicate, regexObject, null, null);
final String table = layoutToTable(layout,
tableLayoutStrategy);
-boolean useBatchScanner = ranges.size() > maxRanges;
+final boolean useBatchScanner = ranges.size() > maxRanges;
FluentCloseableIterable results = null;
if (useBatchScanner) {
-BatchScanner scanner = connector.createBatchScanner(table,
authorizations, numQueryThreads);
+final BatchScanner scanner =
connector.createBatchScanner(table, authorizations, numQueryThreads);
scanner.setRanges(ranges);
fillScanner(scanner, context, null, ttl, null,
tripleRowRegex, ryaQuery.getConf());
-results = FluentCloseableIterable.from(new
ScannerBaseCloseableIterable(scanner))
-
.transform(keyValueToRyaStatementFunctionMap.get(layout));
-} else {
+results = FluentCloseableIterable.from(new
ScannerBaseCloseableIterable(scanner)).transform(keyValueToRyaStatementFunctionMap.get(layout));
+}
+else {
final RyaURI fcontext = context;
final RdfCloudTripleStoreConfiguration fconf =
ryaQuery.getConf();
-FluentIterable fluent =
FluentIterable.from(ranges)
-.transformAndConcat(new Function<Range,
Iterable<Map.Entry<Key, Value>>>() {
-@Override
-public Iterable<Map.Entry<Key, Value>>
apply(Range range) {
-try {
-Scanner scanner =
connector.createScanner(table, authorizations);
-scanner.setRange(range);
-fillScanner(scanner, fcontext, null,
ttl, null, tripleRowRegex, fconf);
-return scanner;
-} catch (Exception e) {
-throw new RuntimeException(e);
-}
-}
-
}).transform(keyValueToRyaStatementFunctionMap.get(layout));
+final FluentIterable fluent =
FluentIterable.from(ranges).transformAndConcat(range -> {
+try {
+final Scanner scanner =
connector.createScanner(table, authorizations);
+scanner.setRange(range);
+fillScanner(scanner, fcontext, null, ttl, null,
tripleRowRegex, fconf);
+return scanner;
+}
+catch (final Exception e) {
+throw new RuntimeException(e);
+}
+
}).transform(keyValueToRyaStatementFunctionMap.get(layout));
results =
FluentCloseableIterable.from(CloseableIterables.wrap(fluent));
}
if (maxResults != null) {
results = results.limit(maxResults.intValue());
}
return results;
-} catch (Exception e) {
+}
+catch (final Exception e) {
throw new RyaDAOException(e);
}
}
-protected void fillScanner(ScannerBase scanner, RyaURI context, String
qualifier, Long ttl, Long currentTime,
-TripleRowRegex tripleRowRegex,
RdfCloudTripleStoreConfiguration conf) throws IOException {
-if (context != null && qualifier != null) {
+private static Text createText(final byte[] data, final RyaType
ryaType) throws RyaTypeResolverException {
--- End diff --
"appendType" would be a better name for this method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #168: Pr jenkins test2
Github user amihalik closed the pull request at: https://github.com/apache/incubator-rya/pull/168 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #167: Testing Jenkins PR Build... Ignore this PR.
Github user amihalik closed the pull request at: https://github.com/apache/incubator-rya/pull/167 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #168: Pr jenkins test2
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/168 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #166: RYA-268 Adding Geoindexing Profile to Web project
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/166 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #167: Testing Jenkins PR Build... Ignore this PR.
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/167 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #168: Pr jenkins test2
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/168 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #166: RYA-268 Adding Geoindexing Profile to Web project
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/166 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #168: Pr jenkins test2
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/168 Pr jenkins test2 ## Description >What Changed? [Brief Description of what changed] ### Tests >Coverage? [Description of what tests were written] ### Links [Jira](https://issues.apache.org/jira/browse/RYA-NUMBER) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Reivew [Add those who should review this] You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya PR-JENKINS-TEST2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/168.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #168 commit 35f789e4ed86f2f3af1cbb5ca11e25858cccdd6d Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-06-02T21:22:24Z making a simple test commit c01a6055699cdcd80f3fb31b679ba2f3eb983d1c Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-06-02T21:37:23Z minor chnage --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #167: Testing Jenkins PR Build... Ignore this PR.
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/167 Testing Jenkins PR Build... Ignore this PR. Testing Jenkins PR Build... Ignore this PR. You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya PR-JENKINS-TEST Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #167 commit 35f789e4ed86f2f3af1cbb5ca11e25858cccdd6d Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-06-02T21:22:24Z making a simple test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #166: RYA-268 Adding Geoindexing Profile to Web p...
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/166 RYA-268 Adding Geoindexing Profile to Web project ## Description Added a geoindexing profile to the web project ### Tests no new tests ### Links [Jira RYA-268](https://issues.apache.org/jira/browse/RYA-268) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Review @meiercaleb @isper3at You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya RYA-268 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/166.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #166 commit 6a3923c73da658006ea57c0e83b1ea48f114bbfd Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-06-02T19:52:30Z RYA-268 Adding Geoindexing Profile to Web project --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #151: RYA-253 Converted the rya.prospector class from Gr...
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/151 asfbot build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya issue #152: RYA-253 Build Fix for Geowave repo move
Github user amihalik commented on the issue: https://github.com/apache/incubator-rya/pull/152 @meiercaleb yep, I built it locally after blowing away geowave in my .m2 repo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #152: RYA-253 Build Fix for Geowave repo move
GitHub user amihalik opened a pull request: https://github.com/apache/incubator-rya/pull/152 RYA-253 Build Fix for Geowave repo move ## Description >What Changed? [Brief Description of what changed] ### Tests >Coverage? [Description of what tests were written] ### Links [Jira](https://issues.apache.org/jira/browse/RYA-NUMBER) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Reivew [Add those who should review this] You can merge this pull request into a Git repository by running: $ git pull https://github.com/amihalik/incubator-rya RYA-128 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/152.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #152 commit 24e01ab7120d4b94b0cb2f2b030198fb01a645a4 Author: Aaron Mihalik <miha...@alum.mit.edu> Date: 2017-04-11T21:36:40Z RYA-253 Build Fix for Geowave repo move --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/139#discussion_r99417564
--- Diff:
extras/rya.geoindexing/src/main/java/org/apache/rya/indexing/accumulo/geo/GeoWaveGeoIndexer.java
---
@@ -0,0 +1,661 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.accumulo.geo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Instance;
+import org.apache.accumulo.core.client.mock.MockInstance;
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.experimental.AbstractAccumuloIndexer;
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.GeoIndexer;
+import org.apache.rya.indexing.Md5Hash;
+import org.apache.rya.indexing.StatementConstraints;
+import org.apache.rya.indexing.StatementSerializer;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.geotools.data.DataStore;
+import org.geotools.data.DataUtilities;
+import org.geotools.data.FeatureSource;
+import org.geotools.data.FeatureStore;
+import org.geotools.factory.CommonFactoryFinder;
+import org.geotools.factory.Hints;
+import org.geotools.feature.DefaultFeatureCollection;
+import org.geotools.feature.SchemaException;
+import org.geotools.feature.simple.SimpleFeatureBuilder;
+import org.geotools.filter.text.cql2.CQLException;
+import org.geotools.filter.text.ecql.ECQL;
+import org.opengis.feature.simple.SimpleFeature;
+import org.opengis.feature.simple.SimpleFeatureType;
+import org.opengis.filter.Filter;
+import org.opengis.filter.FilterFactory;
+import org.opengis.filter.identity.Identifier;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.query.QueryEvaluationException;
+
+import com.vividsolutions.jts.geom.Geometry;
+import com.vividsolutions.jts.io.ParseException;
+
+import info.aduna.iteration.CloseableIteration;
+import mil.nga.giat.geowave.adapter.vector.FeatureDataAdapter;
+import mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveGTDataStore;
+import
mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveGTDataStoreFactory;
+import mil.nga.giat.geowave.adapter.vector.plugin.GeoWavePluginException;
+import mil.nga.giat.geowave.adapter.vector.query.cql.CQLQuery;
+import
mil.nga.giat.geowave.core.geotime.ingest.SpatialDimensionalityTypeProvider;
+import mil.nga.giat.geowave.core.store.CloseableIterator;
+import mil.nga.giat.geowave.core.store.StoreFactoryFamilySpi;
+import mil.nga.giat.geowave.core.store.index.PrimaryIndex;
+import mil.nga.giat.geowave.core.store.memory.MemoryStoreFactoryFamily;
+import mil.nga.giat.geowave.core.store.query.EverythingQuery;
+import mil.nga.giat.geowave.core.store.query.QueryOptions;
+import mil.nga.giat.geowave.datastore.accumulo.AccumuloDataStore;
+import mil.nga.giat.geowave.datastore.accumulo.AccumuloStoreFactoryFamily;
+
+/**
+ * A {@link GeoIndexer} wrapper around a GeoWave {@link
AccumuloDataStore}. This class configures and connects to the Datastore,
creates the
+ * RDF Feature Type, and interacts with the Datastore.
+ *
+ * Spec
[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/139#discussion_r99424446 --- Diff: extras/indexingExample/src/main/assembly/assembly.xml --- @@ -33,6 +33,7 @@ under the License. org.apache.rya:rya.indexing:*:accumulo-server org.locationtech.geomesa:geomesa-accumulo-distributed-runtime:* + mil.nga.giat:geowave-datastore-accumulo:* --- End diff -- I think the dependency is something like `geowave-deploy-0.9.3-accumulo`? I can't find this dependency on their maven repository, though --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #139: RYA-241 Adding GeoWave indexer to rya.geoin...
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/139#discussion_r99424572 --- Diff: pom.xml --- @@ -80,6 +80,7 @@ under the License. 0.9.2 1.3.0-m1 +0.9.4-SNAPSHOT --- End diff -- Can you back this down to a released version (e.g. 0.9.3)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rya pull request #138: [WIP] Rya 236 - GeoTemporal Indexing
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/138#discussion_r96710955
--- Diff:
extras/rya.geotemporalindexing/src/test/java/org/apache/rya/indexing/GeoTemporalIndexerTest.java
---
@@ -0,0 +1,945 @@
+package org.apache.rya.indexing;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.indexing.GeoTemporalIndexer.GeoPolicy;
+import org.apache.rya.indexing.GeoTemporalIndexer.TemporalPolicy;
+import org.apache.rya.indexing.model.GeoTime;
+import org.apache.rya.mongodb.MongoRyaTestBase;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+import org.openrdf.model.Resource;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ContextStatementImpl;
+import org.openrdf.model.impl.StatementImpl;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.helpers.StatementPatternCollector;
+import org.openrdf.query.parser.sparql.SPARQLParser;
+
+import com.vividsolutions.jts.geom.Coordinate;
+import com.vividsolutions.jts.geom.GeometryFactory;
+import com.vividsolutions.jts.geom.LineString;
+import com.vividsolutions.jts.geom.LinearRing;
+import com.vividsolutions.jts.geom.Point;
+import com.vividsolutions.jts.geom.Polygon;
+import com.vividsolutions.jts.geom.PrecisionModel;
+import com.vividsolutions.jts.geom.impl.PackedCoordinateSequence;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * JUnit tests for TemporalIndexer and it's implementation
MongoTemporalIndexer
+ *
+ * If you enjoy this test, please read RyaTemporalIndexerTest and
YagoKBTest, which contain
+ * many example SPARQL queries and updates and attempts to test
independently of Mongo:
+ *
+ *
extras/indexingSail/src/test/java/org.apache/rya/indexing/Mongo/RyaTemporalIndexerTest.java
+ * {@link org.apache.rya.indexing.Mongo.RyaTemporalIndexerTest}
+ * {@link org.apache.rya.indexing.Mongo.YagoKBTest.java}
+ *
+ * Remember, this class in instantiated fresh for each @test method.
+ * so fields are reset, unless they are static.
+ *
+ * These are covered:
+ * Instance {before, equals, after} given Instance
--- End diff --
nope. I don't have a preference. I was just making sure you were aware.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #138: [WIP] Rya 236 - GeoTemporal Indexing
Github user amihalik commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/138#discussion_r96703892 --- Diff: extras/rya.geotemporalindexing/src/test/java/org/apache/rya/indexing/GeoTemporalIndexerTest.java --- @@ -0,0 +1,945 @@ +package org.apache.rya.indexing; + --- End diff -- ð ð ð Other places as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
