[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Puja Valiyil (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115209#comment-16115209
 ] 

Puja Valiyil  commented on RYA-316:
---

It wouldn't be a hash index-- you would just store the hash as if it was the 
original literal.  Basically encoding it.  
Good point about the geo search-- that was a lot of my concern with the 
truncating, so I'm ok with it.  

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Andrew Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115206#comment-16115206
 ] 

Andrew Smith commented on RYA-316:
--

hash indices can't be used in compound indices,
geo searches are performed on the geo field, not the object field.


> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Puja Valiyil (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115198#comment-16115198
 ] 

Puja Valiyil  commented on RYA-316:
---

Since you're also storing the object value still (just basing index values for 
queries off the sha1 hash), you don't have to worry about going back and forth 
from the hash.  So if you had the triple pattern (?s ?pred obj), you would 
query for triples with a hash value field equal to the has of obj.  Compound 
indices would use the hash obj instead of the obj value.  The scan would return 
the entire document, so you wouldn't need to convert back from the hash.  We 
would be storing the object twice, but that's not that big a deal.
As I said before, truncating isn't valid especially in a geo use case- you 
would no longer be storing the entire geo literal which makes it impossible to 
accurately do geo search functions.  I'm not sure how we would query on that 
either-- if you have a long literal do you break it up in the query?  You'd be 
doing a lot of filtering client side.  

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Andrew Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115146#comment-16115146
 ] 

Andrew Smith commented on RYA-316:
--

right, the truncating would end up with:
Subject=Andrew
Predicate=plays
Object=trumpet_sup-
Object_suffix=er long object value
Object type=string

so when you go to retrieve it, the value just gets concatenated.  This should 
be fine since we won't index on the suffix so it can be 16mb long  

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Issue Comment Deleted] (RYA-316) Long LineStrings break MongoDB ingest

[Andrew Smith (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Smith updated RYA-316:
-
Comment: was deleted

(was: I u may have found another solution, if I specify the PO index to be 
a compound index where OBJECT is a Text field, which it is treated as such, it 
works.  I just tried it and no problems
)

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Andrew Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115113#comment-16115113
 ] 

Andrew Smith commented on RYA-316:
--

I u may have found another solution, if I specify the PO index to be a 
compound index where OBJECT is a Text field, which it is treated as such, it 
works.  I just tried it and no problems


> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Andrew Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115102#comment-16115102
 ] 

Andrew Smith commented on RYA-316:
--

the problem is we would have to get rid of all compound indexes that use 
object.  We could compound index over the hashObject field, but I'm not sure 
how that'll work, we would also have to convert to/from sha of the object 
before/after each query and at insert time.  The LOE for truncating and sha-ing 
is the same, so I'm almost done with truncating.  If we decide to switch to SHA 
it'll be trivial


> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RYA-316) Long LineStrings break MongoDB ingest

[Puja Valiyil (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115095#comment-16115095
 ] 

Puja Valiyil  edited comment on RYA-316 at 8/4/17 11:30 PM:


I had thought we would add a hash of the object to each triple document and 
then use that for query/etc.  so for the triple (Andrew, plays, trumpet) we 
would store the following:
Subject=Andrew
Predicate=plays
Object=trumpet
Hashobject=sha 1 hash of trumpet
Object type=string
The Id would be the Concat of subject, predicate, and hash of object
Instead of the po index or so index you would have the hash object be the value 
indexed.  
There's a concern about losing alphabetic sorting for the po index but I'm not 
sure we even make use of that in a way that matters.


was (Author: pujav65):
I had thought we would add a hash of the object to each triple document and 
then use that for query/etc.  so for the triple (Andrew, plays, trumpet) we 
would store the following:
Subject=Andrew
Predicate=plays
Object=trumpet
Hashobject=sha 1 hash of trumpet
Object type=string
The Id would be the Concat of subject, predicate, and hash of object
There's a concern about losing alphabetic sorting for the po index but I'm not 
sure we even make use of that in a way that matters.

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Puja Valiyil (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115100#comment-16115100
 ] 

Puja Valiyil  commented on RYA-316:
---

Oh and I don't think truncating is a valid solution-- you lose portions of very 
long strings.  It would probably be better to not store the triple and instead 
detect and log the error at that point.  

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Puja Valiyil (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115095#comment-16115095
 ] 

Puja Valiyil  commented on RYA-316:
---

I had thought we would add a hash of the object to each triple document and 
then use that for query/etc.  so for the triple (Andrew, plays, trumpet) we 
would store the following:
Subject=Andrew
Predicate=plays
Object=trumpet
Hashobject=sha 1 hash of trumpet
Object type=string
The Id would be the Concat of subject, predicate, and hash of object
There's a concern about losing alphabetic sorting for the po index but I'm not 
sure we even make use of that in a way that matters.

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-316) Long LineStrings break MongoDB ingest

[Andrew Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115032#comment-16115032
 ] 

Andrew Smith commented on RYA-316:
--

Quick fix solution is to truncate the object value to 1024 so it fits in the 
index key, alternatively we can create multiple indexers and gracefully catch 
invalid index keys.  I'll implement the truncate since that's what I talked 
about with Caleb, but I'll look into doing this gracefully with hashing as well 
as an alternative

> Long LineStrings break MongoDB ingest
> -
>
> Key: RYA-316
> URL: https://issues.apache.org/jira/browse/RYA-316
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> MongoDB will reject statements they contain very long linestrings.  
> Basically, the mongodb index key is limited to 1024 chars, so the insert will 
> fail if the literal is longer.
> [Here is some example 
> code|https://github.com/amihalik/rya-mongo-debugging/blob/master/src/main/java/com/github/amihalik/rya/mongo/debugging/linestring/LoadLineString.java].
>   I think the inserts will work if you use 10 points, but fail if you use 
> linestrings with 100 points.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-313) Rya Mongo Blows up on Large result sets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114998#comment-16114998
 ] 

ASF GitHub Bot commented on RYA-313:


Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/196#discussion_r131490821
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -118,7 +124,8 @@ public MongoDBRdfConfiguration getConf() {
 }
 
 // TODO not sure what to do about regex ranges?
-final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(coll, rangeMap, strategy, 
conf.getAuthorizations());
+final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(
+getCollection(conf), rangeMap, strategy, 
conf.getAuthorizations());
--- End diff --

i think other parts of this class use the legacy DBCollection coll still


> Rya Mongo Blows up on Large result sets
> ---
>
> Key: RYA-313
> URL: https://issues.apache.org/jira/browse/RYA-313
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Affects Versions: 3.2.10
> Environment: Mongo DB with Rya 3.2.11-SNAPSHOT with a lot of data in 
> Rya
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> Simple queries that return a lot of results fail because mongo is trying to 
> send all of the results back at once.  For instance, if I have a lot of data 
> and run something like:
> {noformat}
> SELECT * WHERE 
> {
>   ?s a ?t.
> }
> {noformat}
> I will get this exception.
> {noformat}
> Caused by: com.mongodb.MongoCommandException: Command failed with error 
> 16389: 'aggregation result exceeds maximum document size (16MB)' on server 
> localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation 
> result exceeds maximum document size (16MB)", "code" : 16389 }
> {noformat}
> I think we need to toss in a "AggregationOptions with Batch = 1000", but I 
> couldn't get that to work immediately.  Somebody with more mongo experience 
> needs to look at this.
> [Here is the line of 
> code|https://github.com/apache/incubator-rya/blob/master/dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/iter/RyaStatementBindingSetCursorIterator.java#L114]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya pull request #196: RYA-313

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/196#discussion_r131490821
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -118,7 +124,8 @@ public MongoDBRdfConfiguration getConf() {
 }
 
 // TODO not sure what to do about regex ranges?
-final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(coll, rangeMap, strategy, 
conf.getAuthorizations());
+final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(
+getCollection(conf), rangeMap, strategy, 
conf.getAuthorizations());
--- End diff --

i think other parts of this class use the legacy DBCollection coll still


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-306) remove generated source from git

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114996#comment-16114996
 ] 

ASF GitHub Bot commented on RYA-306:


Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/191
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/359/



> remove generated source from git
> 
>
> Key: RYA-306
> URL: https://issues.apache.org/jira/browse/RYA-306
> Project: Rya
>  Issue Type: Bug
>  Components: build
>Reporter: Jeff Dasch
>Assignee: Eric White
>Priority: Trivial
>
> Several files in {{extras/rya.benchmark/src/main/gen}} get modified at 
> compile time.  They should be removed from git and configured to be created 
> in {{extras/rya.benchmark/target/generated-sources}} so they can be removed 
> by {{mvn clean}}.
> Files in question:
> {noformat}
>   modified:   extras/rya.benchmark/src/main/gen/META-INF/sun-jaxb.episode
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/ObjectFactory.java
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/Parameters.java
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/QueriesBenchmarkConf.java
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/Rya.java
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya issue #196: RYA-313

[asfgit]

Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/196
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/358/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-306) remove generated source from git

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114861#comment-16114861
 ] 

ASF GitHub Bot commented on RYA-306:


Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/191
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/357/Build
 result: FAILURE[...truncated 5.53 MB...][INFO] Apache Rya Spark 
Support ... SKIPPED[INFO] Apache Rya Web Projects 
 SKIPPED[INFO] Apache Rya Web Implementation 
.. SKIPPED[INFO] 
[INFO] 
BUILD FAILURE[INFO] 
[INFO] 
Total time: 24:17 min[INFO] Finished at: 2017-08-04T19:23:25+00:00[INFO] Final 
Memory: 419M/3343M[INFO] 
[ERROR] 
Failed to execute goal com.mycila:license-maven-plugin:2.6:format (default) on 
project rya.export.api: Resource 
${rya.project.basedir}/resources/LICENSE_HEADER.txt not found in file system, 
classpath or URL: no protocol: 
${rya.project.basedir}/resources/LICENSE_HEADER.txt -> [Help 1][ERROR] [ERROR] 
To see the full stack trace of the errors, re-run Maven with the -e 
switch.[ERROR] Re-run Maven using the -X switch to enable full debug 
logging.[ERROR] [ERROR] For more information about the errors and possible 
solutions, please read the following articles:[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the 
command[ERROR]   mvn  -rf :rya.export.apichannel stoppedSetting status 
of 69dc245a1fd7b6be21d2309458bf1c7042019dde to FAILURE with url 
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/357/
 and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing



> remove generated source from git
> 
>
> Key: RYA-306
> URL: https://issues.apache.org/jira/browse/RYA-306
> Project: Rya
>  Issue Type: Bug
>  Components: build
>Reporter: Jeff Dasch
>Assignee: Eric White
>Priority: Trivial
>
> Several files in {{extras/rya.benchmark/src/main/gen}} get modified at 
> compile time.  They should be removed from git and configured to be created 
> in {{extras/rya.benchmark/target/generated-sources}} so they can be removed 
> by {{mvn clean}}.
> Files in question:
> {noformat}
>   modified:   extras/rya.benchmark/src/main/gen/META-INF/sun-jaxb.episode
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/ObjectFactory.java
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/Parameters.java
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/QueriesBenchmarkConf.java
>   modified:   
> extras/rya.benchmark/src/main/gen/org/apache/rya/benchmark/query/Rya.java
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya issue #191: RYA-306 Remove rya.benchmark committed gen

[asfgit]

Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/191
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/357/Build
 result: FAILURE[...truncated 5.53 MB...][INFO] Apache Rya Spark 
Support ... SKIPPED[INFO] Apache Rya Web Projects 
 SKIPPED[INFO] Apache Rya Web Implementation 
.. SKIPPED[INFO] 
[INFO] 
BUILD FAILURE[INFO] 
[INFO] 
Total time: 24:17 min[INFO] Finished at: 2017-08-04T19:23:25+00:00[INFO] Final 
Memory: 419M/3343M[INFO] 
[ERROR] 
Failed to execute goal com.mycila:license-maven-plugin:2.6:format (default) on 
project rya.export.api: Resource 
${rya.project.basedir}/resources/LICENSE_HEADER.txt not found in file system, 
classpath or URL: no protocol: 
${rya.project.basedir}/resources/LICENSE_HEADER.txt -> [H
 elp 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven 
with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full 
debug logging.[ERROR] [ERROR] For more information about the errors and 
possible solutions, please read the following articles:[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the 
command[ERROR]   mvn  -rf :rya.export.apichannel stoppedSetting status 
of 69dc245a1fd7b6be21d2309458bf1c7042019dde to FAILURE with url 
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/357/
 and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-313) Rya Mongo Blows up on Large result sets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114846#comment-16114846
 ] 

ASF GitHub Bot commented on RYA-313:


Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/196#discussion_r131464691
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -118,7 +124,8 @@ public MongoDBRdfConfiguration getConf() {
 }
 
 // TODO not sure what to do about regex ranges?
-final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(coll, rangeMap, strategy, 
conf.getAuthorizations());
+final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(
+getCollection(conf), rangeMap, strategy, 
conf.getAuthorizations());
--- End diff --

Since you call getCollection(conf), I believe that coll is now an unused 
field.  Can you remove it?


> Rya Mongo Blows up on Large result sets
> ---
>
> Key: RYA-313
> URL: https://issues.apache.org/jira/browse/RYA-313
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Affects Versions: 3.2.10
> Environment: Mongo DB with Rya 3.2.11-SNAPSHOT with a lot of data in 
> Rya
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> Simple queries that return a lot of results fail because mongo is trying to 
> send all of the results back at once.  For instance, if I have a lot of data 
> and run something like:
> {noformat}
> SELECT * WHERE 
> {
>   ?s a ?t.
> }
> {noformat}
> I will get this exception.
> {noformat}
> Caused by: com.mongodb.MongoCommandException: Command failed with error 
> 16389: 'aggregation result exceeds maximum document size (16MB)' on server 
> localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation 
> result exceeds maximum document size (16MB)", "code" : 16389 }
> {noformat}
> I think we need to toss in a "AggregationOptions with Batch = 1000", but I 
> couldn't get that to work immediately.  Somebody with more mongo experience 
> needs to look at this.
> [Here is the line of 
> code|https://github.com/apache/incubator-rya/blob/master/dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/iter/RyaStatementBindingSetCursorIterator.java#L114]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya pull request #196: RYA-313

[amihalik]

Github user amihalik commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/196#discussion_r131464691
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBQueryEngine.java ---
@@ -118,7 +124,8 @@ public MongoDBRdfConfiguration getConf() {
 }
 
 // TODO not sure what to do about regex ranges?
-final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(coll, rangeMap, strategy, 
conf.getAuthorizations());
+final RyaStatementBindingSetCursorIterator iterator = new 
RyaStatementBindingSetCursorIterator(
+getCollection(conf), rangeMap, strategy, 
conf.getAuthorizations());
--- End diff --

Since you call getCollection(conf), I believe that coll is now an unused 
field.  Can you remove it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-295) Implement owl:allValuesFrom inference

[Jesse Hatfield (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114840#comment-16114840
 ] 

Jesse Hatfield commented on RYA-295:


Alternatively, we could use FixedStatementPattern to enumerate 
predicate/restriction pairs rather than nested unions. Something like:
{noformat}
Join(FSP, Join(StatementPattern(?subject, ?property, ?object), 
StatementPattern(?subject, rdf:type ?restriction)))
{noformat}
where FSP provides bindings for {{?property}} and {{?restriction}}:
{noformat}
FixedStatementPattern(?restriction, owl:onProperty, ?property):
(:r1, onProperty, :p1)
(:r2, onProperty, :p2)
...
(:rN, onProperty, :pN)
{noformat}
In this case, the number of nodes in the query tree is bounded; additional 
restrictions implying the same type are represented as statements in the 
FixedStatementPattern. One top-level Union is still needed to include the 
original statement pattern.

> Implement owl:allValuesFrom inference
> -
>
> Key: RYA-295
> URL: https://issues.apache.org/jira/browse/RYA-295
> Project: Rya
>  Issue Type: Sub-task
>  Components: sail
>Reporter: Jesse Hatfield
>Assignee: Jesse Hatfield
>
> An *{{owl:allValuesFrom}}* restriction defines the set of resources for 
> which, given a particular predicate and other type, every value of that 
> predicate is a member of that type. Note that there may be no values at all.
> For example, the ontology may state that resources of type {{:Person}} have 
> all values from {{:Person}} for type {{:parent}}: that is, a person's parents 
> are all people as well. Therefore, a pattern of the form {{?x rdf:type 
> :Person}} should be expanded to:
> {noformat}
> { ?y rdf:type :Person .
>   ?y :parent ?x }
> UNION
> { ?x rdf:type :Person }
> {noformat}
> i.e. we can infer {{?x}}'s personhood from the fact that child {{?y}} is 
> known to satisfy the restriction.
> Notes:
> -We can infer "x is a person, therefore all of x's parents are people". But 
> we can't infer "all of x's parents are people, therefore x is a person", 
> because of the open world semantics: we don't know that the parents given by 
> the data are in fact all of x's parents. (If there were also a cardinality 
> restriction and we could presume consistency, then we could infer this in the 
> right circumstances, but this is outside the scope of basic allValuesFrom 
> support.) This differs with most other property restriction rules in that we 
> can't infer that an object belongs to the class defined by the restriction, 
> but rather use the fact that an object is already known to belong in that 
> class in order to infer something about its neighbors in the graph (the types 
> of the values).
> -The example above could be applied recursively, but to implement this as a 
> simple query rewrite we'll need to limit recursion depth (and interactions 
> with other rules, for the same reasons).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-294) Implement owl:someValuesFrom inference

[Jesse Hatfield (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114839#comment-16114839
 ] 

Jesse Hatfield commented on RYA-294:


Alternatively, we could use FixedStatementPattern to enumerate 
predicate/restriction pairs rather than nested unions. Something like:
{noformat}
Join(FSP, Join(StatementPattern(?subject, ?property, ?object), 
StatementPattern(?object, rdf:type ?restriction)))
{noformat}
where FSP provides bindings for {{?property}} and {{?restriction}}:
{noformat}
FixedStatementPattern(?restriction, owl:onProperty, ?property):
(:r1, onProperty, :p1)
(:r2, onProperty, :p2)
...
(:rN, onProperty, :pN)
{noformat}
In this case, the number of nodes in the query tree is bounded; additional 
restrictions implying the same type are represented as statements in the 
FixedStatementPattern. One top-level Union is still needed to include the 
original statement pattern.

> Implement owl:someValuesFrom inference
> --
>
> Key: RYA-294
> URL: https://issues.apache.org/jira/browse/RYA-294
> Project: Rya
>  Issue Type: Sub-task
>  Components: sail
>Reporter: Jesse Hatfield
>
> An *{{owl:someValuesFrom}}* restriction defines the set of resources which, 
> for a given predicate and other type, have at least one value of that other 
> type for that predicate.
> If the ontology states that {{:Chair}} is the set of resources who are 
> {{:headOf}} at least one resource which must itself be a {{:Department}}, 
> then:
> {{?x rdf:type :Chair}}
> should be expanded to:
> {noformat}
> { ?x :headOf ?dept .
>   ?dept rdf:type :Department  }
> UNION
> {?x rdf:type :Chair }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-250) Smart URI avoid data duplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114788#comment-16114788
 ] 

ASF GitHub Bot commented on RYA-250:


Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/153
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/356/Failed
 Tests: 3incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.prospector:
 3org.apache.rya.prospector.mr.ProspectorTest.testCountorg.apache.rya.prospector.service.ProspectorServiceEvalStatsDAOTest.testCountorg.apache.rya.prospector.service.ProspectorServiceEvalStatsDAOTest.testNoAuthsCount



> Smart URI avoid data duplication
> 
>
> Key: RYA-250
> URL: https://issues.apache.org/jira/browse/RYA-250
> Project: Rya
>  Issue Type: Task
>  Components: dao
>Affects Versions: 3.2.10
>Reporter: Eric White
>Assignee: Eric White
> Fix For: 3.2.10
>
>
> Implement Smart URI methods for avoiding data duplication.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya issue #153: RYA-250 Smart URI avoiding data duplication

[asfgit]

Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/153
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/356/Failed
 Tests: 3incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.prospector:
 3org.apache.rya.prospector.mr.ProspectorTest.testCountorg.apache.rya.prospector.service.ProspectorServiceEvalStatsDAOTest.testCountorg.apache.rya.prospector.service.ProspectorServiceEvalStatsDAOTest.testNoAuthsCount



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (RYA-295) Implement owl:allValuesFrom inference

[Jesse Hatfield (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RYA-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Hatfield reassigned RYA-295:
--

Assignee: Jesse Hatfield

> Implement owl:allValuesFrom inference
> -
>
> Key: RYA-295
> URL: https://issues.apache.org/jira/browse/RYA-295
> Project: Rya
>  Issue Type: Sub-task
>  Components: sail
>Reporter: Jesse Hatfield
>Assignee: Jesse Hatfield
>
> An *{{owl:allValuesFrom}}* restriction defines the set of resources for 
> which, given a particular predicate and other type, every value of that 
> predicate is a member of that type. Note that there may be no values at all.
> For example, the ontology may state that resources of type {{:Person}} have 
> all values from {{:Person}} for type {{:parent}}: that is, a person's parents 
> are all people as well. Therefore, a pattern of the form {{?x rdf:type 
> :Person}} should be expanded to:
> {noformat}
> { ?y rdf:type :Person .
>   ?y :parent ?x }
> UNION
> { ?x rdf:type :Person }
> {noformat}
> i.e. we can infer {{?x}}'s personhood from the fact that child {{?y}} is 
> known to satisfy the restriction.
> Notes:
> -We can infer "x is a person, therefore all of x's parents are people". But 
> we can't infer "all of x's parents are people, therefore x is a person", 
> because of the open world semantics: we don't know that the parents given by 
> the data are in fact all of x's parents. (If there were also a cardinality 
> restriction and we could presume consistency, then we could infer this in the 
> right circumstances, but this is outside the scope of basic allValuesFrom 
> support.) This differs with most other property restriction rules in that we 
> can't infer that an object belongs to the class defined by the restriction, 
> but rather use the fact that an object is already known to belong in that 
> class in order to infer something about its neighbors in the graph (the types 
> of the values).
> -The example above could be applied recursively, but to implement this as a 
> simple query rewrite we'll need to limit recursion depth (and interactions 
> with other rules, for the same reasons).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RYA-336) Hyphenate artifactIds

[Jeff Dasch (JIRA)]

Jeff Dasch created RYA-336:
--

 Summary: Hyphenate artifactIds
 Key: RYA-336
 URL: https://issues.apache.org/jira/browse/RYA-336
 Project: Rya
  Issue Type: Improvement
  Components: build
Affects Versions: 3.2.11
Reporter: Jeff Dasch
Priority: Trivial
 Fix For: 4.0.0


Change project artifactIds to follow the typical naming convention.
For example:  {{rya-api}} instead of {{rya.api}}.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (RYA-325) Refactoring: rename rya.console to rya.shell

[Jeff Dasch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RYA-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Dasch resolved RYA-325.

Resolution: Done

> Refactoring: rename rya.console to rya.shell
> 
>
> Key: RYA-325
> URL: https://issues.apache.org/jira/browse/RYA-325
> Project: Rya
>  Issue Type: Sub-task
>  Components: clients
>Affects Versions: 3.2.10
>Reporter: Jeff Dasch
>Assignee: Jeff Dasch
> Fix For: 3.2.11
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-298) Implement rdfs:domain inference

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114707#comment-16114707
 ] 

ASF GitHub Bot commented on RYA-298:


GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/197

RYA-298, RYA-299 Domain/range inference.


## Description
InferenceEngine: Added a method to query for and infer domain/range schema, 
incorporating subclass, subproperty, and inverse property information to 
compute the closure. Extended graph traversal methods to allow traversal in 
each direction (findParents has the same behavior as before, but is joined by 
findChildren, both of which call findConnections with a direction parameter).

DomainRangeVisitor: New class rewrites queries for members of a specific 
defined type to check for domain and range implications as well. Applies to 
StatementPatterns of the form , where Class is not a variable, and 
produces (if domain and/or range exist for that type) a union which includes 
the original pattern as well.

RdfCloudTripleStoreConnection: Call the DomainRangeVisitor if inference is 
enabled. Called at the beginning of the sequence of visitors because the 
original statement is preserved as one branch of the union, so this won't stop 
other inference rules from being applied in that branch. However, as with the 
other visitors, the inferred branches of the unions won't be expanded. This 
means we won't infer the type by applying multiple rules at once (regardless of 
order of visitor calls), but we will have a union of alternative ways of 
inferring the type.

### Tests
Added tests to: Verify that InferenceEngine loads and infers expected 
schema (taking into account class hierarchy, property graph, and inverse 
properties); verify that the visitor produces the expected query tree; and 
verify that a type query returns the expected results given an ontology.

### Links
[RYA-298](https://issues.apache.org/jira/browse/RYA-298)
[RYA-299](https://issues.apache.org/jira/browse/RYA-299)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb 
@ejwhite922 
@pujav65
@amihalik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-298-299-domain-range-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #197


commit 36d8f752da4a3c5e8d7ebbbcb790ec0117c790ea
Author: Jesse Hatfield 
Date:   2017-08-04T17:05:45Z

RYA-298, RYA-299 Domain/range inference.




> Implement rdfs:domain inference
> ---
>
> Key: RYA-298
> URL: https://issues.apache.org/jira/browse/RYA-298
> Project: Rya
>  Issue Type: Sub-task
>  Components: sail
>Reporter: Jesse Hatfield
>Assignee: Jesse Hatfield
>
> If a predicate has an *{{rdfs:domain}}* of some class, than the subject of 
> any triple including that predicate belongs to the class.
> If the ontology states that {{:advisor}} has the domain of {{:Person}}, then 
> the inference engine should rewrite queries of the form {{?x rdf:type 
> :Person}} to check for resources which have any {{:advisor}} (as well as any 
> specifically stated to have type {{:Person}} ).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya pull request #197: RYA-298, RYA-299 Domain/range inference.

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/197

RYA-298, RYA-299 Domain/range inference.


## Description
InferenceEngine: Added a method to query for and infer domain/range schema, 
incorporating subclass, subproperty, and inverse property information to 
compute the closure. Extended graph traversal methods to allow traversal in 
each direction (findParents has the same behavior as before, but is joined by 
findChildren, both of which call findConnections with a direction parameter).

DomainRangeVisitor: New class rewrites queries for members of a specific 
defined type to check for domain and range implications as well. Applies to 
StatementPatterns of the form , where Class is not a variable, and 
produces (if domain and/or range exist for that type) a union which includes 
the original pattern as well.

RdfCloudTripleStoreConnection: Call the DomainRangeVisitor if inference is 
enabled. Called at the beginning of the sequence of visitors because the 
original statement is preserved as one branch of the union, so this won't stop 
other inference rules from being applied in that branch. However, as with the 
other visitors, the inferred branches of the unions won't be expanded. This 
means we won't infer the type by applying multiple rules at once (regardless of 
order of visitor calls), but we will have a union of alternative ways of 
inferring the type.

### Tests
Added tests to: Verify that InferenceEngine loads and infers expected 
schema (taking into account class hierarchy, property graph, and inverse 
properties); verify that the visitor produces the expected query tree; and 
verify that a type query returns the expected results given an ontology.

### Links
[RYA-298](https://issues.apache.org/jira/browse/RYA-298)
[RYA-299](https://issues.apache.org/jira/browse/RYA-299)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb 
@ejwhite922 
@pujav65
@amihalik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-298-299-domain-range-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #197


commit 36d8f752da4a3c5e8d7ebbbcb790ec0117c790ea
Author: Jesse Hatfield 
Date:   2017-08-04T17:05:45Z

RYA-298, RYA-299 Domain/range inference.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-313) Rya Mongo Blows up on Large result sets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114658#comment-16114658
 ] 

ASF GitHub Bot commented on RYA-313:


Github user amihalik commented on the issue:

https://github.com/apache/incubator-rya/pull/196
  
Looks good @isper3at.  I'm going to test this out and wait for one more 
before I merge.


> Rya Mongo Blows up on Large result sets
> ---
>
> Key: RYA-313
> URL: https://issues.apache.org/jira/browse/RYA-313
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Affects Versions: 3.2.10
> Environment: Mongo DB with Rya 3.2.11-SNAPSHOT with a lot of data in 
> Rya
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> Simple queries that return a lot of results fail because mongo is trying to 
> send all of the results back at once.  For instance, if I have a lot of data 
> and run something like:
> {noformat}
> SELECT * WHERE 
> {
>   ?s a ?t.
> }
> {noformat}
> I will get this exception.
> {noformat}
> Caused by: com.mongodb.MongoCommandException: Command failed with error 
> 16389: 'aggregation result exceeds maximum document size (16MB)' on server 
> localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation 
> result exceeds maximum document size (16MB)", "code" : 16389 }
> {noformat}
> I think we need to toss in a "AggregationOptions with Batch = 1000", but I 
> couldn't get that to work immediately.  Somebody with more mongo experience 
> needs to look at this.
> [Here is the line of 
> code|https://github.com/apache/incubator-rya/blob/master/dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/iter/RyaStatementBindingSetCursorIterator.java#L114]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya issue #196: RYA-313

[amihalik]

Github user amihalik commented on the issue:

https://github.com/apache/incubator-rya/pull/196
  
Looks good @isper3at.  I'm going to test this out and wait for one more 
before I merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (RYA-313) Rya Mongo Blows up on Large result sets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114656#comment-16114656
 ] 

ASF GitHub Bot commented on RYA-313:


Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/196
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/354/



> Rya Mongo Blows up on Large result sets
> ---
>
> Key: RYA-313
> URL: https://issues.apache.org/jira/browse/RYA-313
> Project: Rya
>  Issue Type: Bug
>  Components: dao
>Affects Versions: 3.2.10
> Environment: Mongo DB with Rya 3.2.11-SNAPSHOT with a lot of data in 
> Rya
>Reporter: Aaron Mihalik
>Assignee: Andrew Smith
>
> Simple queries that return a lot of results fail because mongo is trying to 
> send all of the results back at once.  For instance, if I have a lot of data 
> and run something like:
> {noformat}
> SELECT * WHERE 
> {
>   ?s a ?t.
> }
> {noformat}
> I will get this exception.
> {noformat}
> Caused by: com.mongodb.MongoCommandException: Command failed with error 
> 16389: 'aggregation result exceeds maximum document size (16MB)' on server 
> localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation 
> result exceeds maximum document size (16MB)", "code" : 16389 }
> {noformat}
> I think we need to toss in a "AggregationOptions with Batch = 1000", but I 
> couldn't get that to work immediately.  Somebody with more mongo experience 
> needs to look at this.
> [Here is the line of 
> code|https://github.com/apache/incubator-rya/blob/master/dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/iter/RyaStatementBindingSetCursorIterator.java#L114]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya issue #196: RYA-313

[asfgit]

Github user asfgit commented on the issue:

https://github.com/apache/incubator-rya/pull/196
  

Refer to this link for build results (access rights to CI server needed): 

https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/354/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (RYA-311) Nearby GeoSparql Function

[Aaron Mihalik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RYA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Mihalik closed RYA-311.
-
Resolution: Duplicate

> Nearby GeoSparql Function
> -
>
> Key: RYA-311
> URL: https://issues.apache.org/jira/browse/RYA-311
> Project: Rya
>  Issue Type: New Feature
>  Components: clients
>Affects Versions: 3.2.10
>Reporter: Caleb Meier
>Assignee: Aaron Mihalik
>
> Implement a custom NearBy Filter function for finding all points that occur 
> within a specified distance of a given point.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (RYA-311) Nearby GeoSparql Function

[Aaron Mihalik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RYA-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Mihalik reassigned RYA-311:
-

Assignee: Aaron Mihalik

> Nearby GeoSparql Function
> -
>
> Key: RYA-311
> URL: https://issues.apache.org/jira/browse/RYA-311
> Project: Rya
>  Issue Type: New Feature
>  Components: clients
>Affects Versions: 3.2.10
>Reporter: Caleb Meier
>Assignee: Aaron Mihalik
>
> Implement a custom NearBy Filter function for finding all points that occur 
> within a specified distance of a given point.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RYA-335) MongoDB DAO Batch Writer thread keeps App alive

[Aaron Mihalik (JIRA)]

Aaron Mihalik created RYA-335:
-

 Summary: MongoDB DAO Batch Writer thread keeps App alive
 Key: RYA-335
 URL: https://issues.apache.org/jira/browse/RYA-335
 Project: Rya
  Issue Type: Bug
  Components: dao
Reporter: Aaron Mihalik
Assignee: Eric White


During testing of RYA-307, I notice that my simple java mains did not complete. 
 I had to manually kill the processes.  I'm guessing that the Batch Writer 
Threads in AbstractMongoIndexer are not being shutdown correctly.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RYA-325) Refactoring: rename rya.console to rya.shell

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/RYA-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114363#comment-16114363
 ] 

ASF GitHub Bot commented on RYA-325:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rya/pull/194


> Refactoring: rename rya.console to rya.shell
> 
>
> Key: RYA-325
> URL: https://issues.apache.org/jira/browse/RYA-325
> Project: Rya
>  Issue Type: Sub-task
>  Components: clients
>Affects Versions: 3.2.10
>Reporter: Jeff Dasch
>Assignee: Jeff Dasch
> Fix For: 3.2.11
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-rya pull request #194: RYA-325 Renamed rya.console to rya.shell.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rya/pull/194


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #195: Temp it fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rya/pull/195


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---