[GitHub] jena pull request #319: Jena 1441 tracing

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/319#discussion_r153822758
  
--- Diff: 
jena-text/src/main/java/org/apache/jena/query/text/TextIndexLucene.java ---
@@ -204,7 +204,10 @@ public void close() {
 
 @Override public void updateEntity(Entity entity) {
 if ( log.isDebugEnabled() )
-log.debug("Update entity: " + entity) ;
+if (log.isTraceEnabled() && entity != null)
--- End diff --

Better than checking `log.isDebugEnabled()` is to use the idiom 
`log.debug("Update entity: {}", entity))`, same with other log statements that 
have guards and are calling `toString` implicitly. Using it for `trace` here 
makes sense because you aren't just calling `toString` and sadly, SLF4J doesn't 
yet accept `Supplier`s for messages.


---

[GitHub] jena pull request #320: JENA-1438 rdf:langString args

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/320#discussion_r153824555
  
--- Diff: 
jena-text/src/main/java/org/apache/jena/query/text/TextQueryPF.java ---
@@ -295,6 +295,14 @@ private String chooseGraphURI(ExecutionContext 
execCxt) {
 return results;
 }
 
+private boolean noLang(String lang) {
+return (lang == null || lang.isEmpty() || " ".equals(lang));
--- End diff --

I don't know much about the text indexing, but this single space-- does 
that have some special meaning, or is it just a simple check for whitespace? 
I.e., could we see more than one space char here?


---

[GitHub] jena issue #319: Jena 1441 tracing

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/319
  
Logging statements look good to me!


---

[GitHub] jena pull request #319: Jena 1441 tracing

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/319#discussion_r153861908
  
--- Diff: 
jena-text/src/main/java/org/apache/jena/query/text/assembler/TextIndexLuceneAssembler.java
 ---
@@ -144,6 +144,17 @@ public TextIndex open(Assembler a, Resource root, Mode 
mode) {
 storeValues = svNode.asLiteral().getBoolean();
 }
 
+// use query cache by default
+boolean cacheQueries = true;
+Statement cacheQueriesStatement = 
root.getProperty(pCacheQueries);
+if (null != cacheQueriesStatement) {
+RDFNode cqNode = cacheQueriesStatement.getObject();
+if (! cqNode.isLiteral()) {
+throw new TextIndexException("text:cacheQueries 
property must be a string : " + cqNode);
--- End diff --

This is maybe a little picky, but maybe better: `"text:cacheQueries 
property must be a boolean`? 


---

[GitHub] jena pull request #321: Just doing some syntax cleanup by using the new Grap...

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/321

Just doing some syntax cleanup by using the new Graph::find method



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena FindANYtoFind

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/321.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #321






---

Fwd: [GitHub] jena pull request #321: Just doing some syntax cleanup by using the new Grap...

[ajs6f]

Andy-- this is so trivial I would have just committed it, but I didn't want to 
bump into anything you might be doing to prep for a release.

ajs6f

> Begin forwarded message:
> 
> From: ajs6f 
> Subject: [GitHub] jena pull request #321: Just doing some syntax cleanup by 
> using the new Grap...
> Date: November 29, 2017 at 12:55:26 PM EST
> To: dev@jena.apache.org
> Reply-To: dev@jena.apache.org
> Reply-To: dev@jena.apache.org
> 
> GitHub user ajs6f opened a pull request:
> 
>https://github.com/apache/jena/pull/321
> 
>Just doing some syntax cleanup by using the new Graph::find method
> 
> 
> 
> You can merge this pull request into a Git repository by running:
> 
>$ git pull https://github.com/ajs6f/jena FindANYtoFind
> 
> Alternatively you can review and apply these changes as the patch at:
> 
>https://github.com/apache/jena/pull/321.patch
> 
> To close this pull request, make a commit to your master/trunk branch
> with (at least) the following in the commit message:
> 
>This closes #321
> 
> 
> 
> 
> 
> 
> ---

[GitHub] jena issue #323: JENA-1427: ExtendedIterator.nextOptional

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/323
  
Do we want to return `empty` for `null`? I ask because the "ergonomics" are 
different. The way we have it now:
```
try {
iterator.nextOptional().ifPresent(this::doStuff);
}
catch (NullPointerException e) { worry about that; }
```
vs. just `iterator.nextOptional().ifPresent(this::doStuff);` where 
`doStuff` is responsible for testing for `null`. I actually like the latter 
because `null` might be a legitimate value, but I may be misunderstanding the 
contract of `ExtendedIterator`-- is it guaranteed that `next()` never returns 
`null`? Is `null` actually not ever legit?



---

[GitHub] jena pull request #320: JENA-1438 rdf:langString args

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/320#discussion_r154109219
  
--- Diff: 
jena-text/src/main/java/org/apache/jena/query/text/TextQueryPF.java ---
@@ -319,13 +311,13 @@ private StrMatch objectToStruct(PropFuncArg 
argObject, boolean executionTime) {
 
 String lang = o.getLiteralLanguage();
 RDFDatatype dt = o.getLiteralDatatype() ;
-if (noLang(lang)) {
+if (lang.isEmpty()) {
 if (dt != null && dt != XSDDatatype.XSDstring) {
 log.warn("Object to text query is not a string") ;
 return null ;
 }
 }
-lang = fixLang(lang);
+lang = lang.isEmpty() ? null : lang;
--- End diff --

I like 
[`emptyToNull`](https://google.github.io/guava/releases/19.0/api/docs/com/google/common/base/Strings.html#emptyToNull(java.lang.String))
 for this, but this isn't a problem or anything.


---

[GitHub] jena issue #323: JENA-1427: ExtendedIterator.nextOptional

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/323
  
I'm saying we don't do `orElseThrow` or any other `orElseX`, we leave that 
to `nextOptional().orElseThrow()`, etc. Then at iterator end, `nextOptional()` 
returns `empty` forever. If there are `null`s, it returns `empty`. If you want 
to know whether that is because there are many `null`s or because the iterator 
is done, you call `hasNext()`.

Another way to put it: we shouldn't try to eliminate needing `hasNext()` 
with this-- if we want to do that, we should do something like 
`Spliterator::tryAdvance`, which is a whole different story. You can se how 
that would work a little with `Iterator::forEachRemaining`. You don't need 
`hasNext()` with that.


---

[GitHub] jena issue #323: JENA-1427: ExtendedIterator.nextOptional

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/323
  
That's my point-- I don't think you can do a good fluent API for this 
without narrowing the `Iterator` contract to exclude `null`s, because you can't 
distinguish. We can do that (narrow the contact)-- we own `ExtendedIterator` 
and as you say, we aren't pushing `null`s through it for the `Model` API.

So my vote is that we do what you have here, but _also_ add a note to 
`ExtendedIterator` guaranteeing that `next()` must not return `null`.


---

Re: IteratorIterator deprecated

[ajs6f]

Claude-- have you looked at the Guava machinery underneath 
Iterators.concat(...)? I _believe_ that it is fully lazy, although I haven't 
looked at it in a while.

https://google.github.io/guava/releases/19.0/api/docs/com/google/common/collect/Iterators.html#concat(java.util.Iterator)
https://google.github.io/guava/releases/19.0/api/docs/com/google/common/collect/Iterators.html#concat(java.util.Iterator...)
etc.

ajs6f

> On Dec 2, 2017, at 9:42 AM, Claude Warren  wrote:
> 
> I know that the IteratorIterator class was deprecated some time ago in
> favor of WrappedIterator.createIteratorIterator(). However they have
> different performance characteristics.
> 
> the original IteratorIterator, did not call iter.next() on the base
> iterator until it was needed.  createIteratorIterator() creates the
> iterators using the (iter.next()).andThen( iter.next() ) type construct.
> Basically calling next() on the base iterator to create an iterator over
> the results.
> 
> The problem is that when the base iterator is returning expensive (or very
> memory hungry) iterators they are all loaded before the final iterator is
> ready for use.
> 
> I know that there is a lazy iterator but I don't see how to use that to
> solve the problem.
> 
> I propose we bring back the IteratorIterator and describe when to use it.
> 
> Claude
> 
> -- 
> I like: Like Like - The likeliest place on the web
> <http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren

[GitHub] jena issue #324: JENA-1443: Use DatasetGraph for GraphView TransactionHandle...

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/324
  
`GraphInMemory extends GraphView`, so this should apply automatically to 
TIM, right?


---

[GitHub] jena issue #324: JENA-1443: Use DatasetGraph for GraphView TransactionHandle...

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/324
  
Nice, thanks!


---

[GitHub] jena issue #323: JENA-1427: ExtendedIterator.nextOptional

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/323
  
Okay, that works for me!


---

Re: [jira] [Commented] (JENA-1427) Add nextOrElse() method in ExtendedIterator

[ajs6f]

I don't think nextOptional will ever return null, but I think you mean empty? 
If so, and IIUC, what you say was exactly my objection (see discussion in the 
PR) but as Andy pointed out there, the Model API doesn't ever actually return 
an ExtendedIterator with null values, so the method turns out to be usable in 
the only use case we have for it.

ajs6f

> On Dec 6, 2017, at 5:57 AM, Claude Warren  wrote:
> 
> my reading of this is that nextOptional will return null at the end of the
> iteration and that this is indistinguishable from a null in the iteration
> unless hasNext() is called.  but calling hasNext() after a null only tells
> you if the next item is present not if the last item was present, thus a
> null at the end of the iteration may be lost.  It almost seems like you
> need a lastRead() method to retrieve the last read object again.  It seems
> like this construct will only be useful when there  are no nulls in the
> iteration.
> 
> Claude
> 
> On Mon, Dec 4, 2017 at 3:59 PM, Andy Seaborne (JIRA) 
> wrote:
> 
>> 
>>[ https://issues.apache.org/jira/browse/JENA-1427?page=
>> com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel&focusedCommentId=16276983#comment-16276983 ]
>> 
>> Andy Seaborne commented on JENA-1427:
>> -
>> 
>> {{nextOptional}} added for release 3.6.0.
>> 
>> Proposal: close this JIRA for now, see how {{nextOptional}} works out and
>> revisit {{orElse*}} based on experience.
>> 
>>> Add nextOrElse() method in ExtendedIterator
>>> ---
>>> 
>>>Key: JENA-1427
>>>URL: https://issues.apache.org/jira/browse/JENA-1427
>>>Project: Apache Jena
>>> Issue Type: Improvement
>>> Components: Core
>>>   Affects Versions: Jena 3.5.0
>>>   Reporter: Adam Jacobs
>>>   Priority: Trivial
>>> Labels: easytask
>>> 
>>> Allow a functional approach for returning a default value or throwing a
>> custom exception from a Jena iterator.
>>> The following method may be added to the ExtendedIterator interface.
>>> {noformat}
>>>/**
>>> Answer the next object, if it exists, otherwise invoke the
>> _supplier_.
>>> */
>>>public default T nextOrElse( Supplier supplier ) {
>>>return hasNext() ? next() : supplier.get();
>>>}
>>> {noformat}
>> 
>> 
>> 
>> --
>> This message was sent by Atlassian JIRA
>> (v6.4.14#64029)
>> 
> 
> 
> 
> -- 
> I like: Like Like - The likeliest place on the web
> <http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren

Re: [jira] [Commented] (JENA-1427) Add nextOrElse() method in ExtendedIterator

[ajs6f]

If it turns out that nextOptional is problematic because of nulls in iterators, 
we could, at least in theory, change it to return null for null, empty for 
hasNext=false, and Optional.of(next()) for hasNext=true.

ajs6f

> On Dec 6, 2017, at 9:23 AM, Andy Seaborne  wrote:
> 
> Which is why I think we ought to provide "orElse*" directly.
> 
> As we have it currently, ajs6f's argument for only nextOptional() is to 
> provide "orElse" via Optional so the Optional must be the end of iteration.  
> Then it can't be for nulls during iteration (the stream case see [1]).
> 
> Claude - this is the javadoc: point 3 is that ExtendedIterators do not return 
> null for next() in this usage.
> 
> Do you have a suggestion for improving that?
> 
> /**
>Answer with an {@link Optional}.
>   This operation assumes that the {@code ExtendedIterator} does not return 
> null for {@code next()}.
>   If it does,  {@code NullPointerException} is thrown.
>   
>   If there is no next, return {@code Optional.empty()}
>   If the next object exists, and is not null, return that in the {@link 
> Optional}.
>   If the next object exists, and is null, throw {@code 
> NullPointerException}
>   
> */
> 
>Andy
> 
> http://mail.openjdk.java.net/pipermail/jdk8-dev/2013-September/003274.html
> 
> On 06/12/17 14:07, ajs6f wrote:
>> I don't think nextOptional will ever return null, but I think you mean 
>> empty? If so, and IIUC, what you say was exactly my objection (see 
>> discussion in the PR) but as Andy pointed out there, the Model API doesn't 
>> ever actually return an ExtendedIterator with null values, so the method 
>> turns out to be usable in the only use case we have for it.
>> ajs6f
>>> On Dec 6, 2017, at 5:57 AM, Claude Warren  wrote:
>>> 
>>> my reading of this is that nextOptional will return null at the end of the
>>> iteration and that this is indistinguishable from a null in the iteration
>>> unless hasNext() is called.  but calling hasNext() after a null only tells
>>> you if the next item is present not if the last item was present, thus a
>>> null at the end of the iteration may be lost.  It almost seems like you
>>> need a lastRead() method to retrieve the last read object again.  It seems
>>> like this construct will only be useful when there  are no nulls in the
>>> iteration.
>>> 
>>> Claude
>>> 
>>> On Mon, Dec 4, 2017 at 3:59 PM, Andy Seaborne (JIRA) 
>>> wrote:
>>> 
>>>> 
>>>>[ https://issues.apache.org/jira/browse/JENA-1427?page=
>>>> com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>>> tabpanel&focusedCommentId=16276983#comment-16276983 ]
>>>> 
>>>> Andy Seaborne commented on JENA-1427:
>>>> -
>>>> 
>>>> {{nextOptional}} added for release 3.6.0.
>>>> 
>>>> Proposal: close this JIRA for now, see how {{nextOptional}} works out and
>>>> revisit {{orElse*}} based on experience.
>>>> 
>>>>> Add nextOrElse() method in ExtendedIterator
>>>>> ---
>>>>> 
>>>>>Key: JENA-1427
>>>>>URL: https://issues.apache.org/jira/browse/JENA-1427
>>>>>Project: Apache Jena
>>>>> Issue Type: Improvement
>>>>> Components: Core
>>>>>   Affects Versions: Jena 3.5.0
>>>>>   Reporter: Adam Jacobs
>>>>>   Priority: Trivial
>>>>> Labels: easytask
>>>>> 
>>>>> Allow a functional approach for returning a default value or throwing a
>>>> custom exception from a Jena iterator.
>>>>> The following method may be added to the ExtendedIterator interface.
>>>>> {noformat}
>>>>>/**
>>>>> Answer the next object, if it exists, otherwise invoke the
>>>> _supplier_.
>>>>> */
>>>>>public default T nextOrElse( Supplier supplier ) {
>>>>>return hasNext() ? next() : supplier.get();
>>>>>}
>>>>> {noformat}
>>>> 
>>>> 
>>>> 
>>>> --
>>>> This message was sent by Atlassian JIRA
>>>> (v6.4.14#64029)
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> I like: Like Like - The likeliest place on the web
>>> <http://like-like.xenei.com>
>>> LinkedIn: http://www.linkedin.com/in/claudewarren

Re: [jira] [Resolved] (JENA-1427) Add nextOrElse() method in ExtendedIterator

[ajs6f]

> methods to convert the iterator to a stream should be sufficient.  And i 
> believe that has been implemented

I would say that actually having streams from the API would be sufficient, but 
we are a long way from that. (We should restart that thread about a potential 
new API with immutability and streams and possibly more.)

ajs6f

> On Dec 6, 2017, at 10:21 AM, Claude Warren  wrote:
> 
> Is optional.get() == null an expected or acceptable solution?  I supose it
> might be in this case.  However i am beginning to believe that streaming
> operations should not be applied to iterators and rather that methods to
> convert the iterator to a stream should be sufficient.  And i believe that
> has been implemented.
> 
> Claude
> 
> On 6 Dec 2017 15:38, "Andy Seaborne (JIRA)"  wrote:
> 
>> 
>>[ https://issues.apache.org/jira/browse/JENA-1427?page=
>> com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>> 
>> Andy Seaborne resolved JENA-1427.
>> -
>>  Resolution: Done
>>Assignee: Andy Seaborne
>>   Fix Version/s: Jena 3.6.0
>> 
>>> Add nextOrElse() method in ExtendedIterator
>>> ---
>>> 
>>>   Key: JENA-1427
>>>   URL: https://issues.apache.org/jira/browse/JENA-1427
>>>   Project: Apache Jena
>>>Issue Type: Improvement
>>>Components: Core
>>>  Affects Versions: Jena 3.5.0
>>>  Reporter: Adam Jacobs
>>>  Assignee: Andy Seaborne
>>>  Priority: Trivial
>>>Labels: easytask
>>>   Fix For: Jena 3.6.0
>>> 
>>> 
>>> Allow a functional approach for returning a default value or throwing a
>> custom exception from a Jena iterator.
>>> The following method may be added to the ExtendedIterator interface.
>>> {noformat}
>>>   /**
>>>Answer the next object, if it exists, otherwise invoke the
>> _supplier_.
>>>*/
>>>   public default T nextOrElse( Supplier supplier ) {
>>>   return hasNext() ? next() : supplier.get();
>>>   }
>>> {noformat}
>> 
>> 
>> 
>> --
>> This message was sent by Atlassian JIRA
>> (v6.4.14#64029)
>> 

Re: [jira] [Resolved] (JENA-1427) Add nextOrElse() method in ExtendedIterator

[ajs6f]

> You can call nextOptional() forever which is odd.

This is just as true of nextOrElse*, so they're not better in that regard.

I agree that the semantics don't really work here, and I think Claude is right 
to point at Streams as the place Optional really fits (e.g. Stream::findFirst).

ExtendedIterator (< Iterator) just isn't a Stream.

ajs6f

> On Dec 6, 2017, at 12:21 PM, Andy Seaborne  wrote:
> 
> What this all shows to me is that Optional for end of iterator is not what 
> Optional is for.
> 
> next() has two return paths: an object reference or NoSuchElementException. 
> Converting NSEE into Optional goes against the intent of Optional (see the EG 
> email).  The argument of reusing machinery would be OK if the usage pattern 
> was compatible.
> 
> You can call nextOptional() forever which is odd.
> 
> The nextOptional is looking like it does not stand up to scrutiny even if we 
> accept that ExtendedIterator does not yield nulls.
> 
> Which leaves nextOrElse*
> 
> On 06/12/17 15:21, Claude Warren wrote:
>> Is optional.get() == null an expected or acceptable solution?  I supose it
>> might be in this case.
> 
> Checking the code it looks like you can't create an optional with null in it. 
>  The JDK will throw NPE.
> 
>> However i am beginning to believe that streaming
>> operations should not be applied to iterators and rather that methods to
>> convert the iterator to a stream should be sufficient.  And i believe that
>> has been implemented.
>> Claude
>> On 6 Dec 2017 15:38, "Andy Seaborne (JIRA)"  wrote:
>>> 
>>>  [ https://issues.apache.org/jira/browse/JENA-1427?page=
>>> com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>>> 
>>> Andy Seaborne resolved JENA-1427.
>>> -
>>>Resolution: Done
>>>  Assignee: Andy Seaborne
>>> Fix Version/s: Jena 3.6.0
>>> 
>>>> Add nextOrElse() method in ExtendedIterator
>>>> ---
>>>> 
>>>> Key: JENA-1427
>>>> URL: https://issues.apache.org/jira/browse/JENA-1427
>>>> Project: Apache Jena
>>>>  Issue Type: Improvement
>>>>  Components: Core
>>>>Affects Versions: Jena 3.5.0
>>>>Reporter: Adam Jacobs
>>>>Assignee: Andy Seaborne
>>>>Priority: Trivial
>>>>  Labels: easytask
>>>> Fix For: Jena 3.6.0
>>>> 
>>>> 
>>>> Allow a functional approach for returning a default value or throwing a
>>> custom exception from a Jena iterator.
>>>> The following method may be added to the ExtendedIterator interface.
>>>> {noformat}
>>>> /**
>>>>  Answer the next object, if it exists, otherwise invoke the
>>> _supplier_.
>>>>  */
>>>> public default T nextOrElse( Supplier supplier ) {
>>>> return hasNext() ? next() : supplier.get();
>>>> }
>>>> {noformat}
>>> 
>>> 
>>> 
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v6.4.14#64029)
>>> 

[GitHub] jena issue #314: JENA-1430

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/314
  
Ouch! I realize I wasn't clear enough at all in my last comment: I was 
actually -1 on this. I think the confusion that `ja:data` now has in its 
semantics is too much, and I completely disagree with:
> We can't change the predicates due to existing deployed assemblers

We change the public API when we must, we just do it carefully and after 
clear depreciation with a clear migration path. What's more, the docs have been 
wrong about the assembler type for TIM for a year or more, so I don't think 
there will be users waiting with pitchforks and torches!

I'd like to either revert this merge or add a further commit adding the use 
of `ja:graph` to TIM's assembler (just like the general dataset assembler) and 
depreciating the use of `ja:data` for anything but quads with the expectation 
that we will disallow it in a near-future version. I can do that after a 
release, if that is easier.


---

[GitHub] jena issue #314: JENA-1430

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/314
  
I'm saying that `ja:data` is now used for loading quads into two different 
kinds of datasets, but loading triples into only one. That's confusing at best. 
I want to make it clearer by getting rid of the second use, because we already 
have `ja:graph`, and then what about `ja:content`? But I'm happy to hear other 
ways to clarify the semantics-- I just think that we've made things more 
powerful but more confusing with this PR. I want to keep the power (load quads 
easily, which we just did) while not making it harder to understand how to 
build assembler RDF (which I think we just did).

Re: deprecation: we can deprecate with comments and by throwing warnings 
from inside the assembler code. And again, we have good reason (the mistake in 
the docs that no one noticed enough to complain about) to think that it won't 
be too disruptive.


---

Re: Pooling Http Client lockup

[ajs6f]

Dave--

Jena switched HTTP Commons versions a good bit from 3.1.1 to now, so that may 
be part of it. I will look into whether HTTP Commons Client changed its 
behavior under us.

I agree that just upping the maxes and hoping for the best isn't the best 
outcome at all.

Otherwise, this sounds either buggy (if all the execs really are getting 
closed) or at least not very ergonomic (since you aren't seeing any warnings).

Do I understand correctly that the behavior you would want would be that

1) after the max number of connections have been drawn out of the pool and 
used, the next request should block only until a connection is released,
2) and that closing a query execution should definitely return the connection 
underneath it to the pool, more or less immediately?

It's almost like something above the client is blocking and not letting the 
connections get released...

ajs6f

> On Dec 8, 2017, at 12:55 PM, Dave Reynolds  wrote:
> 
> Hi Rob,
> 
> Thanks, that's useful to know but just raising the limit will just make it 
> less frequent but not cure it.
> 
> The issue is that if you ever get more in-flight than the limit then all 
> future requests are blocked. The in-flight requests return fine, the execs 
> are closed but the httpclient never recovers.
> 
> I will, of course, check again that I'm successfully closing all the execs. 
> However, with Jena 3.1.1 this code has successfully run for months between 
> reboots with request rates in the millions per week (with occasional high 
> bursts). No lock ups at all. As it stands I can't switch it to a newer Jena 
> unless I can absolutely guarantee an upper limit on the number of concurrent 
> requests. That is often possible (through an apache reverse proxy front end 
> with request throttling) but feels like an unsatisfying resolution.
> 
> Dave
> 
> On 08/12/17 16:56, Rob Vesse wrote:
>> I think this relates to a HttpClient behaviour that limits the maximum 
>> connections to a given service
>> At least in how Jena sets it up the default is 5 connections per service 
>> which is more generous than the HTTP client defaults.  Jena appears to read 
>> this from a JVM property http.maxConnections OR you can construct your own 
>> client by calling setMaxConnPerRoute() and setMaxConnTotal() on the builder 
>> to set your desired settings.
>> Rob
>> On 08/12/2017, 16:44, "Dave Reynolds"  wrote:
>> Hi,
>>  I've being updating some rather old libraries that use Jena to issue
>> sparql requests to a remote endpoint and pull back results.
>>  These work under Jena 3.1.1 but there's a fatal problem under Jena 
>> 3.2.0
>> and later ...
>>  If I issue 6 concurrent execSelect calls to a remote sparql endpoint
>> (happens to be fuseki) then 5 will get issued and return correctly but
>> #6 will not and from then onwards no further remote calls will go
>> through. This only happens if at least 5 requests are in flight with no
>> response yet from the remote endpoint before the final one is issued.
>>  It's hard to produce a deterministic, standalone test case because 
>> by
>> definition it depends on an external sparql endpoint being reliably
>> there and reliably not too fast :)
>>  Looking at the stack trace I see:
>>  Unsafe.park(boolean, long) line: not available [native method]
>> LockSupport.park(Object) line: 175
>> AbstractQueuedSynchronizer$ConditionObject.await() line: 2039
>> CPool(AbstractConnPool).getPoolEntryBlocking(T, Object, long,
>> TimeUnit, Future) line: 377
>> AbstractConnPool.access$200(AbstractConnPool, Object, Object,
>> long, TimeUnit, Future) line: 67
>> AbstractConnPool$2.get(long, TimeUnit) line: 243
>> AbstractConnPool$2.get(long, TimeUnit) line: 191
>> PoolingHttpClientConnectionManager.leaseConnection(Future,
>> long, TimeUnit) line: 282
>> PoolingHttpClientConnectionManager$1.get(long, TimeUnit) line: 269
>> MainClientExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
>> HttpExecutionAware) line: 191
>> ProtocolExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
>> HttpExecutionAware) line: 185
>> RetryExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
>> HttpExecutionAware) line: 89
>> RedirectExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext,
>> HttpExecutionAware) line: 111
>> InternalHttpClient.doExecute(HttpHost, HttpRequest, HttpContext) line: 
>> 185
>> InternalHttpClient(CloseableHttpClient).execute(HttpUriR

[GitHub] jena issue #314: JENA-1430

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/314
  
1. The two different datasets at TIM and general. The fact that the code 
for the general dataset assembler doesn't mention `ja:data` is fine, but 
because TIM's assembler does understand `ja:data` and some assembler RDF with 
the general dataset type _can_ get shunted to TIM, `ja:data` can end up being 
successfully used with well-defined semantics in assembler RDF with the general 
dataset type.

This is what I mean by the whole thing having gotten a bit confusing! 
:grin: 

I agree fully that we need to be able to do what you clearly intend to do 
with your example. I just want the the example to look like:
```
<#dataset> a ja:MemoryDataset ;
  ja:data "File for the default graph - also data for dataset" ;
  ja:namedGraph [ ja:graphName "http://example/graph1";; ja:graph "File for 
graph1" ] ;
  ja:namedGraph [ ja:graphName "http://example/graph2";; ja:graph "File for 
graph2" ] ;
.
```
instead so that TIM and general look the same from the assembler POV.

I don't want to block 3.6.0 on this, so what do you say to your three 
points above, plus (after 3.6.0)

4. We gracefully deprecate the use of `ja:data` for anything other than 
loading quads into datasets and make TIM's assembler RDF look like the general 
dataset assembler RDF modulo the dataset type.

Sound okay?


---

[GitHub] jena issue #314: JENA-1430

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/314
  
Okay, then let's break this thing up, as you describe above. Your number 2 
and 3 are uncontroversial. As for number 1, sure, I want that fix too, but this 
PR (#314) didn't do just that. It also conflated the assembler RDF for these 
two types of dataset and I now see that TIM's use of `ja:data` for two purposes 
is a big part of that. Let's revert this and merge #313 instead and we can take 
up JENA-1445 after 3.6.0. 



---

Re: Build failed in Jenkins: Jena_Development_Deploy #1465

[ajs6f]

Looks like Elasticsearch startup went over a minute and that timed out the 
build.

We set that to 60 seconds:

https://github.com/apache/jena/blob/master/jena-text-es/pom.xml#L136

Is it worth bumping that up to 90? I don't want to let CI jobs sit in process 
too long on a shared server, but thirty seconds more seems reasonable.

ajs6f

> On Dec 9, 2017, at 6:38 AM, Apache Jenkins Server  
> wrote:
> 
> See 
> <https://builds.apache.org/job/Jena_Development_Deploy/1465/display/redirect?page=changes>
> 
> Changes:
> 
> [andy] JENA-1430: Read quads for ja:data by filename
> 
> [ajs6f] JENA-1430: Quad loading for in-memory assemblers
> 
> [ajs6f] Correcting dataset type in Fuseki example
> 
> [ajs6f] Better, clearer handling of mixed assembling
> 
> [andy] Put in compatibility for ja:DatasetTxnMem.
> 
> [andy] Improve the message.
> 
> [andy] Use stream.
> 
> [andy] Fix URI
> 
> --
> [...truncated 316.26 KB...]
> [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ jena-text 
> ---
> [INFO] Installing 
> <https://builds.apache.org/job/Jena_Development_Deploy/ws/jena-text/target/jena-text-3.6.0-SNAPSHOT.jar>
>  to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-SNAPSHOT.jar
> [INFO] Installing 
> <https://builds.apache.org/job/Jena_Development_Deploy/ws/jena-text/pom.xml> 
> to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-SNAPSHOT.pom
> [INFO] Installing 
> <https://builds.apache.org/job/Jena_Development_Deploy/ws/jena-text/target/jena-text-3.6.0-SNAPSHOT-sources.jar>
>  to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-SNAPSHOT-sources.jar
> [INFO] Installing 
> <https://builds.apache.org/job/Jena_Development_Deploy/ws/jena-text/target/jena-text-3.6.0-SNAPSHOT-javadoc.jar>
>  to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-SNAPSHOT-javadoc.jar
> [INFO] 
> [INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ jena-text ---
> [INFO] Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/maven-metadata.xml
> [INFO] Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/maven-metadata.xml
>  (1.2 kB at 2.0 kB/s)
> [INFO] Uploading to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-20171209.113552-28.jar
> [INFO] Uploaded to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-20171209.113552-28.jar
>  (96 kB at 75 kB/s)
> [INFO] Uploading to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-20171209.113552-28.pom
> [INFO] Uploaded to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-20171209.113552-28.pom
>  (5.5 kB at 5.1 kB/s)
> [INFO] Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/maven-metadata.xml
> [INFO] Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/maven-metadata.xml
>  (424 B at 751 B/s)
> [INFO] Uploading to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/maven-metadata.xml
> [INFO] Uploaded to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/maven-metadata.xml
>  (1.2 kB at 1.1 kB/s)
> [INFO] Uploading to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/maven-metadata.xml
> [INFO] Uploaded to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/maven-metadata.xml
>  (424 B at 379 B/s)
> [INFO] Uploading to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text/3.6.0-SNAPSHOT/jena-text-3.6.0-20171209.113552-28-sources.jar
> [INFO] Uploaded to apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-text

[GitHub] jena pull request #329: Give more time to ElasticSearch to startup

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/329

Give more time to ElasticSearch to startup



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena MoreTimeForESStartup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #329


commit 90f4e037ef489d446e602f7a45810574fdb9eb82
Author: ajs6f 
Date:   2017-12-12T18:25:00Z

Give more time to ElasticSearch to startup




---

potential new API Was: Immutability

[ajs6f]

Yes, the SPI is likely fine, or we will find out otherwise if we try a new API. 
:)

A new API is clearly no small thing. I suppose one way to approach the thing 
would be to look at what Clerezza and RDF4J 4 and Commons RDF have come up 
with, what worked and what didn't, and then bring in our own use cases and 
ideas...

The two "asks" I've heard recently from Jena users are: immutability and 
Streams. I'm guessing there are others. I would like to take advantage of 
Optional for some of the attributes of a node (e.g. return the lang of a 
literal as Optional).

Other thoughts? It's fun to make wish lists at the end of the year. :grin:

ajs6f

> On Nov 16, 2017, at 11:47 AM, Andy Seaborne  wrote:
> 
> Yes, I think it is best done as a new API.
> 
> Modifications to the existing one are likely to run into problems somewhere 
> and given the number of users, and the change over time on upgrades, having 
> compatibility matters as does gradual migration
> 
> A complicates case: Dataset.getModel(String) returns a model. No idea whether 
> it is immutable or not or how it will be used.  Assigning to a ImmutableModel 
> does not guaratee anything - it helps the caller not make mistakes but the 
> model is not immutable.  Some code calling getModel can change it. And the 
> union model isn't immutable! - it's a dynamic union and any name graph change 
> will be reflected. The world is multithreaded.
> 
> But the main reason for a new API is that this is just one of several 
> features to include so to make it a success these different features need to 
> come together into one *agreed* design.
> 
> The good news is that it can target the SPI.
> (I see no value in changing the SPI for this - that's not it's role)
> 
> ARQ's API is really very small. A design that took a dataset+model view of 
> the world would be a good thing.
> 
>Andy
> 
> See also commons-rdf.
> That is RDF 1.1 compliant
> 
> On 15/11/17 15:52, aj...@apache.org wrote:
>> So it seems to me that if we want to introduce immutable types, we might 
>> want to do that in the context of a completely new API. The use of the Java 
>> 8 Streams API is also something that has been mooted as something that might 
>> merit a new Jena API. (Instead of mixing things up in the current one.)
>> I'm not sure how that plays out with ARQ, though. We would want people to be 
>> able to use the new types with ARQ without much difficulty.

Re: Jena-1427 - nextOptional()

[ajs6f]

I thought the upshot of the discussion was that we are okay with this because 
the Model API (the only place from which we support getting ExtendedIterators) 
will never have a null retrieved value and we clearly warn in the method 
documentation that the retrieved value must not be null. 

If the retrieved value is never null, the usage is congruent with things like 
Stream::findFirst.

I agree that it will be difficult to remove, which is why I don't want to add a 
bunch of other nextOr* methods.

ajs6f

> On Dec 14, 2017, at 12:46 PM, Claude Warren  wrote:
> 
> I was expecting that with the discovery that optional throws a null pointer
> exception when the retrieved value is null would be enough to remove this
> functionality.
> 
> I am concerned that once added it will be difficult to remove and that its
> operation is not congruent with stream based optional usage.
> 
> Claude
> 
> On 14 Dec 2017 10:33, "Andy Seaborne"  wrote:
> 
>> Claude,
>> 
>> The JIRA ticket ends:
>> 
>> [[
>> ASF GitHub Bot added a comment - 04/Dec/17 15:36
>> 
>> That was already the case in the PR and I've added text to call it out
>> explicitly into the javadoc.
>> 
>> githubbot ASF GitHub Bot added a comment - 04/Dec/17 15:40
>> 
>> Github user ajs6f commented on the issue:
>> 
>> https://github.com/apache/jena/pull/323
>> 
>> Okay, that works for me!
>> 
>> Andy Seaborne added a comment - 04/Dec/17 15:58
>> 
>> nextOptional added for release 3.6.0.
>> 
>> Proposal: close this JIRA for now, see how nextOptional works out and
>> revisit orElse* based on experience.
>> 
>> ajs6f A. Soroka added a comment - 1 week ago
>> 
>> +1
>> ]]
>> 
>> Adding nextOptional, and no others, is in the RC.
>> 
>> What were you expecting?
>> 
>>Andy
>> 
>> On 14/12/17 08:44, Claude Warren wrote:
>> 
>>> Perhaps i wasn't paying close enough attention but i didn't think adding
>>> optional and other streaming methods had been agreed.  In fact i thought
>>> quite the oposite.
>>> 
>>> If i am wrong please forgive the noise.  If i am correct shouldn't 1427 be
>>> removed from the release candidate?
>>> 
>>> Claude
>>> 
>>> 

Re: [VOTE] Apache Jena 3.6.0 RC1

[ajs6f]

 X ] +1 Approve the release

Build ran fine (`mvn clean install`) , source is present, I spot-checked the 
sigs and checksums and everything looks good.

Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 
2017-04-03T15:39:06-04:00)
Java version: 1.8.0_65, vendor: Oracle Corporation
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"

ajs6f

> On Dec 16, 2017, at 6:55 PM, Bruno P. Kinoshita 
>  wrote:
> 
> [ X ] +1 Approve the release
> 
> Build passes with no errors with 
> 
> Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 
> 2017-10-18T20:58:13+13:00)
> Maven home: /opt/apache-maven-3.5.2
> Java version: 1.8.0_151, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_NZ, platform encoding: UTF-8
> OS name: "linux", version: "4.4.0-103-generic", arch: "amd64", family: "unix"
> 
> Checked signatures for Maven artefacts (core and tdb) and dist (both source 
> and binaries). Had a look at the generated dist files and found nothing out 
> of ordinary.
> 
> Had a bit more of time so tested JENA-1420 too. Went to jena-fuseki2, found 
> the fuseki-dev script, executed it and got
> 
> ~/Development/java/jena/jena/jena-fuseki2/jena-fuseki-core$ ./fuseki-dev 
> [2017-12-17 12:43:29] ServerINFO  Apache Jena Fuseki 3.6.0
> [2017-12-17 12:43:29] ConfigINFO  
> FUSEKI_HOME=/home/kinow/Development/java/jena/jena/jena-fuseki2/jena-fuseki-core
> [2017-12-17 12:43:29] ConfigINFO  
> FUSEKI_BASE=/home/kinow/Development/java/jena/jena/jena-fuseki2/jena-fuseki-core/run
> [2017-12-17 12:43:30] ConfigINFO  Shiro file: 
> file:///home/kinow/Development/java/jena/jena/jena-fuseki2/jena-fuseki-core/run/shiro.ini
> [2017-12-17 12:43:30] ServerINFO  Started 2017/12/17 12:43:30 NZDT on 
> port 3030
> 
> Saved the example ttl from JENA-1420, uploaded it and got
> 
> 
> """
> file.ttl
> 214 bytes
> 
> Result: success. 3 triples
> """
> 
> So looks like the bug is no happening anymore :D
> 
> I was lurking in the discussion about nextOptional() in GitHub/mailing list 
> and it looks like it's a test to see how it works. Code looks good.
> 
> Libraries updated look good too. There's one that I am interested to test 
> later as we are updating at $work too. So I will have another look at it 
> later today (Sunday NZ time), but don't think it would be of any problem
> 
> So my +1, thanks for preparing the release, and happy holidays everyone!
> 
> Cheers
> Bruno
> 
> 
> From: Andy Seaborne 
> To: "dev@jena.apache.org"  
> Sent: Thursday, 14 December 2017 12:27 PM
> Subject: [VOTE] Apache Jena 3.6.0 RC1
> 
> 
> 
> Hi,
> 
> 
> Here is a vote on a release of Jena 3.6.0.
> 
> 
> This is the first proposed candidate for a 3.5.0 release.
> 
> 
> Note - the deadline is 18:00 UTC on Sunday - not midnight - so that
> 
> bytes can be pushed out and sync over night.
> 
> 
> 
>  Release changes
> 
> 
> 30 JIRA:
> 
> https://s.apache.org/jena-3.6.0-jira
> 
> 
> == Datasets
> 
> 
> JENA-1430
> 
> The preferred way to work with in-memory, plain data is to use
> 
> ja:MemoryDataset.
> 
> 
> == Other
> 
> 
> JENA-1427
> 
> ExtendedIterator gains a new method, nextOptional()
> 
> 
> Feed back on this would most welcome.
> 
> 
> JENA-1420 
> 
> FIX: Fuseki 3.5.0 "upload files" fails
> 
> 
> JENA-1443
> 
> Models from Datasets (Graphs from DatasetGraphs) provide graph 
> 
> transaction handlers.
> 
> 
> == Upgrades to libraries (runtime dependencies):
> 
> 
> No license changes.
> 
> 
> Eclipse Jetty:   9.4.5.v20170502 -> 9.4.7.v20170914
> 
> Apache Shiro:1.2.4 -> 1.2.6
> 
> Apache Commons IO:   2.5 -> 2.6
> 
> Apache Commons CSV:  1.4 -> 1.5
> 
> Apache Commons Codec:1.10 -> 1.11
> 
> Apache Thrift:   0.9.3 -> 0.10.0
> 
> 
>  Release Vote
> 
> 
> Everyone, not just committers, is invited to test and vote.
> 
> Please download and test the proposed release.
> 
> 
> Staging repository:
> 
>  https://repository.apache.org/content/repositories/orgapachejena-1021/
> 
> 
> Proposed dist/ area:
> 
>  https://dist.apache.org/repos/dist/dev/jena/
> 
> 
> Keys:
> 
>  https://svn.apache.org/repos/asf/jena/dist/KEYS
> 
> 
> Git commit (browser URL):
> 
>  http://git-wip-us

Re: consistent blank id values from RDFConnection

[ajs6f]

Claude-- I'm looking at RDFConnection, but it's an interface. I think you mean 
around L220 of JSONInput itself, right?

It looks like SyntaxLabels has some LabelToNode factory methods that might fit 
the bill, like createNodeToLabelAsGiven(), but JSONInput doesn't offer any way 
to select which method to use. At L195 it uses 
SyntaxLabels.createLabelToNode(). 

We could thread such a mapping choice all the way through the call stack, but 
that seems a bit difficult to me. Maybe we could introduce a Context setting 
for this purpose?

ajs6f

> On Dec 17, 2017, at 9:28 AM, Claude Warren  wrote:
> 
> Greetings,
> 
> I am looking at org.apache.jena.sparql.resultset.JSONInput and the way in
> which it parses blank nodes.
> 
> I have a requirement for an application such that the same blank node
> returned on multiple queries returns the same blank node id.
> 
> I have verified that Fuseki does this (given that the data is only loaded
> once -- reloading the data can renumber the nodes).  In any case Fuseki
> seems to return the ids from the underlying data store.
> 
> However, when RDFConnection is executing a query it remaps the ids during
> the query.
> 
> Down around line 220 it uses a labelMap to construct a new value for the
> bnode.  My question is:
> 
> Is there a simple way to have the LabelMap return the same value for the
> same blank node across multiple queries? (assuming the value does not
> change in the data store).
> 
> I know there was a discussion of using UUIDs or some such to generate the
> blank ids on the way into the graph but I don't see any way for
> RDFConnection to return them consistently.
> 
> Claude
> 
> 
> 
> 
> 
> -- 
> I like: Like Like - The likeliest place on the web
> <http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren

Re: consistent blank id values from RDFConnection

[ajs6f]

Where? I found nothing documented. 

ajs6f

> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
> 
> On 17/12/17 15:19, ajs6f wrote:
>> Claude-- I'm looking at RDFConnection, but it's an interface. I think you 
>> mean around L220 of JSONInput itself, right?
>> It looks like SyntaxLabels has some LabelToNode factory methods that might 
>> fit the bill, like createNodeToLabelAsGiven(), but JSONInput doesn't offer 
>> any way to select which method to use. At L195 it uses 
>> SyntaxLabels.createLabelToNode().
>> We could thread such a mapping choice all the way through the call stack, 
>> but that seems a bit difficult to me. Maybe we could introduce a Context 
>> setting for this purpose?
> 
> They already exist!

Re: consistent blank id values from RDFConnection

[ajs6f]

That is useful, and it's undocumented. Is that because it is dangerous (in the 
sense in which you used the phrase "dubious in terms of spec compliance") or 
just because we never have documented it?

ajs6f

> On Dec 17, 2017, at 10:43 AM, Andy Seaborne  wrote:
> 
> ARQ.enableBlankNodeResultLabels()
> 
> On 17/12/17 15:39, ajs6f wrote:
>> Where? I found nothing documented.
>> ajs6f
>>> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
>>> 
>>> On 17/12/17 15:19, ajs6f wrote:
>>>> Claude-- I'm looking at RDFConnection, but it's an interface. I think you 
>>>> mean around L220 of JSONInput itself, right?
>>>> It looks like SyntaxLabels has some LabelToNode factory methods that might 
>>>> fit the bill, like createNodeToLabelAsGiven(), but JSONInput doesn't offer 
>>>> any way to select which method to use. At L195 it uses 
>>>> SyntaxLabels.createLabelToNode().
>>>> We could thread such a mapping choice all the way through the call stack, 
>>>> but that seems a bit difficult to me. Maybe we could introduce a Context 
>>>> setting for this purpose?
>>> 
>>> They already exist!

Re: consistent blank id values from RDFConnection

[ajs6f]

> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
> 
>> Why would it be dangerous?

As I wrote:

>>> (in the sense in which you used the phrase "dubious in terms of spec 
>>> compliance")

It might confuse people into thinking that maintaining bnode labeling is a 
normal part of using SPARQL, when it isn't-- it's something extra that Jena 
provides.

If there's no reason this is an undocumented feature, I'm going to document it 
at:

https://jena.apache.org/documentation/query/app_api.html

ajs6f

> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
> 
> Why would it be dangerous?
> 
> On 17/12/17 15:46, ajs6f wrote:
>> That is useful, and it's undocumented. Is that because it is dangerous (in 
>> the sense in which you used the phrase "dubious in terms of spec 
>> compliance") or just because we never have documented it?
>> ajs6f
>>> On Dec 17, 2017, at 10:43 AM, Andy Seaborne  wrote:
>>> 
>>> ARQ.enableBlankNodeResultLabels()
>>> 
>>> On 17/12/17 15:39, ajs6f wrote:
>>>> Where? I found nothing documented.
>>>> ajs6f
>>>>> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
>>>>> 
>>>>> On 17/12/17 15:19, ajs6f wrote:
>>>>>> Claude-- I'm looking at RDFConnection, but it's an interface. I think 
>>>>>> you mean around L220 of JSONInput itself, right?
>>>>>> It looks like SyntaxLabels has some LabelToNode factory methods that 
>>>>>> might fit the bill, like createNodeToLabelAsGiven(), but JSONInput 
>>>>>> doesn't offer any way to select which method to use. At L195 it uses 
>>>>>> SyntaxLabels.createLabelToNode().
>>>>>> We could thread such a mapping choice all the way through the call 
>>>>>> stack, but that seems a bit difficult to me. Maybe we could introduce a 
>>>>>> Context setting for this purpose?
>>>>> 
>>>>> They already exist!

Re: consistent blank id values from RDFConnection

[ajs6f]

That's not how I understand:

https://www.w3.org/TR/sparql11-query/#BlankNodesInResults

"Blank node labels are scoped to a result set (see "SPARQL Query Results XML 
Format" and "SPARQL 1.1 Query Results JSON Format") or, for the CONSTRUCT query 
form, the result graph."

Multiple calls to a single graph => multiple queries => multiple result sets 
(or result graphs) => multiple scopes.

ajs6f

> On Dec 17, 2017, at 11:15 AM, Adam Jacobs  wrote:
> 
> My understanding is there's nothing wrong with maintaining labels among 
> multiple calls to a single graph.
> The danger would be the risk of maintaining labels among calls to multiple 
> graphs.
> At least, that's what I get out of this SO answer: 
> https://stackoverflow.com/questions/44477876/grouping-by-blank-nodes/44498034#44498034
> 
> 
> From: ajs6f 
> Sent: Sunday, December 17, 2017 10:11 AM
> To: dev@jena.apache.org
> Subject: Re: consistent blank id values from RDFConnection
> 
>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>> 
>>> Why would it be dangerous?
> 
> As I wrote:
> 
>>>> (in the sense in which you used the phrase "dubious in terms of spec 
>>>> compliance")
> 
> It might confuse people into thinking that maintaining bnode labeling is a 
> normal part of using SPARQL, when it isn't-- it's something extra that Jena 
> provides.
> 
> If there's no reason this is an undocumented feature, I'm going to document 
> it at:
> 
> https://jena.apache.org/documentation/query/app_api.html
> 
> ajs6f
> 
>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>> 
>> Why would it be dangerous?
>> 
>> On 17/12/17 15:46, ajs6f wrote:
>>> That is useful, and it's undocumented. Is that because it is dangerous (in 
>>> the sense in which you used the phrase "dubious in terms of spec 
>>> compliance") or just because we never have documented it?
>>> ajs6f
>>>> On Dec 17, 2017, at 10:43 AM, Andy Seaborne  wrote:
>>>> 
>>>> ARQ.enableBlankNodeResultLabels()
>>>> 
>>>> On 17/12/17 15:39, ajs6f wrote:
>>>>> Where? I found nothing documented.
>>>>> ajs6f
>>>>>> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
>>>>>> 
>>>>>> On 17/12/17 15:19, ajs6f wrote:
>>>>>>> Claude-- I'm looking at RDFConnection, but it's an interface. I think 
>>>>>>> you mean around L220 of JSONInput itself, right?
>>>>>>> It looks like SyntaxLabels has some LabelToNode factory methods that 
>>>>>>> might fit the bill, like createNodeToLabelAsGiven(), but JSONInput 
>>>>>>> doesn't offer any way to select which method to use. At L195 it uses 
>>>>>>> SyntaxLabels.createLabelToNode().
>>>>>>> We could thread such a mapping choice all the way through the call 
>>>>>>> stack, but that seems a bit difficult to me. Maybe we could introduce a 
>>>>>>> Context setting for this purpose?
>>>>>> 
>>>>>> They already exist!
> 

Re: consistent blank id values from RDFConnection

[ajs6f]

It's certainly a long-running discussion. I.e., there are sound reasons that 
bnodes are treated as they are in the recommendations, and as Andy remarked 
(and Claude and Andy prove with their use cases), there are sound reasons that 
one might like to enlarge the scope for identity just a bit, now and then.

I think offering configurability is a good middle ground. Claude-- does that 
ARQ option meet your needs?

Andy-- do you think pulling all the related config into Context Symbols would 
be a good way to "pull together" the API?, as you suggest? Context values are a 
little more "fine-grained" than static settings, and can vary within an app 
more easily.

ajs6f

> On Dec 17, 2017, at 11:29 AM, Adam Jacobs  wrote:
> 
> You're right, that section does seem conclusive, at least insofar as SPARQL 
> is concerned.
> 
> 
> ____
> From: ajs6f 
> Sent: Sunday, December 17, 2017 10:19 AM
> To: dev@jena.apache.org
> Subject: Re: consistent blank id values from RDFConnection
> 
> That's not how I understand:
> 
> https://www.w3.org/TR/sparql11-query/#BlankNodesInResults
> 
> "Blank node labels are scoped to a result set (see "SPARQL Query Results XML 
> Format" and "SPARQL 1.1 Query Results JSON Format") or, for the CONSTRUCT 
> query form, the result graph."
> 
> Multiple calls to a single graph => multiple queries => multiple result sets 
> (or result graphs) => multiple scopes.
> 
> ajs6f
> 
>> On Dec 17, 2017, at 11:15 AM, Adam Jacobs  wrote:
>> 
>> My understanding is there's nothing wrong with maintaining labels among 
>> multiple calls to a single graph.
>> The danger would be the risk of maintaining labels among calls to multiple 
>> graphs.
>> At least, that's what I get out of this SO answer: 
>> https://stackoverflow.com/questions/44477876/grouping-by-blank-nodes/44498034#44498034
>> 
>> 
>> From: ajs6f 
>> Sent: Sunday, December 17, 2017 10:11 AM
>> To: dev@jena.apache.org
>> Subject: Re: consistent blank id values from RDFConnection
>> 
>>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>>> 
>>>> Why would it be dangerous?
>> 
>> As I wrote:
>> 
>>>>> (in the sense in which you used the phrase "dubious in terms of spec 
>>>>> compliance")
>> 
>> It might confuse people into thinking that maintaining bnode labeling is a 
>> normal part of using SPARQL, when it isn't-- it's something extra that Jena 
>> provides.
>> 
>> If there's no reason this is an undocumented feature, I'm going to document 
>> it at:
>> 
>> https://jena.apache.org/documentation/query/app_api.html
>> 
>> ajs6f
>> 
>>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>>> 
>>> Why would it be dangerous?
>>> 
>>> On 17/12/17 15:46, ajs6f wrote:
>>>> That is useful, and it's undocumented. Is that because it is dangerous (in 
>>>> the sense in which you used the phrase "dubious in terms of spec 
>>>> compliance") or just because we never have documented it?
>>>> ajs6f
>>>>> On Dec 17, 2017, at 10:43 AM, Andy Seaborne  wrote:
>>>>> 
>>>>> ARQ.enableBlankNodeResultLabels()
>>>>> 
>>>>> On 17/12/17 15:39, ajs6f wrote:
>>>>>> Where? I found nothing documented.
>>>>>> ajs6f
>>>>>>> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
>>>>>>> 
>>>>>>> On 17/12/17 15:19, ajs6f wrote:
>>>>>>>> Claude-- I'm looking at RDFConnection, but it's an interface. I think 
>>>>>>>> you mean around L220 of JSONInput itself, right?
>>>>>>>> It looks like SyntaxLabels has some LabelToNode factory methods that 
>>>>>>>> might fit the bill, like createNodeToLabelAsGiven(), but JSONInput 
>>>>>>>> doesn't offer any way to select which method to use. At L195 it uses 
>>>>>>>> SyntaxLabels.createLabelToNode().
>>>>>>>> We could thread such a mapping choice all the way through the call 
>>>>>>>> stack, but that seems a bit difficult to me. Maybe we could introduce 
>>>>>>>> a Context setting for this purpose?
>>>>>>> 
>>>>>>> They already exist!
>> 
> 

Re: consistent blank id values from RDFConnection

[ajs6f]

Okay, certainly I don't want to document something that is both hidden and 
faulty! :grin:

I guess this is a general question (nothing to do with JSON in particular). How 
about a setting to enable any of the modes grouped in SyntaxLabels, for all 
parsing and output? Does that sound reasonable?  In that case, the API seems a 
bit more complex, because it's something that could arise outside of ARQ, so 
maybe a static setting is better (than a Symbol in Contexts).

ajs6f

> On Dec 17, 2017, at 11:36 AM, Andy Seaborne  wrote:
> 
> I was hoping we'd change it is because ATM it doesn't work properly and is a 
> bad design.
> 
> On 17/12/17 16:11, ajs6f wrote:
>>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>>> 
>>>> Why would it be dangerous?
>> As I wrote:
>>>>> (in the sense in which you used the phrase "dubious in terms of spec 
>>>>> compliance")
>> It might confuse people into thinking that maintaining bnode labeling is a 
>> normal part of using SPARQL, when it isn't-- it's something extra that Jena 
>> provides.
>> If there's no reason this is an undocumented feature, I'm going to document 
>> it at:
>> https://jena.apache.org/documentation/query/app_api.html
> 
> I prefer putting it, eventually, somewhere in advanced features.
> 
>   Andy
> 
>> ajs6f
>>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>>> 
>>> Why would it be dangerous?
>>> 
>>> On 17/12/17 15:46, ajs6f wrote:
>>>> That is useful, and it's undocumented. Is that because it is dangerous (in 
>>>> the sense in which you used the phrase "dubious in terms of spec 
>>>> compliance") or just because we never have documented it?
>>>> ajs6f
>>>>> On Dec 17, 2017, at 10:43 AM, Andy Seaborne  wrote:
>>>>> 
>>>>> ARQ.enableBlankNodeResultLabels()
>>>>> 
>>>>> On 17/12/17 15:39, ajs6f wrote:
>>>>>> Where? I found nothing documented.
>>>>>> ajs6f
>>>>>>> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
>>>>>>> 
>>>>>>> On 17/12/17 15:19, ajs6f wrote:
>>>>>>>> Claude-- I'm looking at RDFConnection, but it's an interface. I think 
>>>>>>>> you mean around L220 of JSONInput itself, right?
>>>>>>>> It looks like SyntaxLabels has some LabelToNode factory methods that 
>>>>>>>> might fit the bill, like createNodeToLabelAsGiven(), but JSONInput 
>>>>>>>> doesn't offer any way to select which method to use. At L195 it uses 
>>>>>>>> SyntaxLabels.createLabelToNode().
>>>>>>>> We could thread such a mapping choice all the way through the call 
>>>>>>>> stack, but that seems a bit difficult to me. Maybe we could introduce 
>>>>>>>> a Context setting for this purpose?
>>>>>>> 
>>>>>>> They already exist!

Re: Jena-1427 - nextOptional()

[ajs6f]

> On Dec 17, 2017, at 2:18 PM, Claude Warren  wrote:
> It seems that much of the rest of  the discussion is covered by adding a 
> method to convert the iterator to a stream (probably just a call to an 
> existing Java method.

Just to this point alone-- this is something that came up _in_ the discussion 
(we even already have Iter::asStream for the purpose), and I don't agree with 
it. Converting an iterator to a stream is a useful thing, to be sure, but 
although it doesn't really speak to the optimizations that people (rightly) 
associate with a stream (i.e. it doesn't push filters or other computations 
back to whatever they might look like closer to the source of elements) it does 
create a bunch of objects and machinery. I'd rather not encourage people to do 
this-- I'd rather encourage them to use plain iterators unless/until we can 
actually supply an API with real streams.

Otherwise:

> Perhaps we should add a next( T default ) method that will return the default 
> if hasNext() returns false.  This seem better than polluting the environment 
> with Optionals that don't really belong.


I don't agree at all that Optional doesn't belong here, but if the original 
requestor can confirm that this would meet the use case, and if we can come up 
with something for throwing a custom exception that does pass Claude's "smell 
test", then I'm fine with it. I'd rather not add method after method to 
ExtendedIterator for every possible syntactic convenience. (This ain't Scala! 
:wink:)

ajs6f

> On Dec 17, 2017, at 2:18 PM, Claude Warren  wrote:
> 
> From the original request it seems the requirements were:
> 1. return a default value if the iterator is empty.
> 2. throw a custom exception if the iterator is empty.
> 
> 
> Perhaps we should add a next( T default ) method that will return the
> default if hasNext() returns false.  This seem better than polluting the
> environment with Optionals that don't really belong.
> 
> If we want to provide a method to throw an custom exception we could add
> something like next( Class e ) to create and throw the
> exception.  But I don't like this one for some reason.  Like the Optional
> it doesn't pass the smell test for me.
> 
> It seems that much of the rest of  the discussion is covered by adding a
> method to convert the iterator to a stream (probably just a call to an
> existing Java method.
> 
> Claude
> 
> 
> On Sun, Dec 17, 2017 at 5:50 PM, Andy Seaborne  wrote:
> 
>> We seem to have drifted a bit here - the original use case wasn't about
>> streaming as I read it. list* get used to write zero/one tests and one/many
>> tests dealing with properties etc.
>> 
>> And, of course, we want to avoid the anti-pattern of Optional/if-empty.
>> 
>> Claude - thoughts on the use case? How might we evolve the current API
>> without method bloat?
>> 
>>Andy
>> 
>> 
>> On 14/12/17 17:46, Claude Warren wrote:
>> 
>>> I was expecting that with the discovery that optional throws a null
>>> pointer
>>> exception when the retrieved value is null would be enough to remove this
>>> functionality.
>>> 
>>> I am concerned that once added it will be difficult to remove and that its
>>> operation is not congruent with stream based optional usage.
>>> 
>>> Claude
>>> 
>>> On 14 Dec 2017 10:33, "Andy Seaborne"  wrote:
>>> 
>>> Claude,
>>>> 
>>>> The JIRA ticket ends:
>>>> 
>>>> [[
>>>> ASF GitHub Bot added a comment - 04/Dec/17 15:36
>>>> 
>>>> That was already the case in the PR and I've added text to call it out
>>>> explicitly into the javadoc.
>>>> 
>>>> githubbot ASF GitHub Bot added a comment - 04/Dec/17 15:40
>>>> 
>>>> Github user ajs6f commented on the issue:
>>>> 
>>>> https://github.com/apache/jena/pull/323
>>>> 
>>>> Okay, that works for me!
>>>> 
>>>> Andy Seaborne added a comment - 04/Dec/17 15:58
>>>> 
>>>> nextOptional added for release 3.6.0.
>>>> 
>>>> Proposal: close this JIRA for now, see how nextOptional works out and
>>>> revisit orElse* based on experience.
>>>> 
>>>> ajs6f A. Soroka added a comment - 1 week ago
>>>> 
>>>> +1
>>>> ]]
>>>> 
>>>> Adding nextOptional, and no others, is in the RC.
>>>> 
>>>> What were you expecting?
>>>> 
>>>> Andy
>>>> 
>>>> On 14/12/17 08:44, Claude Warren wrote:
>>>> 
>>>> Perhaps i wasn't paying close enough attention but i didn't think adding
>>>>> optional and other streaming methods had been agreed.  In fact i thought
>>>>> quite the oposite.
>>>>> 
>>>>> If i am wrong please forgive the noise.  If i am correct shouldn't 1427
>>>>> be
>>>>> removed from the release candidate?
>>>>> 
>>>>> Claude
>>>>> 
>>>>> 
>>>>> 
>>> 
> 
> 
> -- 
> I like: Like Like - The likeliest place on the web
> <http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren

[GitHub] jena issue #329: Give more time to ElasticSearch to startup

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/329
  
Dang it! This is the second time this has happened. I've been merging my PR 
branches into `master` and then pushing them. It normally works fine... I need 
to figure out what's nutty on my end. In the meantime, I'll clean this up.


---

[GitHub] jena pull request #329: Give more time to ElasticSearch to startup

[ajs6f]

Github user ajs6f closed the pull request at:

https://github.com/apache/jena/pull/329


---

[GitHub] jena issue #329: Give more time to ElasticSearch to startup

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/329
  
Ah, I think I figured it out-- for future reference, this seems to happen 
when you merge a branch into `master` and push it, but the branch on Github 
isn't up to date with the local repo. Okay, I should be able to stop annoying 
people with this.


---

[GitHub] jena pull request #331: JENA-1450: fn:error

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/331#discussion_r157378417
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/function/library/FN_Error.java ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.function.library;
+
+import java.util.List ;
+
+import org.apache.jena.atlas.lib.Lib;
+import org.apache.jena.query.QueryBuildException;
+import org.apache.jena.sparql.ARQInternalErrorException;
+import org.apache.jena.sparql.expr.ExprEvalException ;
+import org.apache.jena.sparql.expr.ExprList ;
+import org.apache.jena.sparql.expr.NodeValue ;
+import org.apache.jena.sparql.expr.nodevalue.NodeFunctions;
+import org.apache.jena.sparql.function.FunctionBase;
+
+/** XPath and XQuery Functions and Operators 3.1
+ *  
+ * {@code fn:error()}
+ * 
+ * {@code fn:error(message)}
+ */
+public class FN_Error extends FunctionBase {
+
+@Override
+public void checkBuild(String uri, ExprList args) {
+if ( args.size() != 0 && args.size() != 1 )
+throw new QueryBuildException("Function 
'"+Lib.className(this)+"' takes zero or one arguments") ;
+}
+
+@Override
+public NodeValue exec(List args) {
+if ( args == null )
+// The contract on the function interface is that this should 
not happen.
+throw new ARQInternalErrorException("FunctionBase1: Null args 
list") ;
+
+int size = args.size();
--- End diff --

Did you mean to use this `size` var below? It doesn't seem to get used at 
all...


---

Re: consistent blank id values from RDFConnection

[ajs6f]

> On Dec 18, 2017, at 8:17 AM, Andy Seaborne  wrote:
> On 17/12/17 16:35, ajs6f wrote:
>> 
>> I think offering configurability is a good middle ground. Claude-- does that 
>> ARQ option meet your needs?
>> Andy-- do you think pulling all the related config into Context Symbols 
>> would be a good way to "pull together" the API?, as you suggest? 
> 
> It is the end-to-end coordination of a number of steps in different places. 
> Graphs haven't even been mentioned so far.  I think exposing the machinery 
> piecemeal is bad (= unhelpful; encourages misuse; does not work in the system 
> as a whole where other things are going on).  We have the right architecture, 
> let's use it.

I don't understand this very well-- to what architecture are you referring? How 
are we to use it? I'm all in favor of a unified exposure of a "maintain bnode 
identity" functionality, but to do that we have to find a way to expose it, 
period. Are you saying that Contexts are not the right choice because, as I 
pointed out elsewhere, they are an ARQ-specific construct?

ajs6f
> 
>Andy
> 
>> Context values are a little more "fine-grained" than static settings, and 
>> can vary within an app more easily.
>> ajs6f
>>> On Dec 17, 2017, at 11:29 AM, Adam Jacobs  wrote:
>>> 
>>> You're right, that section does seem conclusive, at least insofar as SPARQL 
>>> is concerned.
>>> 
>>> 
>>> 
>>> From: ajs6f 
>>> Sent: Sunday, December 17, 2017 10:19 AM
>>> To: dev@jena.apache.org
>>> Subject: Re: consistent blank id values from RDFConnection
>>> 
>>> That's not how I understand:
>>> 
>>> https://www.w3.org/TR/sparql11-query/#BlankNodesInResults
>>> 
>>> "Blank node labels are scoped to a result set (see "SPARQL Query Results 
>>> XML Format" and "SPARQL 1.1 Query Results JSON Format") or, for the 
>>> CONSTRUCT query form, the result graph."
>>> 
>>> Multiple calls to a single graph => multiple queries => multiple result 
>>> sets (or result graphs) => multiple scopes.
>>> 
>>> ajs6f
>>> 
>>>> On Dec 17, 2017, at 11:15 AM, Adam Jacobs  wrote:
>>>> 
>>>> My understanding is there's nothing wrong with maintaining labels among 
>>>> multiple calls to a single graph.
>>>> The danger would be the risk of maintaining labels among calls to multiple 
>>>> graphs.
>>>> At least, that's what I get out of this SO answer: 
>>>> https://stackoverflow.com/questions/44477876/grouping-by-blank-nodes/44498034#44498034
>>>> 
>>>> 
>>>> From: ajs6f 
>>>> Sent: Sunday, December 17, 2017 10:11 AM
>>>> To: dev@jena.apache.org
>>>> Subject: Re: consistent blank id values from RDFConnection
>>>> 
>>>>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>>>>> 
>>>>>> Why would it be dangerous?
>>>> 
>>>> As I wrote:
>>>> 
>>>>>>> (in the sense in which you used the phrase "dubious in terms of spec 
>>>>>>> compliance")
>>>> 
>>>> It might confuse people into thinking that maintaining bnode labeling is a 
>>>> normal part of using SPARQL, when it isn't-- it's something extra that 
>>>> Jena provides.
>>>> 
>>>> If there's no reason this is an undocumented feature, I'm going to 
>>>> document it at:
>>>> 
>>>> https://jena.apache.org/documentation/query/app_api.html
>>>> 
>>>> ajs6f
>>>> 
>>>>> On Dec 17, 2017, at 11:08 AM, Andy Seaborne  wrote:
>>>>> 
>>>>> Why would it be dangerous?
>>>>> 
>>>>> On 17/12/17 15:46, ajs6f wrote:
>>>>>> That is useful, and it's undocumented. Is that because it is dangerous 
>>>>>> (in the sense in which you used the phrase "dubious in terms of spec 
>>>>>> compliance") or just because we never have documented it?
>>>>>> ajs6f
>>>>>>> On Dec 17, 2017, at 10:43 AM, Andy Seaborne  wrote:
>>>>>>> 
>>>>>>> ARQ.enableBlankNodeResultLabels()
>>>>>>> 
>>>>>>> On 17/12/17 15:39, ajs6f wrote:
>>>>>>>> Where? I found nothing documented.
>>>>>>>> ajs6f
>>>>>>>>> On Dec 17, 2017, at 10:38 AM, Andy Seaborne  wrote:
>>>>>>>>> 
>>>>>>>>> On 17/12/17 15:19, ajs6f wrote:
>>>>>>>>>> Claude-- I'm looking at RDFConnection, but it's an interface. I 
>>>>>>>>>> think you mean around L220 of JSONInput itself, right?
>>>>>>>>>> It looks like SyntaxLabels has some LabelToNode factory methods that 
>>>>>>>>>> might fit the bill, like createNodeToLabelAsGiven(), but JSONInput 
>>>>>>>>>> doesn't offer any way to select which method to use. At L195 it uses 
>>>>>>>>>> SyntaxLabels.createLabelToNode().
>>>>>>>>>> We could thread such a mapping choice all the way through the call 
>>>>>>>>>> stack, but that seems a bit difficult to me. Maybe we could 
>>>>>>>>>> introduce a Context setting for this purpose?
>>>>>>>>> 
>>>>>>>>> They already exist!
>>>> 
>>> 

Re: potential new API Was: Immutability

[ajs6f]

Progress on 1391 hasn't been blocked on a new API, not at all. Just taking a 
good while because of general busy-ness. I was able to write some more tests 
over the last few days and I think I should be able to finish up very early in 
January. You can track progress at:

https://github.com/apache/jena/compare/master...ajs6f:JENA-1391

ajs6f

> On Dec 17, 2017, at 8:15 PM, Adam Jacobs  wrote:
> 
> I just want to mention that it was not my intent to derail progress on 
> Jena-1391 (https://issues.apache.org/jira/browse/JENA-1391) by raising the 
> issue of immutability,
> particularly if immutability would be implemented via a new API. I don't 
> think it should be considered a blocker to that story. Apologies if I threw a 
> wrench into the plan.
> 
> 
> ____
> From: ajs6f 
> Sent: Wednesday, December 13, 2017 10:39 AM
> To: dev@jena.apache.org
> Subject: potential new API Was: Immutability
> 
> Yes, the SPI is likely fine, or we will find out otherwise if we try a new 
> API. :)
> 
> A new API is clearly no small thing. I suppose one way to approach the thing 
> would be to look at what Clerezza and RDF4J 4 and Commons RDF have come up 
> with, what worked and what didn't, and then bring in our own use cases and 
> ideas...
> 
> The two "asks" I've heard recently from Jena users are: immutability and 
> Streams. I'm guessing there are others. I would like to take advantage of 
> Optional for some of the attributes of a node (e.g. return the lang of a 
> literal as Optional).
> 
> Other thoughts? It's fun to make wish lists at the end of the year. :grin:
> 
> ajs6f
> 
>> On Nov 16, 2017, at 11:47 AM, Andy Seaborne  wrote:
>> 
>> Yes, I think it is best done as a new API.
>> 
>> Modifications to the existing one are likely to run into problems somewhere 
>> and given the number of users, and the change over time on upgrades, having 
>> compatibility matters as does gradual migration
>> 
>> A complicates case: Dataset.getModel(String) returns a model. No idea 
>> whether it is immutable or not or how it will be used.  Assigning to a 
>> ImmutableModel does not guaratee anything - it helps the caller not make 
>> mistakes but the model is not immutable.  Some code calling getModel can 
>> change it. And the union model isn't immutable! - it's a dynamic union and 
>> any name graph change will be reflected. The world is multithreaded.
>> 
>> But the main reason for a new API is that this is just one of several 
>> features to include so to make it a success these different features need to 
>> come together into one *agreed* design.
>> 
>> The good news is that it can target the SPI.
>> (I see no value in changing the SPI for this - that's not it's role)
>> 
>> ARQ's API is really very small. A design that took a dataset+model view of 
>> the world would be a good thing.
>> 
>>   Andy
>> 
>> See also commons-rdf.
>> That is RDF 1.1 compliant
>> 
>> On 15/11/17 15:52, aj...@apache.org wrote:
>>> So it seems to me that if we want to introduce immutable types, we might 
>>> want to do that in the context of a completely new API. The use of the Java 
>>> 8 Streams API is also something that has been mooted as something that 
>>> might merit a new Jena API. (Instead of mixing things up in the current 
>>> one.)
>>> I'm not sure how that plays out with ARQ, though. We would want people to 
>>> be able to use the new types with ARQ without much difficulty.
> 

Re: Jira and Gitbox integration?

[ajs6f]

Nope. I have not been able to find out anything more about this, and INFRA 
never answered my last email. I agree that the current workflow is not terrible 
at all. I do think going the other way might be a little better, but as you 
say, it's a question of balance. I think the most interesting aspect would be 
lowering the bar to small (one-commit-ish) contributions as low as possible. I 
would like to think we might get more contribution that way, but I don't claim 
to be sure about that.

As an not-very-experienced Apache-er, I probably should have asked this a while 
ago, but is there a protocol about bugging INFRA? If an email goes unanswered 
for several weeks, better to email again or better to file a ticket on INFRA 
asking for info?

ajs6f

> On Dec 17, 2017, at 11:28 AM, Andy Seaborne  wrote:
> 
> Any news?
> 
> Even after reading around, I still don't know how JIRA fits in (if at all). 
> It might only send emails c.f commits@.
> 
> A github only flow is fine by me. Loosing JIRA will be a bump but it is a 
> question of balance - more GH input vs undoing what we have talked about for 
> some time (JIRA). or we have to run both nodes (not so different).
> 
> e.g.
> http://opennlp.apache.org/using-git.html
> 
> Any other good links?
> 
> (there is more that we can do with the contribution process like ask for 
> one-commit (mostly) contributions,
> 
> Of course, all this takes time to do, like documentation.
> 
> The current GH-mirror workflow isn't that problematic at the moment albeit it 
> a few extra steps.
> 
> (the main annoyance for me is deleting local branch and the gitbox does not 
> change this one way of the other as far as I can see).
> 
>   Andy
> 
> On 17/11/17 14:39, ajs6f wrote:
>> Hi, INFRA--
>> Here at Jena we are considering moving our Apache git <-> Github mirroring 
>> to accept changes at Github and mirror them to Apache git (currently it's 
>> the other way around). But right now we have some nice Jira integrations, 
>> and so we have some questions about how that would work if we reversed the 
>> mirroring.
>> Currently, any mention of a Jira ticket (e.g. "I think this could affect 
>> JENA-1234") in a Github PR automatically copies the conversation for that PR 
>> over to the comments in that Jira ticket. Will we be able to keep that 
>> integration if we reverse the mirroring?
>> Github treats issues/tickets and PRs very similarly-- is it possible to 
>> integrate Jira in a similar way so that a PR that doesn't mention an extant 
>> particular Jira ticket automatically files a new Jira ticket?
>> Thanks for any info and all that you already do for us!
>> ajs6f

Re: Jira and Gitbox integration?

[ajs6f]

Bruno++ 

Excellent advice, Bruno! I got answers in minutes. I include a transcript 
below, but the upshot is:

1) We would lose the Jira integration we now have, but INFRA is aware that 
people like to have PR conversations auto-copied to Jira tickets and are 
actively working on that feature.

2) We are the first people to ask about an idea Andy put forward: that a PR 
that doesn't mention an extant Jira ticket should auto-create a new one. INFRA 
seemed... intrigued?

To my memory, losing the Jira integration didn't seem like a blocker to 
anyone-- annoying at best, but not a blocker. Also, a detail: reversing the 
mirroring from what we have now would mean that commits could go to _either_ 
gitbox or Github, which is nice.

ajs6f

Adam Soroka (ajs6f) 
9:27 AM
Hi, INFRA-folks! I'm from Apache Jena, and we've got a question or two about 
Gitbox and possible arrangements between Github, Apache git, and Jira. I sent 
some mail to users@infra, but I think it got lost in the holidays. :grin: Is 
this a good place to ask? They are purely informational questions, for now.
Chris Thistlethwaite
9:27 AM
hey hey
sure ask away!
Adam Soroka (ajs6f) 
9:29 AM
Thanks! Currently, Apache git is the primary repo, and Github is mirrored from 
it. We're thinking about voting to have that reversed, so that we can accept 
PRs and use the Github conversation machinery a little more fluidly. (It's 
pretty good now, but we are hoping that making it a bit easier will encourage 
small contributions.)
So first question: Currently, any mention of a Jira ticket (e.g. "I think this 
could affect JENA-1234") in a Github PR automatically copies the conversation 
for that PR over to the comments in that Jira ticket. Will we be able to keep 
that integration if we reverse the mirroring?
Chris Thistlethwaite
9:30 AM
so gitbox would enable you to use PRs and GitHub as primary
enables mirroring back and forth
so you could either use gitbox.apache.org or github.com to push changes (though 
we encourage using gitbox as primary)
Adam Soroka (ajs6f) 
9:32 AM
So our current Jira integration should just stay the same, if we "go gitbox"?
Chris Thistlethwaite
9:32 AM
as for the PR integration with gitbox/JIRA, there is some tie in, but not 
specifically for what you're asking
Adam Soroka (ajs6f) 
9:32 AM
Ah, okay. So we would lose that.
Chris Thistlethwaite
9:33 AM
for now, yes but it's a feature we're working on
Adam Soroka (ajs6f) 
9:33 AM
Cool, we already discussed that and agreed that it wouldn't be a blocker.
Chris Thistlethwaite
9:33 AM
awesome!
Adam Soroka (ajs6f) 
9:33 AM
Thanks! Second question: Github treats issues/tickets and PRs very similarly-- 
is it possible to integrate Jira in a similar way so that a PR that doesn't 
mention an extant particular Jira ticket automatically files a new Jira ticket?
We aren't inextricably bound to Jira, but we do use it heavily.
Chris Thistlethwaite
9:36 AM
that I don't think we have on the feature to-do list. GH issues/tickets are 
separate from JIRA and this is the first I've heard of someone wanting to tie 
the two together

ajs6f

> On Dec 18, 2017, at 10:17 PM, Bruno P. Kinoshita 
>  wrote:
> 
> For OpenNLP, I created a ticket for INFRA, but most of the interaction 
> happened via Hipchat + e-mails, but it was Joern who contacted them.
> https://www.apache.org/dev/infra-contact#how
> I tried accessing Hipchat with little luck in the past. Don't remember if it 
> was my account type, or apache e-mail, that didn't work...
> 
> But probably Hipchat is the best option.
> HTH
> Bruno
> 
>  From: ajs6f 
> To: dev@jena.apache.org 
> Sent: Tuesday, 19 December 2017 3:53 AM
> Subject: Re: Jira and Gitbox integration?
> 
> Nope. I have not been able to find out anything more about this, and INFRA 
> never answered my last email. I agree that the current workflow is not 
> terrible at all. I do think going the other way might be a little better, but 
> as you say, it's a question of balance. I think the most interesting aspect 
> would be lowering the bar to small (one-commit-ish) contributions as low as 
> possible. I would like to think we might get more contribution that way, but 
> I don't claim to be sure about that.
> 
> As an not-very-experienced Apache-er, I probably should have asked this a 
> while ago, but is there a protocol about bugging INFRA? If an email goes 
> unanswered for several weeks, better to email again or better to file a 
> ticket on INFRA asking for info?
> 
> ajs6f
> 
>> On Dec 17, 2017, at 11:28 AM, Andy Seaborne  wrote:
>> 
>> Any news?
>> 
>> Even after reading around, I still don't know how JIRA fits in (if at all). 
>> It might only send emails c.f commits@.
&g

[GitHub] jena pull request #333: JENA-1451: "contains" for special graphs

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/333#discussion_r158256408
  
--- Diff: 
jena-arq/src/test/java/org/apache/jena/sparql/core/AbstractTestDataset.java ---
@@ -116,4 +116,24 @@
 ds.addNamedModel(graphName, model1) ;
 assertFalse("Dataset should not be empty after a named graph has 
been added!", ds.isEmpty());
 }
+
+@Test public void dataset_07()
+{
+String graphName = "http://example/"; ;
+Dataset ds = createDataset() ;
+ds.addNamedModel(graphName, model1) ;
+assertTrue("Named graph not found", 
ds.containsNamedModel(graphName));
+}
+
+// Even if empty, union and named default graph exist (sort of).
+
+@Test public void dataset_08() {
+Dataset ds = createDataset();
+assertTrue("Union named graph not found", 
ds.containsNamedModel(Quad.unionGraph.getURI()));
+}
+
+@Test public void dataset_09() {
+Dataset ds = createDataset();
+assertTrue("Named default graph not found", 
ds.containsNamedModel(Quad.defaultGraphIRI.getURI()));
--- End diff --

This message seems a little confusing to me (`Named default graph`), maybe:
```
Default graph not found under its URI ( "+Quad.defaultGraphIRI.getURI()+" )"
```
is verbose, but very clear.


---

[GitHub] jena pull request #332: JENA-1452: GraphUnionRead; Union of one.

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/332#discussion_r158292354
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/graph/GraphUnionRead.java ---
@@ -82,16 +94,29 @@ protected PrefixMapping createPrefixMapping() {
 if ( graphs == null ) {
 // This produces unique quads with the same graph node,
 // hence the triples are distinct. 
-Iterator qIter = dataset.findNG(Quad.unionGraph, 
m.getSubject(), m.getPredicate(), m.getObject());
-Iterator tIter = Iter.map(qIter, 
quad->quad.asTriple());
-return WrappedIterator.createNoRemove(tIter) ;
+return quadsToTriples(dataset, Quad.unionGraph, m);
+}
+if ( graphs.isEmpty() )
+return NullIterator.instance();
+if ( graphName != null ) {
+if ( ! dataset.containsGraph(graphName) )
+// Avoid auto-creation.
+return NullIterator.instance();
+// Avoid needing distinct.
+return dataset.getGraph(graphName).find(m);
 }
 // Only certain graphs.
 IteratorConcat iter = new IteratorConcat<>() ;
 forEachGraph((g) -> iter.add(g.find(m))) ;
 return WrappedIterator.createNoRemove(Iter.distinct(iter)) ;
 }
 
+private static ExtendedIterator quadsToTriples(DatasetGraph 
dsg, Node graphName, Triple m) {
--- End diff --

Is this different from `dsg.getGraph(graphName).find(m)`?


---

[GitHub] jena pull request #332: JENA-1452: GraphUnionRead; Union of one.

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/332#discussion_r158302470
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/graph/GraphUnionRead.java ---
@@ -82,16 +94,29 @@ protected PrefixMapping createPrefixMapping() {
 if ( graphs == null ) {
 // This produces unique quads with the same graph node,
 // hence the triples are distinct. 
-Iterator qIter = dataset.findNG(Quad.unionGraph, 
m.getSubject(), m.getPredicate(), m.getObject());
-Iterator tIter = Iter.map(qIter, 
quad->quad.asTriple());
-return WrappedIterator.createNoRemove(tIter) ;
+return quadsToTriples(dataset, Quad.unionGraph, m);
+}
+if ( graphs.isEmpty() )
+return NullIterator.instance();
+if ( graphName != null ) {
+if ( ! dataset.containsGraph(graphName) )
+// Avoid auto-creation.
+return NullIterator.instance();
+// Avoid needing distinct.
+return dataset.getGraph(graphName).find(m);
 }
 // Only certain graphs.
 IteratorConcat iter = new IteratorConcat<>() ;
 forEachGraph((g) -> iter.add(g.find(m))) ;
 return WrappedIterator.createNoRemove(Iter.distinct(iter)) ;
 }
 
+private static ExtendedIterator quadsToTriples(DatasetGraph 
dsg, Node graphName, Triple m) {
--- End diff --

Ah, that makes more sense, thanks!


---

Re: CMS diff: Security in Fuseki2

[ajs6f]

Committed, thanks!

ajs6f

> On Dec 21, 2017, at 11:32 AM, Adrian Gschwend  wrote:
> 
> Clone URL (Committers only):
> https://cms.apache.org/redirect?new=anonymous;action=diff;uri=http://jena.apache.org/documentation%2Ffuseki2%2Ffuseki-security.mdtext
> 
> Adrian Gschwend
> 
> Index: trunk/content/documentation/fuseki2/fuseki-security.mdtext
> ===
> --- trunk/content/documentation/fuseki2/fuseki-security.mdtext
> (revision 1818862)
> +++ trunk/content/documentation/fuseki2/fuseki-security.mdtext
> (working copy)
> @@ -1,44 +1,51 @@
> Title: Security in Fuseki2
> 
> -Fuseki2 provides security by using 
> -[Apache Shiro](http://shiro.apache.org/). This is controlled by a
> -configuration file `shiro.ini` located at `$FUSEKI_BASE/shiro.ini`.
> -If not found, the server initializes this with an preset initial
> -configuration. This can then be replaced or edited as required. This file
> +Fuseki2 provides security by using  [Apache Shiro](http://shiro.apache.org/).
> +This is controlled by the configuration file `shiro.ini` located at
> +`$FUSEKI_BASE/shiro.ini`. If not found, the server initializes with a default
> +configuration. This can then be replaced or edited as required. An existing 
> file
> is never overwritten by the server.
> 
> -The default is that the SPARQL protocols are open but the administrative
> -actions are limited to the localhost. "localhost" is determined by
> -connecting using the `http://localhost:.../...`. It must be "localhost", or
> -`127.0.0.1` (IPv4), or `[::1]` (IPv6), not the external IP address of the
> -machine.
> +In its default configuration, SPARQL endpoints are open to the public but
> +administrative functions are limited to `localhost`. One can access it via
> +`http://localhost:.../...`. Or the according IPv4 or IPv6 address, for 
> example
> +`127.0.0.1` (IPv4), or `[::1]` (IPv6). Access from an external machine is not
> +considered as localhost and thus restricted.
> 
> -Once shiro has been configured to perform user authentication it provides 
> -a good foundation on which to implement the [Jena 
> Permissions](../permissions/)
> -layer.  There is an [example implementation](../permissions/example.html) 
> -documented in the Jena Permissions section.  The Jena Permissions layer can 
> -be used to restrict access to specific graphs or triples within graphs.
> +Once Shiro has been configured to perform user authentication it provides  a
> +good foundation on which the [Jena Permissions](../permissions/) layer can be
> +configured. There is an [example implementation](../permissions/example.html)
> +documented in the Jena Permissions section. The Jena Permissions layer can be
> +used to restrict access to specific graphs or triples within graphs.
> 
> -There is an example to enable simple user/password security; this is only
> -suitable where the connection is secure, is shown `shiro.ini` file with
> -defaults user 'admin' and password 'pw'.  These should be changed before
> -use.
> +A simple example to enable basic user/password authentication is shown in the
> +default `shiro.ini` configuration. The default admin user is `admin` and the
> +password is `pw`. This can be changed directly in the INI file. Note that 
> this
> +setup is not recommended for production for various reasons  (no TLS, 
> passwords
> +in plain text etc.), consult the [Shiro
> +INI](https://shiro.apache.org/configuration.html#Configuration-INISections)
> +documentation for best practices.
> 
> -This has some use where the server is in a secure network environment with
> -additional restrictions on external requests also applied.  behind a
> -reverse proxy and the connection can have addition security (e.g. no access
> -to URLs starting '/$/').
> +As mentioned above, the default setup only restricts access to the admin 
> pages
> +of Fuseki. To avoid clashes with dataset names, the namespace of the admin
> +interface starts with '/$/', consult the [Fuseki HTTP Administration Protocol
> +](../fuseki2/fuseki-server-protocol.html) documentation for more details.
> 
> -The Apache Shiro website has documentation for creating more sophisticated
> -setups.
> +If access to SPARQL endpoints should be restricted, additional [Shiro
> +ACLs](https://shiro.apache.org/web.html#Web-WebINIconfiguration) are 
> necessary.
> +This is done in the `[urls]` section of the configuration. As an example,
> +restricting access to the `../query` SPARQL endpoint for all datasets on 
> Fuseki
> +could be done with this wildcard pattern:
> 
> -The security provided in Fuseki is not intended to replace existing
> -m

Re: RDF Diff/patch

[ajs6f]

I'm curious too, Claude. Is the idea that one assumes that bnodes are already 
using the same pool of labels, or something like that? IOW, if I have dataset1:

_:a a my:type .
_:b a my:type .

and dataset2:

_:c a my:type .

and I want to convert dataset1 into dataset2, will your algorithm delete both 
triples and add a new one, or just remove a triple, and if so, is that 
deterministic? If dataset2 is instead:

_:a a my:type .

will the algorithm only remove one triple and be done, or remove both and add a 
new one?

ajs6f

> On Dec 27, 2017, at 11:00 AM, Andy Seaborne  wrote:
> 
> It would be interesting to see especially the handling of blank nodes cycles 
> and other structures.
> 
> Please don't call it "RDF Patch" or a names similar to that - that term is 
> already used.
> 
>Andy
> 
> On 26/12/17 18:17, Claude Warren wrote:
>> Howdy,
>> I am working on a tool that can create UpdateRequests that will convert one
>> Dataset into another.
>> The basic idea is to extract the quads sorted by (g,s,p,o) and then perform
>> a diff on the lists (like a text diff but each quad is a "line").
>> The result is that I can create statements to delete insert and delete one
>> dataset to make it "identical" to the other.  Identical in this case means
>> that each model in the two datasets are isomorphic.
>> Is anyone else interested in this?
>> Claude

Re: CMS diff: Command-line and other tools for Jena developers

[ajs6f]

Committed, thanks!

ajs6f

> On Dec 27, 2017, at 8:04 PM, Albert Min  wrote:
> 
> Clone URL (Committers only):
> https://cms.apache.org/redirect?new=anonymous;action=diff;uri=http://jena.apache.org/documentation%2Ftools%2Findex.mdtext
> 
> Albert Min
> 
> Index: trunk/content/documentation/tools/index.mdtext
> ===
> --- trunk/content/documentation/tools/index.mdtext(revision 1819309)
> +++ trunk/content/documentation/tools/index.mdtext(working copy)
> @@ -21,7 +21,7 @@
> **On Windows**
> 
>   - `SET JENA_HOME =`*the directory you downloaded Jena to*
> -  - `SET PATH=%PATH%;% JENA_HOME%\bat`
> +  - `SET PATH=%PATH%;%JENA_HOME%\bat`
> 
> ### Running the Tools
> 
> 

Re: TDB Graph usage question

[ajs6f]

TIM uses a subclass of GraphView called GraphInMemory which is indeed also 
stateless.

ajs6f

> On Dec 30, 2017, at 5:54 AM, Andy Seaborne  wrote:
> 
> Have you tried TDB2 and TIM yet?
> 
> On 30/12/17 00:09, Claude Warren wrote:
>> When I tried to add a transaction on dest there was an error with a
>> transaction already  being active.
> 
> This is what I mean by details - you are now saying that graphs are from the 
> same dataset. That's an additional feature imposed by this use of the test. 
> I'm sure if source and est were the same graph, nasty things will happen. 
> (Elsewhere, GraphUtil works carefully for the cases when the graphs are the 
> same).
> 
> As a general Java comment on iterators over the same data structure:
> 
> Don't modify the datastructure while iterating.
> 
> You'll get either an exception (best case) or inconsistent results and 
> possibly differently inconsistent from run to run.
> 
>> Does the concurrent modification error indicate that the model was somehow
>> changed outside of an iterator?  That is where I am used to seeing it.
> 
> Graphs from dataset are stateless views of datasets. See the code in 
> GraphView.
> 
> (IIRC -- they are in TDB2, and I've been slowly working on improving the 
> situation for TDB1 without making a breaking change. TIM - I think it's OK.)
> 
>> The iterator call is in a recursive call to extractInfo().
>> Both graphs are in the same dataset.
>> all txnX() methods are of the form:
>> public static void txnBegin(Graph g) {
>> if (g.getTransactionHandler().transactionsSupported()) {
>> g.getTransactionHandler().begin();
>> }
>> }
> 
> 
> Unrelated to the iterator: The test code could be shorted if:
> 
> public static void txn(Graph g, Runnable action) {
>if ( g.getTransactionHandler().transactionsSupported() )
>g.getTransactionHandler().execute(action);
>else
>   action.run();
> }
> 
> then
>   txn(source, ()->
>  GraphExtract e = 
>  e.extractInto(dest, node("a"), source)
>   );
> 
> The try-catch stuff is handled by TransactionHandler.execute.
> 
> BTW You are lucky - you are passing graphs across transactions which was a 
> no-no in TDB1 until quite recently.
> 
> Graph transactions and dataset transactions used to be unconnected. Graph 
> transactions were written for the pre-dataset world.
> 
>Andy
> 
>> Stepping through the extract code there is this method:
>>  public void extractInto( Node root  )
>> {
>> active.add( root );
>> Iterator it = extractFrom.find( root, Node.ANY,
>> Node.ANY );
>> while (it.hasNext())
>> {
>> Triple t = it.next();
>> Node subRoot = t.getObject();
>> toUpdate.add( t );
>> if (! (active.contains( subRoot ) || b.stopAt( t )))
>> extractInto( subRoot );
>> }
>> }
>> }
>> When the toUpdate.add(t) is called the
>> DatasetControlMRSW$IteratorCheckNotConcurrent.eCount
>> is incremented.
>> I see in the code comments for IteratorCheckNotConcurrent that it expects
>> writer to be on a different thread than reader.  Is there a way around that
>> restriction?  I ask because all other graphs that I have tested work fine
>> when reading and writing on the same thread.
>> Thoughts?
>> Claude
>> On Fri, Dec 29, 2017 at 10:28 PM, Andy Seaborne  wrote:
>>> The exception says there is incorrect MRSW use of iterators in the test.
>>> 
>>> Iterator checking is very old code and I believe it works reliably - it is
>>> not guaranteed to always catch errors though.  That was never the claim.
>>> 
>>> Too many details missing to say much else but I don't see any txn on dest.
>>> 
>>> Andy
>>> 
>>> 
>>> On 29/12/17 13:10, Claude Warren wrote:
>>> 
>>>> fair enough.
>>>> 
>>>> I created a ContractTest for GraphTxnTDB and discovered a lot of issues
>>>> with transactions in the contract tests.  I cleaned most of them up but am
>>>> left with the following:
>>>> 
>>>>  /**
>>>>   * This test exposed that the update-existing-graph functionality was
>>>> broken
>>>>   * if the target graph already contained any statements with a
>>>> subject S
>>>>   * appearing as subject in the source gr

[GitHub] jena pull request #336: JENA-1458: Transaction promotion

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/336#discussion_r159162581
  
--- Diff: jena-arq/src/main/java/org/apache/jena/query/TxnType.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.query;
+
+import java.util.Objects;
+
+import org.apache.jena.sparql.JenaTransactionException;
+
+public enum TxnType {
+/** Transaction mode:
+ * 
+ * {@code WRITE}: this gaurantees a WRITE will complete if {@code 
commit()} is
--- End diff --

typo: `gaurantees` -> `guarantees`


---

[GitHub] jena pull request #336: JENA-1458: Transaction promotion

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/336#discussion_r159162566
  
--- Diff: jena-arq/src/main/java/org/apache/jena/query/TxnType.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.query;
+
+import java.util.Objects;
+
+import org.apache.jena.sparql.JenaTransactionException;
+
+public enum TxnType {
+/** Transaction mode:
+ * 
+ * {@code WRITE}: this gaurantees a WRITE will complete if {@code 
commit()} is
+ * called. The same as {@code begin(ReadWrite.WRITE)}.
+ * 
+ * {@code READ}: the transaction can not promote to WRITE,ensuring 
read-only
+ * access to the data. The same as {@code begin(ReadWrite.READ)}.
+ * 
+ * {@code READ_PROMOTE}: the transaction will go from "read" to 
"write" if the
+ * dataset has not been modified but if it has, the promotion fails 
with
+ * exception.
+ * 
+ * {@code READ_COMMITTED_PROMOTE}: Use this with care. The 
promotion will succeed but 
+ * changes from oher transactions become visible.
+ * 
+ * 
+ * Read committed: at the point transaction attempts promotion from 
"read" to
+ * "write", the sytem checks if the datset has chnage since the 
trsnaction started
+ * (called {@code begin}). If {@code READ_PROMOTE}, the dataset must 
not have
+ * changed; if {@code READ_COMMITTED_PROMOTE} anyh intermediate 
changes are
--- End diff --

typo : `anyh` -> `any`


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/337

JENA-1391: Dataset dyadic views, Model and Dataset Stream collectors

This is a bit preliminary-- probably needs some more tests. But I wanted to 
get it out there for early comments so I can take them into account and respond 
to criticism.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena JENA-1391

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/337.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #337


commit 9fd1f41870b730fcf5ea528b9fd27cd7a4ceb438
Author: ajs6f 
Date:   2017-10-01T14:02:28Z

First draft of Collector utilities

commit 197abd4e0addd131714a541b5a900cea719b2c37
Author: ajs6f 
Date:   2017-10-12T14:07:12Z

Refactoring to assume fresh Context for any Dataset view

commit 4d6d36969c502ad2d5df83c6324b3813993691ca
Author: ajs6f 
Date:   2017-11-03T18:06:22Z

Better use of preexisting Pair type, refactoring, introducing 
DifferenceDatasetGraph

commit 8416e028a8a77e44f351609f4056f0a032cfedd3
Author: ajs6f 
Date:   2017-11-03T19:25:04Z

Adding method impls to DatasetLib

commit ccf011aca6d5aef9186a510a53f3b32bb28d2982
Author: ajs6f 
Date:   2017-11-04T17:59:16Z

Adding IntersectionDatasetGraph

commit 53861bb36212da86f7e88815bf7a29ffbd47bc7e
Author: ajs6f 
Date:   2017-11-04T18:02:07Z

Cleaner threading of Context through constructors and statics

commit e5cc28ba852153057b37b6881c4ec1f48013de2d
Author: ajs6f 
Date:   2017-11-04T18:11:46Z

Cleanup between DatasetLib and DatasetCollector

commit de74807555a2efce4081eee2028cdc4e7f92504e
Author: ajs6f 
Date:   2017-11-05T14:13:32Z

Correcting whitespace

commit 272b3c10687b9e643f869d62055edec68d202e53
Author: ajs6f 
Date:   2017-11-05T14:15:04Z

Correcting type param names

commit d4a22357ec714ac2dde218d242a36bf7f1df7d2d
Author: ajs6f 
Date:   2017-11-17T17:05:18Z

Further simplifying new Collector API

commit 5aa0e26ca4d3dd46f77a9c993006b5cba1a6b178
Author: ajs6f 
Date:   2017-11-19T21:37:54Z

Minor refactoring

commit 9fd57ea0593d755ceaa9ae2a78283a4940cd8d48
Author: ajs6f 
Date:   2017-12-15T20:55:05Z

More tests

commit bbf0fc6c6ac78111ffd8827ac03a80d6ff687a7e
Author: ajs6f 
Date:   2017-12-18T14:43:10Z

Removing extra file

commit b90c14364e3355f66ab063c797e8c476515b0ef3
Author: ajs6f 
Date:   2017-12-29T14:03:02Z

Further test coverage

commit d1e6b90d9cedfe063ae088d3ec595f1b629827e8
Author: ajs6f 
Date:   2017-12-29T14:29:40Z

Further test coverage

commit ca28e9f77bc8bd4f1e6de34bd6cb7cd8d793f324
Author: ajs6f 
Date:   2017-12-29T15:04:36Z

Fixing license headers, more tests

commit dcd5cd6337f063e832bb9f569eb27da4d167814a
Author: ajs6f 
Date:   2017-12-29T15:08:26Z

Adding license header

commit 973c6dec5111bcc8a617e0ba8a27bbafba0c759c
Author: ajs6f 
Date:   2017-12-29T15:13:04Z

Adding license headers

commit 5e62aff38ea56a76af047451b546153c3b3fc76a
Author: ajs6f 
Date:   2017-12-29T15:37:28Z

Adding more public API

commit 6162bc0928d453bc193a1b28e8d5626610927376
Author: ajs6f 
Date:   2017-12-29T15:53:27Z

Cleanup in TIM

commit 8e9881f52e1d8c9b0ce62883d6485e367689fd7e
Author: ajs6f 
Date:   2017-12-29T16:33:55Z

Linking tests into build suite

commit 01ed060f71e577d41edf2c8501e155eafbd1bb56
Author: ajs6f 
Date:   2017-12-29T17:54:56Z

Beginning tests of dataset collectors

commit 5fa9101f383b7f86103024a3aa218754dc00ab8f
Author: ajs6f 
Date:   2017-12-31T18:45:09Z

Tests for UnionDatasetCollector

commit 9562e71fb05d5c9910e43457a51f21eeeb188721
Author: ajs6f 
Date:   2017-12-31T18:45:47Z

Add license headers

commit 446a6093885920b0a516187cc53aa7e4158ad70c
Author: ajs6f 
Date:   2017-12-31T19:19:06Z

Test for TestIntersectionDatasetCollector now passes

commit 8f725b6064820930a7299cd16f06a0bb40029ed8
Author: ajs6f 
Date:   2017-12-31T19:26:34Z

Stronger test for TestIntersectionDatasetCollector

commit 51df4d8c9b386e82edf8a6de98f13cfc917e1f9e
Author: ajs6f 
Date:   2017-12-31T19:29:11Z

License header

commit 086f9aa4a8486587699faf99f144da54718f2cfa
Author: ajs6f 
Date:   2017-12-31T19:32:17Z

Locking down Collector API

commit 1f794c7f6e1d05ed3cddcd3760cee3f601677bb9
Author: ajs6f 
Date:   2018-01-01T01:28:26Z

Correct behavior on Transational::abort

commit e1c09188f720a7308f3b7b0b0c8c313eefcd9657
Author: ajs6f 
Date:   2018-01-01T01:34:29Z

Better importing of throwNoMutationAllowed()




---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159250681
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/query/util/DatasetCollector.java ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.query.util;
--- End diff --

I'm 100% fine with not making a new one-- where might be a better place? 
Right in  `org.apache.jena.query`?


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159250915
  
--- Diff: jena-arq/src/main/java/org/apache/jena/sparql/util/Context.java 
---
@@ -199,13 +193,12 @@ public long getLong(Symbol symbol, long defaultValue) 
{
 }
 }
 
-public void putAll(Context other) {
+public Context putAll(Context other) {
 if ( readonly )
--- End diff --

I think that was leftover from when I was still trying to find a way to 
take context from both underlying datasets. Would you rather "fluent-ify" all 
such methods?


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159251025
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/util/ViewDatasetGraph.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.util;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.jena.atlas.iterator.Iter.count;
+import static org.apache.jena.atlas.iterator.Iter.map;
+import static 
org.apache.jena.ext.com.google.common.collect.Iterators.concat;
+import static org.apache.jena.graph.Node.ANY;
+import static org.apache.jena.query.ReadWrite.READ;
+import static org.apache.jena.sparql.core.Quad.defaultGraphIRI;
+import static org.apache.jena.sparql.util.graph.GraphUtils.triples2quads;
+
+import java.util.Iterator;
+
+import org.apache.jena.atlas.lib.Pair;
+import org.apache.jena.graph.Graph;
+import org.apache.jena.graph.Node;
+import org.apache.jena.graph.compose.MultiUnion;
+import org.apache.jena.query.ReadWrite;
+import org.apache.jena.shared.Lock;
+import org.apache.jena.sparql.core.DatasetGraph;
+import org.apache.jena.sparql.core.Quad;
+
+public abstract class ViewDatasetGraph extends 
Pair.OfSameType implements DatasetGraph {
+
--- End diff --

So `Pair.Dyadic`?


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159251106
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/util/ViewDatasetGraph.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.util;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.jena.atlas.iterator.Iter.count;
+import static org.apache.jena.atlas.iterator.Iter.map;
+import static 
org.apache.jena.ext.com.google.common.collect.Iterators.concat;
+import static org.apache.jena.graph.Node.ANY;
+import static org.apache.jena.query.ReadWrite.READ;
+import static org.apache.jena.sparql.core.Quad.defaultGraphIRI;
+import static org.apache.jena.sparql.util.graph.GraphUtils.triples2quads;
+
+import java.util.Iterator;
+
+import org.apache.jena.atlas.lib.Pair;
+import org.apache.jena.graph.Graph;
+import org.apache.jena.graph.Node;
+import org.apache.jena.graph.compose.MultiUnion;
+import org.apache.jena.query.ReadWrite;
+import org.apache.jena.shared.Lock;
+import org.apache.jena.sparql.core.DatasetGraph;
+import org.apache.jena.sparql.core.Quad;
+
+public abstract class ViewDatasetGraph extends 
Pair.OfSameType implements DatasetGraph {
+
+private Context context;
+
+private final Lock lock;
+
+public ViewDatasetGraph(DatasetGraph left, DatasetGraph right, Context 
c) {
+super(requireNonNull(left), requireNonNull(right));
--- End diff --

Can do.


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159251422
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/util/ViewDatasetGraph.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.util;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.jena.atlas.iterator.Iter.count;
+import static org.apache.jena.atlas.iterator.Iter.map;
+import static 
org.apache.jena.ext.com.google.common.collect.Iterators.concat;
+import static org.apache.jena.graph.Node.ANY;
+import static org.apache.jena.query.ReadWrite.READ;
+import static org.apache.jena.sparql.core.Quad.defaultGraphIRI;
+import static org.apache.jena.sparql.util.graph.GraphUtils.triples2quads;
+
+import java.util.Iterator;
+
+import org.apache.jena.atlas.lib.Pair;
+import org.apache.jena.graph.Graph;
+import org.apache.jena.graph.Node;
+import org.apache.jena.graph.compose.MultiUnion;
+import org.apache.jena.query.ReadWrite;
+import org.apache.jena.shared.Lock;
+import org.apache.jena.sparql.core.DatasetGraph;
+import org.apache.jena.sparql.core.Quad;
+
+public abstract class ViewDatasetGraph extends 
Pair.OfSameType implements DatasetGraph {
+
+private Context context;
+
+private final Lock lock;
+
+public ViewDatasetGraph(DatasetGraph left, DatasetGraph right, Context 
c) {
+super(requireNonNull(left), requireNonNull(right));
+this.context = requireNonNull(c);
+this.lock = new PairLock(left.getLock(), right.getLock());
+}
+
+protected static void throwNoMutationAllowed() {
+throw new UnsupportedOperationException("This view does not allow 
mutation!");
+}
+
+@Override
+public void commit() {
+throwNoMutationAllowed();
+}
+
+@Override
+public void begin(ReadWrite readWrite) {
+switch (readWrite) {
+case WRITE:
+throwNoMutationAllowed();
+case READ:
+forEach(dsg -> dsg.begin(READ));
--- End diff --

Yeah, I was wondering about that. I was trying to avoid `synchronized`. Do 
you think a new lock in `ViewDatasetGraph` would be more clear?



---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159251828
  
--- Diff: jena-base/src/main/java/org/apache/jena/atlas/lib/Pair.java ---
@@ -37,6 +40,30 @@
 public A car() { return a ; }
 public B cdr() { return b ; }
 
+public static class OfSameType extends Pair {
+
--- End diff --

I'm up for another design, as long as `ViewDatasetGraph` and `PairLock` 
stay reasonably clear and concise. A set of `static` methods could work fine 
here. I'm not sure what you mean by "even "X extends interface" for the "as 
boolean" nature?", can you explain that?


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159252167
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/util/ViewDatasetGraph.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.util;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.jena.atlas.iterator.Iter.count;
+import static org.apache.jena.atlas.iterator.Iter.map;
+import static 
org.apache.jena.ext.com.google.common.collect.Iterators.concat;
+import static org.apache.jena.graph.Node.ANY;
+import static org.apache.jena.query.ReadWrite.READ;
+import static org.apache.jena.sparql.core.Quad.defaultGraphIRI;
+import static org.apache.jena.sparql.util.graph.GraphUtils.triples2quads;
+
+import java.util.Iterator;
+
+import org.apache.jena.atlas.lib.Pair;
+import org.apache.jena.graph.Graph;
+import org.apache.jena.graph.Node;
+import org.apache.jena.graph.compose.MultiUnion;
+import org.apache.jena.query.ReadWrite;
+import org.apache.jena.shared.Lock;
+import org.apache.jena.sparql.core.DatasetGraph;
+import org.apache.jena.sparql.core.Quad;
+
+public abstract class ViewDatasetGraph extends 
Pair.OfSameType implements DatasetGraph {
+
--- End diff --

Oh, cool. Yes, the `View`ishness is really different than the 
`Pair`ishness. It's easy to imagine a view that isn't a pair or a pair that is 
more than a view. I'll go with `DyadicDatasetGraph`, since we already have, as 
you say, `Dyadic`.


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159252897
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/util/ViewDatasetGraph.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.util;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.jena.atlas.iterator.Iter.count;
+import static org.apache.jena.atlas.iterator.Iter.map;
+import static 
org.apache.jena.ext.com.google.common.collect.Iterators.concat;
+import static org.apache.jena.graph.Node.ANY;
+import static org.apache.jena.query.ReadWrite.READ;
+import static org.apache.jena.sparql.core.Quad.defaultGraphIRI;
+import static org.apache.jena.sparql.util.graph.GraphUtils.triples2quads;
+
+import java.util.Iterator;
+
+import org.apache.jena.atlas.lib.Pair;
+import org.apache.jena.graph.Graph;
+import org.apache.jena.graph.Node;
+import org.apache.jena.graph.compose.MultiUnion;
+import org.apache.jena.query.ReadWrite;
+import org.apache.jena.shared.Lock;
+import org.apache.jena.sparql.core.DatasetGraph;
+import org.apache.jena.sparql.core.Quad;
+
+public abstract class ViewDatasetGraph extends 
Pair.OfSameType implements DatasetGraph {
+
+private Context context;
+
+private final Lock lock;
+
+public ViewDatasetGraph(DatasetGraph left, DatasetGraph right, Context 
c) {
+super(requireNonNull(left), requireNonNull(right));
+this.context = requireNonNull(c);
+this.lock = new PairLock(left.getLock(), right.getLock());
+}
+
+protected static void throwNoMutationAllowed() {
+throw new UnsupportedOperationException("This view does not allow 
mutation!");
+}
+
+@Override
+public void commit() {
+throwNoMutationAllowed();
+}
+
+@Override
+public void begin(ReadWrite readWrite) {
+switch (readWrite) {
+case WRITE:
+throwNoMutationAllowed();
+case READ:
+forEach(dsg -> dsg.begin(READ));
--- End diff --

That's true. Okay, let's go with `synchronized` and if problems result, 
I'll deal with them, but they probably won't develop anyway. Most fears never 
give birth!


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159255082
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/query/util/DatasetCollector.java ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.query.util;
--- End diff --

I'll use `compose` for now and if more stuff accumulates, let's do a big 
`util`.


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159261113
  
--- Diff: jena-base/src/main/java/org/apache/jena/atlas/lib/Pair.java ---
@@ -37,6 +40,30 @@
 public A car() { return a ; }
 public B cdr() { return b ; }
 
+public static class OfSameType extends Pair {
+
--- End diff --

To be clear, it has to do with `Pair` insofar as it subclasses `Pair`. The 
idea is to
1. restrict `Pair` to `Pair`s with both elements of the same type and
2. offer some convenience methods that only make sense when both halves of 
a pair are the same type.

I've broken it out as a separate class, but I'm still up for turning it 
into a couple of functions. I just don't want to do that until I understand 
your third point better.


---

[GitHub] jena issue #335: Jena 1453 reduce docs

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/335
  
This is a naive question, so feel free to answer it as briefly as it 
deserves, but since for some instances reindexing may be a really considerable 
amount of work, would it be useful/possible to include a very simple migrator 
class that can strip out the no-longer-used fields and produce a new leaner 
text index that could then be swapped in?


---

[GitHub] jena issue #335: Jena 1453 reduce docs

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/335
  
@xristy That's what I meant-- I didn't mean to suggest an in-place op, 
sorry.


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159527929
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/util/ViewDatasetGraph.java ---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.sparql.util;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.jena.atlas.iterator.Iter.count;
+import static org.apache.jena.atlas.iterator.Iter.map;
+import static 
org.apache.jena.ext.com.google.common.collect.Iterators.concat;
+import static org.apache.jena.graph.Node.ANY;
+import static org.apache.jena.query.ReadWrite.READ;
+import static org.apache.jena.sparql.core.Quad.defaultGraphIRI;
+import static org.apache.jena.sparql.util.graph.GraphUtils.triples2quads;
+
+import java.util.Iterator;
+
+import org.apache.jena.atlas.lib.Pair;
+import org.apache.jena.graph.Graph;
+import org.apache.jena.graph.Node;
+import org.apache.jena.graph.compose.MultiUnion;
+import org.apache.jena.query.ReadWrite;
+import org.apache.jena.shared.Lock;
+import org.apache.jena.sparql.core.DatasetGraph;
+import org.apache.jena.sparql.core.Quad;
+
+public abstract class ViewDatasetGraph extends 
Pair.OfSameType implements DatasetGraph {
+
+private Context context;
+
+private final Lock lock;
+
+public ViewDatasetGraph(DatasetGraph left, DatasetGraph right, Context 
c) {
+super(requireNonNull(left), requireNonNull(right));
+this.context = requireNonNull(c);
+this.lock = new PairLock(left.getLock(), right.getLock());
+}
+
+protected static void throwNoMutationAllowed() {
+throw new UnsupportedOperationException("This view does not allow 
mutation!");
+}
+
+@Override
+public void commit() {
+throwNoMutationAllowed();
+}
+
+@Override
+public void begin(ReadWrite readWrite) {
+switch (readWrite) {
+case WRITE:
+throwNoMutationAllowed();
+case READ:
+forEach(dsg -> dsg.begin(READ));
--- End diff --

Done.


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159528705
  
--- Diff: jena-arq/src/main/java/org/apache/jena/sparql/util/Context.java 
---
@@ -199,13 +193,12 @@ public long getLong(Symbol symbol, long defaultValue) 
{
 }
 }
 
-public void putAll(Context other) {
+public Context putAll(Context other) {
 if ( readonly )
--- End diff --

Pulled out.


---

[GitHub] jena pull request #337: JENA-1391: Dataset dyadic views, Model and Dataset S...

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/337#discussion_r159529387
  
--- Diff: jena-base/src/main/java/org/apache/jena/atlas/lib/Pair.java ---
@@ -37,6 +40,30 @@
 public A car() { return a ; }
 public B cdr() { return b ; }
 
+public static class OfSameType extends Pair {
+
--- End diff --

Okay, I think I see where you are going-- let me try to work this out. 
Should be able to commit something before the weekend, if we don't get swamped 
by a blizzard. sigh!


---

[GitHub] jena pull request #336: JENA-1458: Transaction promotion

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/336#discussion_r159729634
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/core/mem/DatasetGraphInMemory.java
 ---
@@ -146,25 +162,84 @@ private void _begin(ReadWrite readWrite) {
 }
 
 /** Called transaction start code at most once per transaction. */ 
-private void startTransaction(ReadWrite mode) {
-transactionLock.enterCriticalSection(mode.equals(READ)); // get 
the dataset write lock, if needed.
-transactionType(mode);
+private void startTransaction(TxnType txnType, ReadWrite mode) {
+transactionLock.enterCriticalSection(mode.equals(ReadWrite.READ)); 
// get the dataset write lock, if needed.
+transactionType.set(txnType);
+transactionMode(mode);
 isInTransaction(true);
 }
 
 /** Called transaction ending code at most once per transaction. */ 
 private void finishTransaction() {
 isInTransaction.remove();
 transactionType.remove();
+transactionMode.remove();
 version.remove();
 transactionLock.leaveCriticalSection();
 }
  
+@Override
+public boolean promote() {
+if (!isInTransaction())
+throw new JenaTransactionException("Tried to promote outside a 
transaction!");
+if ( transactionMode().equals(ReadWrite.WRITE) )
+return true;
+
+boolean readCommitted;
+// Initial state
+switch(transactionType.get()) {
+case WRITE :
+return true;
+case READ :
+throw new JenaTransactionException("Tried to promote READ 
transaction");
+case READ_COMMITTED_PROMOTE :
+readCommitted = true;
+case READ_PROMOTE :
+readCommitted = false;
+// Maybe!
+break;
+default:
+throw new NullPointerException();
+}
+
+try {
+_promote(readCommitted);
+return true;
+} catch (JenaTransactionException ex) {
+return false ;
+}
+}
+
+private void _promote(boolean readCommited) {
+//System.err.printf("Promote: version=%d generation=%d\n", 
version.get() , generation.get()) ;
+
+// Outside lock.
+if ( ! readCommited && version.get() != generation.get() )  {
--- End diff --

Just checking that I understand the algo here...

1. Every transaction pulls a version number from the current generation 
number.
2. A commit increments the generation number.
3. When a transaction attempts to promote, it checks to see that the 
generation hasn't incremented out from under it (indicating that some other 
transaction committed since this one began). If that happened, this transaction 
will throw an exception as shown below.
4. Otherwise, it's okay to promote.

So two questions:
1. Is that understanding accurate?
2. Is that essentially how this is implemented in other dataset impls, or 
do I need to go read them as well?


---

[GitHub] jena pull request #336: JENA-1458: Transaction promotion

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/336#discussion_r159748796
  
--- Diff: 
jena-arq/src/main/java/org/apache/jena/sparql/core/mem/DatasetGraphInMemory.java
 ---
@@ -146,25 +162,84 @@ private void _begin(ReadWrite readWrite) {
 }
 
 /** Called transaction start code at most once per transaction. */ 
-private void startTransaction(ReadWrite mode) {
-transactionLock.enterCriticalSection(mode.equals(READ)); // get 
the dataset write lock, if needed.
-transactionType(mode);
+private void startTransaction(TxnType txnType, ReadWrite mode) {
+transactionLock.enterCriticalSection(mode.equals(ReadWrite.READ)); 
// get the dataset write lock, if needed.
+transactionType.set(txnType);
+transactionMode(mode);
 isInTransaction(true);
 }
 
 /** Called transaction ending code at most once per transaction. */ 
 private void finishTransaction() {
 isInTransaction.remove();
 transactionType.remove();
+transactionMode.remove();
 version.remove();
 transactionLock.leaveCriticalSection();
 }
  
+@Override
+public boolean promote() {
+if (!isInTransaction())
+throw new JenaTransactionException("Tried to promote outside a 
transaction!");
+if ( transactionMode().equals(ReadWrite.WRITE) )
+return true;
+
+boolean readCommitted;
+// Initial state
+switch(transactionType.get()) {
+case WRITE :
+return true;
+case READ :
+throw new JenaTransactionException("Tried to promote READ 
transaction");
+case READ_COMMITTED_PROMOTE :
+readCommitted = true;
+case READ_PROMOTE :
+readCommitted = false;
+// Maybe!
+break;
+default:
+throw new NullPointerException();
+}
+
+try {
+_promote(readCommitted);
+return true;
+} catch (JenaTransactionException ex) {
+return false ;
+}
+}
+
+private void _promote(boolean readCommited) {
+//System.err.printf("Promote: version=%d generation=%d\n", 
version.get() , generation.get()) ;
+
+// Outside lock.
+if ( ! readCommited && version.get() != generation.get() )  {
--- End diff --

Yeah, I knew this stuff was here but I didn't read it carefully when you 
put it in months ago because we weren't turning it on. Okay, I'm :+1: . Let 'er 
rip!


---

[GitHub] jena issue #337: JENA-1391: Dataset dyadic views, Model and Dataset Stream c...

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/337
  
I'll let this hang out until EOD today in case anyone else wants to opine. 
Happy new year everyone!


---

[GitHub] jena pull request #338: Removing very old deprecated types

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/338

Removing very old deprecated types

Just some cleanup in `jena-base` and `jena-core`. All of the few types 
being removed have been publicly deprecated for at least two years.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena RemovingVeryOldDeprecatedTypes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/338.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #338


commit e7cd4600e70a139a86d7450704816cc34ef58539
Author: ajs6f 
Date:   2018-01-05T16:30:46Z

Removing types from jena-base that have been deprecated for more than two 
years

commit ff5ba86d058fa1c30c8e31ec2534b9c79b90a823
Author: ajs6f 
Date:   2018-01-05T16:35:35Z

Removing types from jena-core that have been deprecated for more than two 
years




---

[GitHub] jena issue #338: Removing very old deprecated types

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/338
  
I'll get something in Jira, for sure, and I'll wait to merge this to hear 
from @Claudenw .


---

Re: PROV vocabulary class

[ajs6f]

This list doesn't permit attachments. Can you please send your work as a PR on 
Github, or failing that, as a gist or pastebin or the like?

Adam Soroka

> On Jan 7, 2018, at 12:00 PM, Cotton Franck  wrote:
> 
> Hello
> 
> For my own needs, I created this class corresponding to the W3C PROV 
> vocabulary. It could fit in the org.apache.jena.vocabulary package. Please 
> fill free to add the relevant license text, I don't claim any IP on the class.
> 
> Cheers
> Franck

Re: PROV vocabulary class

[ajs6f]

Not really, but it's not unheard of. See:

https://github.com/apache/jena/blob/master/jena-core/src/main/java/org/apache/jena/vocabulary/DC_11.java

The Model isn't public, but it is available from most of the public fields via 
RDFNode::getModel.

What can be done instead is not have a Model involved and instead use 
ResourceFactory.

ajs6f

> On Jan 7, 2018, at 8:09 PM, Adam Jacobs  wrote:
> 
> Is it a good idea to have a public mutable model?
> 
> 
> 
> 
> From: Cotton Franck 
> Sent: Sunday, January 7, 2018 11:47 AM
> To: dev@jena.apache.org
> Subject: RE:PROV vocabulary class
> 
> Hi Adam. Here: 
> https://gist.github.com/FranckCo/889d1f137dc412ceab214159a49375e2
> 
> Franck
> 
> De : ajs6f [aj...@apache.org]
> Envoyé : dimanche 7 janvier 2018 18:07
> À : dev@jena.apache.org
> Objet : Re: PROV vocabulary class
> 
> This list doesn't permit attachments. Can you please send your work as a PR 
> on Github, or failing that, as a gist or pastebin or the like?
> 
> Adam Soroka
> 
>> On Jan 7, 2018, at 12:00 PM, Cotton Franck  wrote:
>> 
>> Hello
>> 
>> For my own needs, I created this class corresponding to the W3C PROV 
>> vocabulary. It could fit in the org.apache.jena.vocabulary package. Please 
>> fill free to add the relevant license text, I don't claim any IP on the 
>> class.
>> 
>> Cheers
>> Franck
> 

[GitHub] jena issue #339: resolve JENA-1459 add jena text highlighting

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/339
  
It's no big deal-- happens all the time! Do you know how to rebase and 
resolve conflicts in Git?


https://help.github.com/articles/resolving-merge-conflicts-after-a-git-rebase/

https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/


---

[GitHub] jena issue #339: resolve JENA-1459 add jena text highlighting

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/339
  
Cool. That's essentially an in-line version of the same workflow, minus the 
rebase. That means you end up with an extra no-real-content commit 
(https://github.com/apache/jena/pull/339/commits/c9453b4c835ac7ab3eb9b0c9bcf8857d11b1a8b8)
 but you get where you need to be.


---

[GitHub] jena issue #339: resolve JENA-1459 add jena text highlighting

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/339
  
Relax-- it's not nearly as bad as all that. :grin: I often say that having 
to rebase PRs a lot is good-- it means the project is lively and the code is 
evolving.

For me, the trick (as almost always with git) was to remember that you are 
making/managing/corralling _deltas_ (aka commits), not versions. A branch can 
be thought of as a chain of deltas that starts somewhere and ends with a 
specific commit. Rebasing means taking a chain of commits that ends one branch 
and swapping what comes before it to be another branch. So in this case, you 
would have swapped the prefix of your branch (which has all the commits in 
master _except_ those that were in #335) for master itself. You would have had 
to do exactly the same adjustments, but you would have ended up with a series 
of commits that appeared (from the POV of seeing how deltas add up to change) 
as though you had begun work after #335 merged, which makes for a cleaner 
public history. Of course your commits always keep their metadata, so we will 
always actually know when they occurred in clock time.


---

[GitHub] jena issue #339: resolve JENA-1459 add jena text highlighting

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/339
  
That is one prob with using the Github on-line editors. But they _are_ 
convenient! I use them for corrections to human-centric docs like READMEs, etc.


---

Re: riot/jena - limiting CPU and memory usage

[ajs6f]

Can you please tell us a bit about the machine? How many CPUs does it have / 
how many cores? What are you actually doing with riot? (riot can't generate 
RDF. It only translates it, so what are the original files from which you are 
working.)

NTriples is almost always the least resource-intensive output format.

ajs6f

> On Jan 19, 2018, at 4:14 PM, Erman Korkut (BLOOMBERG/ 120 PARK) 
>  wrote:
> 
> Hi all,
> 
> I was running riot to generate ttl files with 50+ millions triples in it in a 
> production machine and it took over the entire cpu resources of that box, 
> basically taking down the entire machine. Is there a way to limit 
> parallelization that it leverages and/or cpu (and memory usage) and alike? 
> 
> I am going to be using cgroups (a linux kernel feature) to limit resources 
> allocated to riot processes, but although I know it is a long shot, I wanted 
> to check if there is a similar setting in riot/jena itself.
> 
> Thanks,
> Erman

Re: FILTER EXISTS evaluation of different triple stores

[ajs6f]

There is a W3C Community Group discussing related issues, chaired by Andy. 
Unfortunately, I think progress has stalled on disagreements as to the best 
road forward.

ajs6f

> On Jan 28, 2018, at 5:56 AM, Lorenz B.  wrote:
> 
> Hi all,
> 
> just if somebody is interested in reading about the different behavior
> of triple stores when evaluating FILTER EXISTS. [1]
> 
> Not sure how often this feature is used nowadays, but anyways
> interesting to know ( though I'd never write the SPARQL query as given
> in the example of the paper, seems odd)
> 
> 
> Cheers,
> 
> Lorenz
> 
> [1] https://arxiv.org/pdf/1801.04387.pdf
> 

[GitHub] jena pull request #349: Cleanup comments and code around txn

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/349#discussion_r164806834
  
--- Diff: 
jena-tdb/src/main/java/org/apache/jena/tdb/store/GraphNonTxnTDB.java ---
@@ -32,25 +34,58 @@
  * @see GraphTxnTDB
  */
 public class GraphNonTxnTDB extends GraphTDB implements Closeable, Sync {
-private final DatasetGraphTDBdataset ;
+private final DatasetGraphTDBdataset;
 
 public GraphNonTxnTDB(DatasetGraphTDB dataset, Node graphName) {
-super(dataset, graphName) ;
-this.dataset = dataset ;
+super(dataset, graphName);
+this.dataset = dataset;
 }
 
 @Override
 public DatasetGraphTDB getDatasetGraphTDB() {
-return dataset ;
+return dataset;
 }
 
 @Override
 protected DatasetGraphTDB getBaseDatasetGraphTDB() {
-return dataset ;
+return dataset;
 }
 
 @Override
 public TransactionHandler getTransactionHandler() {
-return new TransactionHandlerTDB(this) ;
+return new TransactionHandlerTDB(this);
+}
+
+// Transaction handler for non-transactional use.
+// Does not support transactions, but syncs on commit which is the 
best it
+// can do without being transactional, which is striongly preferrerd.
+// For backwards compatibility only.
+private static class TransactionHandlerTDB extends 
TransactionHandlerBase //implements TransactionHandler 
+{
+private final Graph graph;
+
+public TransactionHandlerTDB(GraphTDB graph) {
+this.graph = graph ;
+}
+
+@Override
+public void abort() {
+// Not the Jena old-style transaction interface
--- End diff --

Not quite sure what this comment means-- is there a specific type you are 
mentioning?


---

Re: Welcome Chris!

[ajs6f]

Amen! I'm looking forward to your next PR!

ajs6f

> On Feb 1, 2018, at 3:40 AM, Osma Suominen  wrote:
> 
> Welcome Chris! Great to have you as a committer. You've done excellent work 
> on jena-text recently.
> 
> -Osma
> 
> 
> Andy Seaborne kirjoitti 01.02.2018 klo 00:11:
>> We're pleased to announce that Chris Tomlinson (codeferret@) has become a 
>> committer for Apache Jena.
>> Welcome Chris!
>> Andy
>> on behalf of the Apache Jena PMC
> 
> 
> -- 
> Osma Suominen
> D.Sc. (Tech), Information Systems Specialist
> National Library of Finland
> P.O. Box 26 (Kaikukatu 4)
> 00014 HELSINGIN YLIOPISTO
> Tel. +358 50 3199529
> osma.suomi...@helsinki.fi
> http://www.nationallibrary.fi

Re: Building with Java9

[ajs6f]

Ditto (haven't been paying much attention).

A six month cycle is pretty fast, in some ways. At least it will be if we have 
to do a lot of work for each version.

ajs6f

> On Feb 5, 2018, at 11:34 AM, Andy Seaborne  wrote:
> 
> I hadn't grok'ed (or, to be honest, paid attention to) the details:
> 
> http://blog.joda.org/2018/02/java-9-has-six-weeks-to-live.html
> 
> Java 8 LTS -> Java 11 LTS (Sep 2018)
> 
> then only four months until end of LTS for Java8.
> 
>Andy

Re: Building with Java9

[ajs6f]

Works for me. I think it's good to take as many steps as we can to do this (= 
make each one as small as we can).

ajs6f

> On Feb 6, 2018, at 8:00 AM, Andy Seaborne  wrote:
> 
> Would there be any problems if the plugin settings for a building wither 
> Java9 were put into master? (not building _for_ java9).
> 
> One thing needed is a minimum version of maven of 3.5.0 (3.5.2 is the current 
> latest).
> 
> If that's OK, the Jenkins job can be activated to build for java8 with java9 
> for the -Pdev build to ensure no bad stuff gets in.
> 
>Andy
> 
> On 02/02/18 18:02, Andy Seaborne wrote:
>> JENA-1475
>> I've managed to the build working with java9, outputting java8 compatible 
>> classes.
>> PR#350.
>> https://github.com/apache/jena/pull/350
>> This is using Apache parent v18 (so not JENA-1474)
>> but
>> 1/ -Pdev build
>> Works:
>>   mvn clean install -Pdev
>> because it sets -Dmaven.javadoc.skip=true
>> 2/Full build, jaavdoc
>> javadoc fails at jena-core (despite it compiled). The problem is 
>> java.xml.bind - it looks to me like the javadoc is running in "java9" mode 
>> and I haven't found out how to override that.
>> 3/ Full build, no javadoc
>>   mvn clean install -Dmaven.javadoc.skip=true
>> runs until jena-elephas-common.
>> [ERROR] Failed to execute goal on project jena-elephas-common:
>> Could not resolve dependencies for project 
>> org.apache.jena:jena-elephas-common:jar:3.7.0-SNAPSHOT:
>> Could not find artifact jdk.tools:jdk.tools:jar:1.6 at specified path 
>> /usr/lib/jvm/java-9-openjdk-amd64/../lib/tools.jar -> [Help 1]
>> which comes from
>> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.6.0:provided
>> [INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0:provided
>> [INFO] |  |  \- jdk.tools:jdk.tools:jar:1.6:system
>> Andy

Re: Building with Java9

[ajs6f]

Nope, not using anything from Java 9, myself.

But I haven't looked that carefully at library changes. (I am looking forward 
to private members on interfaces.) I don't personally know anyone using Jigsaw 
(the new module system), and to my (low level of) understanding, it isn't 
really meant for applications (at least not yet), it's meant to modularize the 
Java platform itself. When I need an application modularity framework right 
now, I'm still using the tried-and-true OSGi.

Lorenz, are you hinting at the possibility that we just go from LTS to LTS? 
Because that has crossed my mind...

ajs6f

> On Feb 6, 2018, at 2:57 AM, Lorenz B.  wrote:
> 
> Well, I read about this some time ago and was wondering how many people
> are really aware of the fact that no more security updates will be
> delivered 6 month after the release of a new version. Most people in our
> group are still working on Java 8, if not necessary probably nobody here
> will upgrade until the next LTS.
> 
> Is anybody of you using features from Java 9? The only things that I
> find interesting are some extended Stream functions like takeWhile,
> dropWhile, etc. Who is using the modularity feature?
> 
> 
> Cheers,
> 
> Lorenz
> 
> 
> On 05.02.2018 17:41, ajs6f wrote:
>> Ditto (haven't been paying much attention).
>> 
>> A six month cycle is pretty fast, in some ways. At least it will be if we 
>> have to do a lot of work for each version.
>> 
>> ajs6f
>> 
>>> On Feb 5, 2018, at 11:34 AM, Andy Seaborne  wrote:
>>> 
>>> I hadn't grok'ed (or, to be honest, paid attention to) the details:
>>> 
>>> http://blog.joda.org/2018/02/java-9-has-six-weeks-to-live.html
>>> 
>>> Java 8 LTS -> Java 11 LTS (Sep 2018)
>>> 
>>> then only four months until end of LTS for Java8.
>>> 
>>>   Andy
> 

Re: CMS diff: Jena RDF/XML How-To

[ajs6f]

Committed, thank you!

ajs6f

> On Feb 6, 2018, at 11:46 AM, Ian Dunlop  wrote:
> 
> Clone URL (Committers only):
> https://cms.apache.org/redirect?new=anonymous;action=diff;uri=http://jena.apache.org/documentation%2Fio%2Frdfxml_howto.mdtext
> 
> Ian Dunlop
> 
> Index: trunk/content/documentation/io/rdfxml_howto.mdtext
> ===
> --- trunk/content/documentation/io/rdfxml_howto.mdtext(revision 
> 1823266)
> +++ trunk/content/documentation/io/rdfxml_howto.mdtext(working copy)
> @@ -250,7 +250,7 @@
> properties so that it behaves exactly as required in your application,
> and then to do all subsequent I/O through it.
> 
> -Model m = Modelfactory.createDefaultModel();
> +Model m = ModelFactory.createDefaultModel();
> RDFWriter writer = m.getRDFWriter();
> m = null; // m is no longer needed.
> writer.setErrorHandler(myErrorHandler);
> @@ -342,7 +342,7 @@
> data that uses unqualified RDF attributes such as "about" instead
> of "rdf:about", then the following code is appropriate:
> 
> -Model m = Modelfactory.createDefaultModel();
> +Model m = ModelFactory.createDefaultModel();
> RDFReader arp = m.getReader();
> m = null; // m is no longer needed.
> // initialize arp
> @@ -379,7 +379,7 @@
> 
> InputStream in = ... ;
> String baseURI = ... ;
> -Model model = Modelfactory.createDefaultModel();
> +Model model = ModelFactory.createDefaultModel();
> RDFReader r = model.getReader("RDF/XML");
> r.setProperty("iri-rules", "strict") ;
> r.setProperty("error-mode", "strict") ; // Warning will be errors.
> 

Re: merging pull requests

[ajs6f]

I'd suggest the wiki for now, but that's just my preference.

In related news, I haven't heard any more about Gitbox and reversing the 
direction of mirroring from INFRA. Some may recall that we ask about that a few 
months ago. With a reversed mirroring, we would could use Github's web UI to do 
merges (as Claude was just expecting to do) and the repo would get mirrored 
back to Apache. The cost would be that we lose our current convenient Jira 
integration, although INFRA is aware of that and is interested in working on 
that problem.

ajs6f

> On Feb 7, 2018, at 10:37 AM, Andy Seaborne  wrote:
> 
> I feel bad this isn't documented properly so here is a first draft to rectify 
> that.
> 
> We can put it on the confluence wiki or have a folder in git for such docs.  
> If the preference is git, maybe migrate the release process there as well 
> (sometime, somehow)?
> 
> Anyone got other tips to add?
> 
>Andy
> 
> # Setup in .git/config (in your clone of the Apache repo)
> ---
> [user]
>   name = ...
>   email = 
> 
> [remote "github"]
>   url = https://github.com/apache/jena.git
>   fetch = +refs/heads/*:refs/remotes/github/*
> 
> 
> # Pull from GH, push to ASF
> 
> In a terminal (it's going to fire off vi)
> 
>cd YourApacheJenaClone
> 
> Pull from the GH mirror into the local clone of the ASF primary repo.
> 
>git pull github pull/NNN/head --no-ff
> 
> "github" -- taken from .git/config.. You can use the full URL.
> "--no-ff" -- No fast forward.
> This will stop git skipping asking for a commit comment.
> 
> When the comment edit comes up:
> 
> Put "JENA-: " on the front of the first line (the merge commit comment)
> 
> Add this line in the comment body:
> 
>This closes #NNN.
> 
> which will close the PR when the repo sync up.
> 
> # Check:
> 
>mvn clean install -Pdev
> 
> or whatever is appropriate.
> 
> # Push to Apache.
> git push
> 
> # Finish up.
> 
> Wait a few moments for them to sync.  It is pretty quick.
> 
> You can now:
> 
> 1/ Resolve the JIRA
> 
> 2/ close the branch - go to the PR, which is now under "closed", and click 
> "delete branch".  This delete the PR branch from your GH cloned repo.
> 
> 3/ "git pull -p" to pull from your GH clone and prune tracked branches.
> 
> 4/ "git branch -d 
> 
> 
> On 07/02/18 09:52, Rob Vesse wrote:
>> This is normal because GitHub is a mirror of the Apache repository. If you 
>> check the email archive for the original notification of the pull request it 
>> includes instructions on the necessary steps to merge
>> Rob
>> On 07/02/2018, 07:39, "Claude Warren"  wrote:
>> Greetings,
>>  I was looking at the git hub repository online and did not see a 
>> way to
>> merge the pull request from the web.  Is this normal (i.e. we always 
>> merge
>> on local machine and push) or do I just not have permissions, or am I
>> looking at the wrong place?
>>  thx,
>> Claude
>>  --
>> I like: Like Like - The likeliest place on the web
>> <http://like-like.xenei.com>
>> LinkedIn: http://www.linkedin.com/in/claudewarren
>> 

Re: [08/10] jena git commit: Merge branch 'master' into FixGraphContractTestTransactionUsage

[ajs6f]

+1!

My habit long ago became to rebase constantly while working up a PR (and 
working on feedback) and rebasing right up until I merge it. It makes the 
public history much easier to understand, and because Git keeps reasonable 
metadata on all commits, it doesn't "hide" or destroy any genuinely useful info.


ajs6f

> On Feb 8, 2018, at 9:51 AM, Andy Seaborne  wrote:
> 
> Claude,
> 
> I think using "git rebase" rather than "git merge" would have made this 
> cleaner. Rebase undoes the differences, merges cleanly the other branch (fast 
> forward) then redoes the differences.
> 
>Andy
> 
> On 07/02/18 20:30, cla...@apache.org wrote:
>> Merge branch 'master' into FixGraphContractTestTransactionUsage
>> Project: http://git-wip-us.apache.org/repos/asf/jena/repo
>> Commit: http://git-wip-us.apache.org/repos/asf/jena/commit/567f4053
>> Tree: http://git-wip-us.apache.org/repos/asf/jena/tree/567f4053
>> Diff: http://git-wip-us.apache.org/repos/asf/jena/diff/567f4053
>> Branch: refs/heads/master
>> Commit: 567f40531b02ea4808a51837ae8005b98da76886
>> Parents: 02ae269 568b822
>> Author: Claude Warren 
>> Authored: Sun Feb 4 11:05:28 2018 +
>> Committer: Claude Warren 
>> Committed: Sun Feb 4 11:05:28 2018 +
>> --
>>  .../java/org/apache/jena/riot/RDFLanguages.java | 16 +++--
>>  .../jena/sparql/algebra/table/TableBase.java|  2 +
>>  .../jena/sparql/algebra/table/TableEmpty.java   |  4 ++
>>  .../jena/sparql/algebra/table/TableUnit.java|  9 ++-
>>  .../sparql/graph/TransactionHandlerNull.java|  2 +-
>>  .../jena/sparql/sse/writers/WriterOp.java   |  9 ++-
>>  .../java/org/apache/jena/riot/TestLangRIOT.java |  5 ++
>>  .../apache/jena/sparql/algebra/TS_Algebra.java  |  3 +-
>>  .../jena/sparql/algebra/TestOpAsQuery.java  | 26 +++-
>>  .../apache/jena/sparql/algebra/TestTable.java   | 68 
>>  .../algebra/optimize/TestTransformFilters.java  |  2 +-
>>  .../transaction/AbstractTestTransPromote.java   | 26 
>>  .../AbstractTestTransactionLifecycle.java   | 24 ++-
>>  .../main/java/org/apache/jena/tdb2/TDB2.java|  3 +
>>  .../jena/tdb2/store/GraphViewSwitchable.java|  5 --
>>  .../org/apache/jena/fuseki/async/AsyncPool.java | 25 +--
>>  .../src/main/java/org/apache/jena/tdb/TDB.java  |  3 +
>>  .../jena/tdb/graph/TransactionHandlerTDB.java   | 62 --
>>  .../apache/jena/tdb/store/GraphNonTxnTDB.java   | 56 
>>  .../tdb/transaction/TransactionManager.java | 59 ++---
>>  20 files changed, 269 insertions(+), 140 deletions(-)
>> --

[GitHub] jena pull request #358: JENA-1484: Correcting behavior in ApplyElementTransf...

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/358

JENA-1484: Correcting behavior in ApplyElementTransformVisitor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena JENA-1484

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/358.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #358


commit c706ea4a7792221aa83bc4d5a9600ca4be89c4d2
Author: ajs6f 
Date:   2018-02-10T19:14:12Z

JENA-1484: Correcting behavior in ApplyElementTransformVisitor




---

Re: [DRAFT] Apache Jena and Java9

[ajs6f]

LGTM, with one suggestion: when discussing point [4] modules, maybe use the 
terms "JPMS" or Jigsaw or "new Java Platform Module System" or something to be 
very explicit. Some folks pay closer attention than others to "the state of the 
Java" and it may not be obvious to some what we mean .

ajs6f

> On Feb 11, 2018, at 1:39 PM, Andy Seaborne  wrote:
> 
> I thought that some text about the status of Jena and Java9 would be useful 
> to users@ to state the current state. This could go in the 3.7.0 release 
> notes.
> 
>Andy
> 
> DRAFT:: For discussion
> 
> -
> Java8 is the current LTS version of Java and receives security updates. The 
> next LTS is Java11. [Sched]
> 
> [Sched] http://www.oracle.com/technetwork/java/eol-135779.html
> 
> There are different cases for running with Java9:
> 
> [1] Run on a Java9 JVM
> [2] Build using Java9 JDK, output Java8 classfiles.
> [3] Java9 required - language features and runtime library
> [4] Java9 modules
> 
> 
> [1] at Jena 3.7.0, running with a Java9 platform (except as noted below) is 
> supported.
> 
> [2] at Jena 3.7.0, running maven using Java9 to produce Java8 bytecode is 
> supported (expect as noted).  We now run a Jenkins job daily to check this 
> for the main jars.
> 
> [3] The project has no current plans to require a Java9 language runtime. The 
> language requirement is still Java8.
> 
> [4] Jena jars can work as automatic modules but Jena itself does not provide 
> a "modules" version. Proper migration needs all the dependencies to be 
> modules, and also has implications on the upstream. Contribution and 
> discussions are welcome!
> 
> For discussion and background see
> 
> [A] http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html
> [B] http://blog.joda.org/2017/04/java-se-9-jpms-module-naming.html
> 
> 
> And note: Java 9 is obsolete March 2018, as soon as java10 comes out.
> http://openjdk.java.net/projects/jdk/10/#Schedule
> 
> Notes:
> 
> jena-elephas depends on Hadoop that depends on jdk.tools that is not 
> available in Java9.

[GitHub] jena issue #359: Fixes for transactions after JENA-1458

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/359
  
I meant code review. (I have no problem believing that I overlooked them, 
just surprised that everyone did.)


---

[GitHub] jena pull request #362: JENA-1389: Return 'this' rather than 'void' from Dat...

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/362

JENA-1389: Return 'this' rather than 'void' from Dataset



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena JENA-1389

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/362.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #362


commit d21aba27b73d3340e595f22b2ae58f7019b455aa
Author: ajs6f 
Date:   2018-02-15T15:47:37Z

JENA-1389: Return 'this' rather than 'void' from Dataset




---

[GitHub] jena pull request #361: Small improvements

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/361#discussion_r168523058
  
--- Diff: jena-tdb/src/main/java/org/apache/jena/tdb/solver/BindingTDB.java 
---
@@ -120,6 +121,8 @@ public Node get1(Var var)
 if ( id == null )
 return null ; 
 n = nodeTable.getNodeForNodeId(id) ;
+if ( n == null )
+throw new TDBException("No node in NodeTable for NodeId 
"+id);
--- End diff --

Do we know all/most of the circumstances that might cause this? Is it 
basically an interrupted write? If so, it might be worth including some error 
logging with that hint, but if there are more than one or two potential causes, 
never mind. I mention it because getting from "a missing entry in the 
`NodeTable`" to "that was the aftereffect of a failed write" is going to bring 
some users to the list when they might be able to figure out what happened with 
a simple hint.


---

[GitHub] jena pull request #361: Small improvements

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/361#discussion_r168644094
  
--- Diff: jena-tdb/src/main/java/org/apache/jena/tdb/solver/BindingTDB.java 
---
@@ -120,6 +121,8 @@ public Node get1(Var var)
 if ( id == null )
 return null ; 
 n = nodeTable.getNodeForNodeId(id) ;
+if ( n == null )
+throw new TDBException("No node in NodeTable for NodeId 
"+id);
--- End diff --

Right-o, sounds like it's much better to leave it be.


---

[GitHub] jena pull request #363: JENA-1491: Migrate FindBugs to StopBugs

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/363

JENA-1491: Migrate FindBugs to StopBugs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena JENA-1491

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/363.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #363


commit 8cc26701f842584c16d2120f2e0013e24482c1cc
Author: ajs6f 
Date:   2018-02-16T21:31:25Z

JENA-1491: Migrate FindBugs to StopBugs




---

[GitHub] jena pull request #367: JENA-1391: Add methods for ModelCollectors to API in...

[ajs6f]

GitHub user ajs6f opened a pull request:

https://github.com/apache/jena/pull/367

JENA-1391: Add methods for ModelCollectors to API in ModelUtils



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajs6f/jena JENA-1391b

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/367.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #367


commit b74d3775f180c1077669ba7084eab3e8ef363cbe
Author: ajs6f 
Date:   2018-03-01T14:13:35Z

JENA-1391: Add methods for ModelCollectors to API in ModelUtils




---

[GitHub] jena pull request #368: JENA-1495 Return Model from PrefixMapping methods.

[ajs6f]

Github user ajs6f commented on a diff in the pull request:

https://github.com/apache/jena/pull/368#discussion_r172049845
  
--- Diff: jena-core/src/main/java/org/apache/jena/rdf/model/Model.java ---
@@ -1051,4 +1051,12 @@ Remove all the statements matching (s, p, o) from 
this model.
 Answer true iff .close() has been called on this Model.
 */
 public boolean isClosed();
+
+// Override return type for methods inherited from PrefixMapping
--- End diff --

We should use `@Override` for explicitness on these changes.


---

[GitHub] jena issue #368: JENA-1495 Return Model from PrefixMapping methods.

[ajs6f]

Github user ajs6f commented on the issue:

https://github.com/apache/jena/pull/368
  
This seems like a useful change, but it is not back-compatible, so in 
theory we should hold it for a major version release.


---