[jira] [Commented] (OAK-8696) [Direct Binary Access] upload algorithm documentation should also be on the website

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983924#comment-16983924
 ] 

Alexander Klimetschek commented on OAK-8696:


Link is fine if it stays up to date.

However, to me as a developer using Oak, the javadoc is actually the place that 
is duplicated (different versions, urls have changed and broke). To most 
readers, only the published javadoc counts, not the source code. So I think 
from the perspective of the readers the documentation page is the best central 
place, also given that this is what a Google search for “oak direct binary” et 
al. will surface first.

The algorithm description doesn’t really change much anymore and copy pasting 
should be minimal effort whenever that might happen again - simply copy the 
generated javadoc html into the markdown.

> [Direct Binary Access] upload algorithm documentation should also be on the 
> website
> ---
>
> Key: OAK-8696
> URL: https://issues.apache.org/jira/browse/OAK-8696
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: doc
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> The upload algorithm description that is currently a bit hidden in the 
> javadocs of 
> [https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]
> should also be included on the documentation site: 
> [https://jackrabbit.apache.org/oak/docs/features/direct-binary-access.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-8695) [Direct Binary Access] upload algorithm documentation should make it clear that not all URIs have to be used

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983922#comment-16983922
 ] 

Alexander Klimetschek commented on OAK-8695:


I was thinking this detail should be mentioned inside the detailed algorithm 
description (currently in the javadoc, see description) not in the high level 
steps on the documentation page.

> [Direct Binary Access] upload algorithm documentation should make it clear 
> that not all URIs have to be used
> 
>
> Key: OAK-8695
> URL: https://issues.apache.org/jira/browse/OAK-8695
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: api, doc
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> Regarding [BinaryUpload 
> javadoc|http://jackrabbit.apache.org/oak/docs/apidocs/org/apache/jackrabbit/api/binary/BinaryUpload.html]
>  ([code 
> here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]):
> In the "Steps" section for #3, it should explicitly state that not all the 
> URIs need to be used. It could be inferred based on what is there, but it's 
> not obvious.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (OAK-8696) [Direct Binary Access] upload algorithm documentation should also be on the website

[Alexander Klimetschek (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-8696:
--

Assignee: Matt Ryan

> [Direct Binary Access] upload algorithm documentation should also be on the 
> website
> ---
>
> Key: OAK-8696
> URL: https://issues.apache.org/jira/browse/OAK-8696
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: doc
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> The upload algorithm description that is currently a bit hidden in the 
> javadocs of 
> [https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]
> should also be included on the documentation site: 
> [https://jackrabbit.apache.org/oak/docs/features/direct-binary-access.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (OAK-8695) [Direct Binary Access] upload algorithm documentation should make it clear that not all URIs have to be used

[Alexander Klimetschek (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-8695:
--

Assignee: Matt Ryan  (was: Matt Ryan)

> [Direct Binary Access] upload algorithm documentation should make it clear 
> that not all URIs have to be used
> 
>
> Key: OAK-8695
> URL: https://issues.apache.org/jira/browse/OAK-8695
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: api, doc
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> Regarding [BinaryUpload 
> javadoc|http://jackrabbit.apache.org/oak/docs/apidocs/org/apache/jackrabbit/api/binary/BinaryUpload.html]
>  ([code 
> here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]):
> In the "Steps" section for #3, it should explicitly state that not all the 
> URIs need to be used. It could be inferred based on what is there, but it's 
> not obvious.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (OAK-8695) [Direct Binary Access] upload algorithm documentation should make it clear that not all URIs have to be used

[Alexander Klimetschek (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-8695:
--

Assignee: Matt Ryan

> [Direct Binary Access] upload algorithm documentation should make it clear 
> that not all URIs have to be used
> 
>
> Key: OAK-8695
> URL: https://issues.apache.org/jira/browse/OAK-8695
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: api, doc
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> Regarding [BinaryUpload 
> javadoc|http://jackrabbit.apache.org/oak/docs/apidocs/org/apache/jackrabbit/api/binary/BinaryUpload.html]
>  ([code 
> here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]):
> In the "Steps" section for #3, it should explicitly state that not all the 
> URIs need to be used. It could be inferred based on what is there, but it's 
> not obvious.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-8695) [Direct Binary Access] upload algorithm documentation should make it clear that not all URIs have to be used

[Alexander Klimetschek (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-8695:
---
Component/s: doc

> [Direct Binary Access] upload algorithm documentation should make it clear 
> that not all URIs have to be used
> 
>
> Key: OAK-8695
> URL: https://issues.apache.org/jira/browse/OAK-8695
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: api, doc
>Reporter: Alexander Klimetschek
>Priority: Major
>
> Regarding [BinaryUpload 
> javadoc|http://jackrabbit.apache.org/oak/docs/apidocs/org/apache/jackrabbit/api/binary/BinaryUpload.html]
>  ([code 
> here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]):
> In the "Steps" section for #3, it should explicitly state that not all the 
> URIs need to be used. It could be inferred based on what is there, but it's 
> not obvious.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (OAK-8696) [Direct Binary Access] upload algorithm documentation should also be on the website

[Alexander Klimetschek (Jira)]

Alexander Klimetschek created OAK-8696:
--

 Summary: [Direct Binary Access] upload algorithm documentation 
should also be on the website
 Key: OAK-8696
 URL: https://issues.apache.org/jira/browse/OAK-8696
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: doc
Reporter: Alexander Klimetschek


The upload algorithm description that is currently a bit hidden in the javadocs 
of 
[https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]

should also be included on the documentation site: 
[https://jackrabbit.apache.org/oak/docs/features/direct-binary-access.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (OAK-8695) [Direct Binary Access] upload algorithm documentation should make it clear that not all URIs have to be used

[Alexander Klimetschek (Jira)]

Alexander Klimetschek created OAK-8695:
--

 Summary: [Direct Binary Access] upload algorithm documentation 
should make it clear that not all URIs have to be used
 Key: OAK-8695
 URL: https://issues.apache.org/jira/browse/OAK-8695
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: api
Reporter: Alexander Klimetschek


Regarding [BinaryUpload 
javadoc|http://jackrabbit.apache.org/oak/docs/apidocs/org/apache/jackrabbit/api/binary/BinaryUpload.html]
 ([code 
here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-jackrabbit-api/src/main/java/org/apache/jackrabbit/api/binary/BinaryUpload.java]):

In the "Steps" section for #3, it should explicitly state that not all the URIs 
need to be used. It could be inferred based on what is there, but it's not 
obvious.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917841#comment-16917841
 ] 

Alexander Klimetschek edited comment on OAK-8552 at 8/28/19 2:54 PM:
-

[~amitjain] I don’t think [~ianeboston] was suggesting to use the 
jcr:lastModified JCR property for that. All of these could be internal fields 
not exposed on the JCR level.

With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set. But there is still copying/versioning of 
nodes which leads to multiple references to the same blob - does this lead to a 
last modified update of the blob? (Just curious because I wasn’t aware of that)

Nonetheless, for the issue at hand I don’t think we need to change anything 
stored in the NodeStore. The size/length is already stored in the NS as part of 
the internal blob id (OAK-8314).


was (Author: alexander.klimetschek):
[~amitjain] I don’t think [~ianeboston] was suggesting to use the 
jcr:lastModified JCR property for that. All of these could be internal fields 
not exposed on the JCR level.

With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set. But there is still copying/versioning of 
nodes which leads to multiple references to the same blob - does this lead to a 
last modified update of the blob? (Just curious because I wasn’t aware of that)

Nonetheless, for the issue at hand I don’t think we need to change anything 
stored in the NodeStore. The size/length is already stored in the NS as part of 
the internal blob id.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917841#comment-16917841
 ] 

Alexander Klimetschek commented on OAK-8552:


[~amitjain] I don’t think [~ianeboston] was suggesting to use the 
jcr:lastModified JCR property for that. All of these could be internal fields 
not exposed on the JCR level.

With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set. But there is still copying/versioning of 
nodes which leads to multiple references to the same blob - does this lead to a 
last modified update of the blob? (Just curious because I wasn’t aware of that)

Nonetheless, for the issue at hand I don’t think we need to change anything 
stored in the NodeStore. The size/length is already stored in the NS as part of 
the internal blob id.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917291#comment-16917291
 ] 

Alexander Klimetschek commented on OAK-8580:


A few more thoughts based on practical usage of the above patch:
 * it logs downloading of {{reference.key}} which could probably be filtered out
 * since this is in the BlobStore impl, this will only log once, but then it's 
in the DS cache, and thus the code is not called again. this seems like a 
feature, as it reduces the log output a lot. and the goal of the log output is 
to get the unique stack traces, not every single invocation.
 * we cannot see the JCR path in the logs, which would be extremely helpful. 
there is typically the GET /some/url in the log output as part of the thread 
name, which helps to some degree, but not always.

it is not trivial because down in the blob store the knowledge of the binary 
has been lost. but maybe there is a trick to add it, for example by updating 
the thread name somewhere higher up the stack where we retrieve the binary:

{code:java}
Thread.currentThread().setName(Thread.currentThread().getName() + " (jcr path: 
" + jcrPath + ")"); {code}

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917271#comment-16917271
 ] 

Alexander Klimetschek commented on OAK-8580:


Here is a patch for the AzureBlobStoreBackend:  [^OAK-8580.patch]

This adds a new logger "oak.datastore.download.streams" to make it separate 
from the existing logger based on the class. The same name could be used in the 
S3 BlobStore implementation. Better name welcome.

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Alexander Klimetschek (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-8580:
---
Attachment: OAK-8580.patch

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Alexander Klimetschek (Jira)]

Alexander Klimetschek created OAK-8580:
--

 Summary: Stack trace logging for all binary downloading of cloud 
blob stores
 Key: OAK-8580
 URL: https://issues.apache.org/jira/browse/OAK-8580
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: blob-cloud, blob-cloud-azure
Affects Versions: 1.8.16, 1.16.0
Reporter: Alexander Klimetschek
Assignee: Matt Ryan


We want to be able to trace application code that invokes 
Binary.getInputStream() when a cloud blob store is present.

The Azure blob store has one place where it supports trace logging 
[here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
 but this misses another place blobs are fetched in 
[read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].

Furthermore, it would be nice to have a separate logger for that in order to 
create a log file that _only_ includes these stack traces, and no other debug 
logs of the AzureBackend class.

And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916427#comment-16916427
 ] 

Alexander Klimetschek commented on OAK-8552:


[~amitjain] +1 to Blob#isInlined(), that seems like the right solution in place 
of the getReference() == null.



> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916411#comment-16916411
 ] 

Alexander Klimetschek edited comment on OAK-8552 at 8/27/19 6:22 AM:
-

To add some context:

In our case that uncovered this issue, the problem really only exists because 
of a pretty special case: we uploaded binaries through JCR (InputStream) with 
async uploads in caching DS enabled. Then immediately after the session.save() 
we requested presigned GET URLs for these assets to pass them on to an external 
service. But the blobs were still uploading to the Azure blob store at that 
point and were not reliably available yet for the external service.

That lead us first to make the presigned URL generation support existence check 
(OAK-7998, plus the implicit check in getReference()) and „polling“ by 
returning null for not-yet-in-blob-store cases. However, that is shifting the 
solution to the wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is added to the NodeStore.

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. The binary is not available either way. But the latter allows us to 
completely drop any existence checks upon presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.


was (Author: alexander.klimetschek):
To add some context:

In our case that uncovered this issue, the problem really only exists because 
in some special case, we uploaded binaries through JCR with async uploads in 
caching DS enabled, and then immediately after the session.save() requested 
presigned GET URLs to pass them on to an external service. That lead us first 
to make the presigned URL generation support existence check (OAK-7998, plus 
the implicit check in getReference()) and „polling“ by returning null for 
not-yet-in-blob-store cases. However, that is shifting the solution to the 
wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is added to the NodeStore.

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. The binary is not available either way. But the latter allows us to 
completely drop any existence checks upon presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916411#comment-16916411
 ] 

Alexander Klimetschek edited comment on OAK-8552 at 8/27/19 6:19 AM:
-

To add some context:

In our case that uncovered this issue, the problem really only exists because 
in some special case, we uploaded binaries through JCR with async uploads in 
caching DS enabled, and then immediately after the session.save() requested 
presigned GET URLs to pass them on to an external service. That lead us first 
to make the presigned URL generation support existence check (OAK-7998, plus 
the implicit check in getReference()) and „polling“ by returning null for 
not-yet-in-blob-store cases. However, that is shifting the solution to the 
wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is added to the NodeStore.

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. The binary is not available either way. But the latter allows us to 
completely drop any existence checks upon presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.


was (Author: alexander.klimetschek):
To add some context:

In our case that uncovered this issue, the problem really only exists because 
in some special case, we uploaded binaries through JCR with async uploads in 
caching DS enabled, and then immediately after the session.save() requested 
presigned GET URLs to pass them on to an external service. That lead us first 
to make the presigned URL generation support existence check and „polling“ by 
returning null for not-yet-in-blob-store cases. However, that is shifting the 
solution to the wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is added to the NodeStore.

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. The binary is not available either way. But the latter allows us to 
completely drop any existence checks upon presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916411#comment-16916411
 ] 

Alexander Klimetschek edited comment on OAK-8552 at 8/27/19 6:16 AM:
-

To add some context:

In our case that uncovered this issue, the problem really only exists because 
in some special case, we uploaded binaries through JCR with async uploads in 
caching DS enabled, and then immediately after the session.save() requested 
presigned GET URLs to pass them on to an external service. That lead us first 
to make the presigned URL generation support existence check and „polling“ by 
returning null for not-yet-in-blob-store cases. However, that is shifting the 
solution to the wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is added to the NodeStore.

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. The binary is not available either way. But the latter allows us to 
completely drop any existence checks upon presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.


was (Author: alexander.klimetschek):
To add some context:

In our case that uncovered this issue, the problem really only exists because 
in some special case, we uploaded binaries through JCR with async uploads in 
caching DS enabled, and then immediately after the session.save() requested 
presigned GET URLs to pass them on to an external service. That lead us first 
to make the presigned URL generation support existence check and „polling“ by 
returning null for not-yet-in-blob-store cases. However, that is shifting the 
solution to the wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is in the NodeStore. 

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. But the latter allows us to completely drop any existence checks upon 
presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916411#comment-16916411
 ] 

Alexander Klimetschek commented on OAK-8552:


To add some context:

In our case that uncovered this issue, the problem really only exists because 
in some special case, we uploaded binaries through JCR with async uploads in 
caching DS enabled, and then immediately after the session.save() requested 
presigned GET URLs to pass them on to an external service. That lead us first 
to make the presigned URL generation support existence check and „polling“ by 
returning null for not-yet-in-blob-store cases. However, that is shifting the 
solution to the wrong end and increasing application complexity (polling loop).

(Also note this edge case is a short term solution to be replaced at some point 
with proper direct binary access for upload)

The source of the problem here is the async upload: we need to switch this to 
synchronous uploads = blocking session.save(), to avoid the issue in the first 
place.

In all regular cases, binaries are uploaded through the new direct binary 
access, which by design guarantees the presence of the binary when the 
reference is in the NodeStore. 

If the binary gets deleted from the blob store due to some malfunctioning, then 
it does not matter to the application if we return null or an URL that returns 
404. But the latter allows us to completely drop any existence checks upon 
presigned GET URL generation.

Same for inlined: configuration must prevent this in the first place, then no 
special check is required at access time.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8520) [Direct Binary Access] Avoid overwriting existing binaries via direct binary upload

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901583#comment-16901583
 ] 

Alexander Klimetschek commented on OAK-8520:


[~mattvryan] Awesome, thanks! Looks like it was even simpler than I thought.

> [Direct Binary Access] Avoid overwriting existing binaries via direct binary 
> upload
> ---
>
> Key: OAK-8520
> URL: https://issues.apache.org/jira/browse/OAK-8520
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-cloud, blob-cloud-azure, blob-plugins, doc
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.18.0, 1.10.4
>
>
> Since direct binary upload generates a unique blob ID for each upload, it is 
> generally impossible to overwrite any existing binary.  However, if a client 
> issues the {{completeBinaryUpload()}} call more than one time with the same 
> upload token, it is possible to overwrite an existing binary.
> One use case where this can happen is if a client call to complete the upload 
> times out.  Lacking a successful return a client could assume that it needs 
> to repeat the call to complete the upload.  If the binary was already 
> uploaded before, the subsequent call to complete the upload would have the 
> effect of overwriting the binary with new content generated from any 
> uncommitted uploaded blocks.  In practice usually there are no uncommitted 
> blocks so this generates a zero-length binary.
> There may be a use case for a zero-length binary so simply failing in such a 
> case is not sufficient.
> One easy way to handle this would be to simply check for the existence of the 
> binary before completing the upload.  This would have the effect of making 
> uploaded binaries un-modifiable by the client.  In such a case the 
> implementation could throw an exception indicating that the binary already 
> exists and cannot be written again.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (OAK-8520) [Direct Binary Access] Avoid overwriting existing binaries via direct binary upload

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900629#comment-16900629
 ] 

Alexander Klimetschek commented on OAK-8520:


IMO this should be classified as a bug and given a higher priority. The blob 
store implementation has to ensure the immutability of blobs and not wipe them 
out if applications (accidentally) call completeBinaryUpload() twice.

AFACS {{completeBinaryUpload()}} should simply be an idempotent operation, 
returning the existing blob as {{Binary}} if called the 2nd (or 3rd, or 4th...) 
time with the same upload token. Then clients can safely retry their request 
that leads to the application code to call {{completeBinaryUpload() }}and try 
writing the same JCR structure.

> [Direct Binary Access] Avoid overwriting existing binaries via direct binary 
> upload
> ---
>
> Key: OAK-8520
> URL: https://issues.apache.org/jira/browse/OAK-8520
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure, blob-plugins, doc
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
>
> Since direct binary upload generates a unique blob ID for each upload, it is 
> generally impossible to overwrite any existing binary.  However, if a client 
> issues the {{completeBinaryUpload()}} call more than one time with the same 
> upload token, it is possible to overwrite an existing binary.
> One use case where this can happen is if a client call to complete the upload 
> times out.  Lacking a successful return a client could assume that it needs 
> to repeat the call to complete the upload.  If the binary was already 
> uploaded before, the subsequent call to complete the upload would have the 
> effect of overwriting the binary with new content generated from any 
> uncommitted uploaded blocks.  In practice usually there are no uncommitted 
> blocks so this generates a zero-length binary.
> There may be a use case for a zero-length binary so simply failing in such a 
> case is not sufficient.
> One easy way to handle this would be to simply check for the existence of the 
> binary before completing the upload.  This would have the effect of making 
> uploaded binaries un-modifiable by the client.  In such a case the 
> implementation could throw an exception indicating that the binary already 
> exists and cannot be written again.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Comment Edited] (OAK-7998) [DirectBinaryAccess] Verify that binary exists in cloud before creating signed download URI

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894125#comment-16894125
 ] 

Alexander Klimetschek edited comment on OAK-7998 at 7/26/19 8:54 PM:
-

[~mattvryan]
How is it possible to get the presigned URL if the upload has not completed 
yet? There should be no reference in the NodeStore at all before the complete.

And after a complete, if the blob/file is empty, that is IMO a perfectly valid 
situation from the blob store perspective (it doesn't know whether a client 
might want to store an empty file deliberately), for which a presigned GET URL 
should be generated.

We designed it explicitly so that the binary in Oak is guaranteed to be 
immutable, and thus only visible after upload has completed.


was (Author: alexander.klimetschek):
[~mattvryan]
How is it possible to get the presigned URL if the upload has not completed 
yet? There should be no reference in the NodeStore at all before the complete.

And after a complete, if the blob/file is empty, that is IMO a perfectly valid 
situation from the blob store perspective (it doesn't know whether a client 
might want to store an empty file deliberately), for which a presigned GET URL 
should be generated.

> [DirectBinaryAccess] Verify that binary exists in cloud before creating 
> signed download URI
> ---
>
> Key: OAK-7998
> URL: https://issues.apache.org/jira/browse/OAK-7998
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.10.0
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.16.0, 1.10.4
>
>
> The direct binary access download logic doesn't actually verify that the 
> requested blob is available in the cloud before creating the signed download 
> URI.  It is possible that a user could request a download URI for a blob that 
> is "in the repo" but hasn't actually been uploaded yet.
> We should verify this by uploading a new blob, preventing it being uploaded 
> to the cloud (retain in cache), and then request the download URI.  We should 
> get a null back or get some other error or exception; if we get a URI it 
> would return an HTTP 404 if the blob is not actually uploaded yet (maybe this 
> would also be ok).



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (OAK-7998) [DirectBinaryAccess] Verify that binary exists in cloud before creating signed download URI

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894125#comment-16894125
 ] 

Alexander Klimetschek commented on OAK-7998:


[~mattvryan]
How is it possible to get the presigned URL if the upload has not completed 
yet? There should be no reference in the NodeStore at all before the complete.

And after a complete, if the blob/file is empty, that is IMO a perfectly valid 
situation from the blob store perspective (it doesn't know whether a client 
might want to store an empty file deliberately), for which a presigned GET URL 
should be generated.

> [DirectBinaryAccess] Verify that binary exists in cloud before creating 
> signed download URI
> ---
>
> Key: OAK-7998
> URL: https://issues.apache.org/jira/browse/OAK-7998
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.10.0
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.16.0, 1.10.4
>
>
> The direct binary access download logic doesn't actually verify that the 
> requested blob is available in the cloud before creating the signed download 
> URI.  It is possible that a user could request a download URI for a blob that 
> is "in the repo" but hasn't actually been uploaded yet.
> We should verify this by uploading a new blob, preventing it being uploaded 
> to the cloud (retain in cache), and then request the download URI.  We should 
> get a null back or get some other error or exception; if we get a URI it 
> would return an HTTP 404 if the blob is not actually uploaded yet (maybe this 
> would also be ok).



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (OAK-7702) [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for uploads

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826481#comment-16826481
 ] 

Alexander Klimetschek commented on OAK-7702:


S3 Transfer acceleration doesn’t cache and is not a CDN. 

> [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for 
> uploads
> -
>
> Key: OAK-7702
> URL: https://issues.apache.org/jira/browse/OAK-7702
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> Azure Blob Store and S3 support CDNs in front of private containers/buckets, 
> which also work with presigned URLs ([S3 
> docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html],
>  [cloudfront presigning 
> java|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html],
>  [Azure 
> docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). 
> The binary access feature should support these for download URLs via a 
> configuration on the DataStore.
> Currently, the S3 datastore has a config for [transfer 
> acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html]
>  that if enabled, will make both upload & download URLs use the acceleration 
> option. However, this feature only makes sense for uploads, especially if CDN 
> is an option for downloads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8013) [Direct Binary Access] DataRecordDownloadOptions creates invalid Content-Disposition headers

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762939#comment-16762939
 ] 

Alexander Klimetschek commented on OAK-8013:


+1 for dropping {{filename*}} as intermediate fix.

While not the optimal, this increases the range of supported filenames a lot 
(since spaces are common and non-ISO-8859-1 are not).

> [Direct Binary Access] DataRecordDownloadOptions creates invalid 
> Content-Disposition headers
> 
>
> Key: OAK-8013
> URL: https://issues.apache.org/jira/browse/OAK-8013
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
>
> DataRecordDownloadOptions always adds the extended parameter filename* to the 
> header, [without any 
> escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].
> Such extended parameters must not include spaces (and only a small predefined 
> list of basic ascii chars), otherwise they have to be percent encoded. The 
> RFC is https://tools.ietf.org/html/rfc5987 and note the definition of 
> value-chars in the grammar.
> Because of this, if a filename includes a space or another character that 
> must be percent encoded, this currently creates an invalid header that fails 
> to be parsed by other clients.
> See also https://github.com/jshttp/content-disposition/issues/24
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-8013) [Direct Binary Access] DataRecordDownloadOptions creates invalid Content-Disposition headers

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-8013:
---
Description: 
DataRecordDownloadOptions always adds the extended parameter filename* to the 
header, [without any 
escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].

Such extended parameters must not include spaces (and only a small predefined 
list of basic ascii chars), otherwise they have to be percent encoded. The RFC 
is https://tools.ietf.org/html/rfc5987 and note the definition of value-chars 
in the grammar.

Because of this, if a filename includes a space or another character that must 
be percent encoded, this currently creates an invalid header that fails to be 
parsed by other clients.

See also https://github.com/jshttp/content-disposition/issues/24

 

  was:
DataRecordDownloadOptions always adds the extended parameter filename* to the 
header, [without any 
escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].

Such extended parameters must not include spaces (and only a small predefined 
list of basic ascii chars), otherwise they have to be percent escaped. The RFC 
is https://tools.ietf.org/html/rfc5987 and note the definition of value-chars 
in the grammar.

Because of this, if a filename includes a space, this currently creates an 
invalid header that fails to be parsed by other clients.

See also https://github.com/jshttp/content-disposition/issues/24

 


> [Direct Binary Access] DataRecordDownloadOptions creates invalid 
> Content-Disposition headers
> 
>
> Key: OAK-8013
> URL: https://issues.apache.org/jira/browse/OAK-8013
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Priority: Major
>
> DataRecordDownloadOptions always adds the extended parameter filename* to the 
> header, [without any 
> escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].
> Such extended parameters must not include spaces (and only a small predefined 
> list of basic ascii chars), otherwise they have to be percent encoded. The 
> RFC is https://tools.ietf.org/html/rfc5987 and note the definition of 
> value-chars in the grammar.
> Because of this, if a filename includes a space or another character that 
> must be percent encoded, this currently creates an invalid header that fails 
> to be parsed by other clients.
> See also https://github.com/jshttp/content-disposition/issues/24
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-8013) [Direct Binary Access] DataRecordDownloadOptions creates invalid Content-Disposition headers

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-8013:
---
Description: 
DataRecordDownloadOptions always adds the extended parameter filename* to the 
header, [without any 
escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].

Such extended parameters must not include spaces (and only a small predefined 
list of basic ascii chars), otherwise they have to be percent escaped. The RFC 
is https://tools.ietf.org/html/rfc5987 and note the definition of value-chars 
in the grammar.

Because of this, if a filename includes a space, this currently creates an 
invalid header that fails to be parsed by other clients.

See also https://github.com/jshttp/content-disposition/issues/24

 

  was:
DataRecordDownloadOptions always adds the extended parameter filename* to the 
header, [without any 
escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].

Such extended parameters must not include spaces (and only a small predefined 
list of basic ascii chars), otherwise they have to be percent escaped. The RFC 
is https://tools.ietf.org/html/rfc5987 and note the definition of the 
value-chars in the 

Because of this, if a filename includes a space, this currently creates an 
invalid header that fails to be parsed by other clients.

See also https://github.com/jshttp/content-disposition/issues/24

 


> [Direct Binary Access] DataRecordDownloadOptions creates invalid 
> Content-Disposition headers
> 
>
> Key: OAK-8013
> URL: https://issues.apache.org/jira/browse/OAK-8013
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Priority: Major
>
> DataRecordDownloadOptions always adds the extended parameter filename* to the 
> header, [without any 
> escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].
> Such extended parameters must not include spaces (and only a small predefined 
> list of basic ascii chars), otherwise they have to be percent escaped. The 
> RFC is https://tools.ietf.org/html/rfc5987 and note the definition of 
> value-chars in the grammar.
> Because of this, if a filename includes a space, this currently creates an 
> invalid header that fails to be parsed by other clients.
> See also https://github.com/jshttp/content-disposition/issues/24
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-8013) [Direct Binary Access] DataRecordDownloadOptions creates invalid Content-Disposition headers

[Alexander Klimetschek (JIRA)]

Alexander Klimetschek created OAK-8013:
--

 Summary: [Direct Binary Access] DataRecordDownloadOptions creates 
invalid Content-Disposition headers
 Key: OAK-8013
 URL: https://issues.apache.org/jira/browse/OAK-8013
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: blob-plugins
Reporter: Alexander Klimetschek


DataRecordDownloadOptions always adds the extended parameter filename* to the 
header, [without any 
escaping|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordDownloadOptions.java#L130].

Such extended parameters must not include spaces (and only a small predefined 
list of basic ascii chars), otherwise they have to be percent escaped. The RFC 
is https://tools.ietf.org/html/rfc5987 and note the definition of the 
value-chars in the 

Because of this, if a filename includes a space, this currently creates an 
invalid header that fails to be parsed by other clients.

See also https://github.com/jshttp/content-disposition/issues/24

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7702) [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for uploads

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7702:
---
Description: 
Azure Blob Store and S3 support CDNs in front of private containers/buckets, 
which also work with presigned URLs ([S3 
docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html],
 [cloudfront presigning 
java|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html],
 [Azure 
docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). The 
binary access feature should support these for download URLs via a 
configuration on the DataStore.

Currently, the S3 datastore has a config for [transfer 
acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html]
 that if enabled, will make both upload & download URLs use the acceleration 
option. However, this feature only makes sense for uploads, especially if CDN 
is an option for downloads.

  was:
Azure Blob Store and S3 support CDNs in front of private containers/buckets, 
which also work with presigned URLs ([S3 
docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html],
 [Azure 
docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). The 
binary access feature should support these for download URLs via a 
configuration on the DataStore.

Currently, the S3 datastore has a config for [transfer 
acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html]
 that if enabled, will make both upload & download URLs use the acceleration 
option. However, this feature only makes sense for uploads, especially if CDN 
is an option for downloads.


> [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for 
> uploads
> -
>
> Key: OAK-7702
> URL: https://issues.apache.org/jira/browse/OAK-7702
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Alexander Klimetschek
>Priority: Major
>
> Azure Blob Store and S3 support CDNs in front of private containers/buckets, 
> which also work with presigned URLs ([S3 
> docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html],
>  [cloudfront presigning 
> java|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html],
>  [Azure 
> docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). 
> The binary access feature should support these for download URLs via a 
> configuration on the DataStore.
> Currently, the S3 datastore has a config for [transfer 
> acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html]
>  that if enabled, will make both upload & download URLs use the acceleration 
> option. However, this feature only makes sense for uploads, especially if CDN 
> is an option for downloads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek resolved OAK-7832.

   Resolution: Fixed
 Assignee: Alexander Klimetschek
Fix Version/s: 1.8.9
   1.10

Committed to trunk in 1844932.

Backported to 1.8 in 1844933.

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Fix For: 1.10, 1.8.9
>
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654724#comment-16654724
 ] 

Alexander Klimetschek commented on OAK-7832:


Note that JsonSerializer continues to behave like before (throw on any 
exception), unless the new constructor with the new „checkException“ argument 
set to true is used, which only the export command from oak-run does. But 
please double check.

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654436#comment-16654436
 ] 

Alexander Klimetschek commented on OAK-7832:


Fixed the license header (my IDE should have a smart copyright profile 
detection :D).

I could commit myself.

This change was done against the 1.8 branch, but it patches without  changes 
against trunk. Given many production users (for which oak-run is useful) are on 
1.8, should I commit to both?

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654436#comment-16654436
 ] 

Alexander Klimetschek edited comment on OAK-7832 at 10/18/18 12:27 AM:
---

[~chetanm]

Fixed the license header (my IDE should have a smart copyright profile 
detection :D).

I could commit myself.

This change was done against the 1.8 branch, but it patches without  changes 
against trunk. Given many production users (for which oak-run is useful) are on 
1.8, *should I commit to both*?


was (Author: alexander.klimetschek):
Fixed the license header (my IDE should have a smart copyright profile 
detection :D).

I could commit myself.

This change was done against the 1.8 branch, but it patches without  changes 
against trunk. Given many production users (for which oak-run is useful) are on 
1.8, should I commit to both?

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: OAK-7832.patch

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: (was: OAK-7832.patch)

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650789#comment-16650789
 ] 

Alexander Klimetschek edited comment on OAK-7832 at 10/15/18 9:04 PM:
--

[~chetanm] Could you maybe take a look at the patch?

Some notes:
 * for the {{pn}} command I copied the toString code from NodeState et al into 
the new NodeStateHelper.java inside oak-run
 * for the {{export}} command I had to touch the JsonSerializer of 
oak-store-spi which I see is also used in other cases (possibly at runtime, not 
just oak-run utility), so I added a new flag "checkExceptions" which is off by 
default to trigger the new behavior. By default the class will behave as before 
and throw any exception (like SNFE).
 * A related improvement for the {{export}} would be to create one json file 
per page/asset/folder /jcr:content etc., especially if one is exporting larger 
trees. A single nodestate.json is hard to "browse" as a human. That's probably 
something for another issue I guess.


was (Author: alexander.klimetschek):
[~chetanm] Could you maybe take a look at the patch?

Some notes:
 * for the {{pn}} command I copied the toString code from NodeState et al into 
the new NodeStateHelper.java inside oak-run
 * for the {{export}} command I had to touch the JsonSerializer of 
oak-store-spi which I see is also used in other cases (possibly at runtime, not 
just oak-run utility), so I added a new flag "checkExceptions" which is off by 
default to trigger the new behavior. By default the class will behave as before 
and throw any exception (like SNFE).

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650789#comment-16650789
 ] 

Alexander Klimetschek commented on OAK-7832:


[~chetanm] Could you maybe take a look at the patch?

Some notes:
 * for the {{pn}} command I copied the toString code from NodeState et al into 
the new NodeStateHelper.java inside oak-run
 * for the {{export}} command I had to touch the JsonSerializer of 
oak-store-spi which I see is also used in other cases (possibly at runtime, not 
just oak-run utility), so I added a new flag "checkExceptions" which is off by 
default to trigger the new behavior. By default the class will behave as before 
and throw any exception (like SNFE).

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-7832:
--

Assignee: (was: Alexander Klimetschek)

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649199#comment-16649199
 ] 

Alexander Klimetschek commented on OAK-7832:


Attached is a  [^OAK-7832.patch] against the oak 1.8 branch that. Also attached 
a ready to use jar with the changes included: [^oak-run-1.8.9-SNAPSHOT.jar].

1. Changed the console {{export}} command to log exceptions and include them in 
the json, but then continue and produce a valid json.
Log output:
{noformat}
/content/some/folder> export
01:02:01.589 [main] ERROR o.a.j.oak.json.JsonSerializer - Cannot read property 
value /content/some/folder/mypage/jcr:content/cq:lastModifiedBy : 
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
dec74dbe-d24f-4f3d-a7ef-75374bca3572 not found
01:02:01.596 [main] ERROR o.a.j.oak.json.JsonSerializer - Cannot read node 
/content/some/folder/mypage/jcr:content/content/paragraph/par/component0 : 
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
e8c16c25-cc3c-4874-ab0b-f515d6523c24 not found
{noformat}
nodestates.json output:
{noformat}
...
"cq:cssClass": "ERROR: Cannot read property value 
/content/some/folder/mypage/jcr:content/content/paragraph/par/component/cq:cssClass
 : org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
e8c16c25-cc3c-4874-ab0b-f515d6523c24 not found"
...
   "component0": {
"_error": "ERROR: Cannot read node 
/content/some/folder/mypage/jcr:content/content/paragraph/par/component0 : 
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
e8c16c25-cc3c-4874-ab0b-f515d6523c24 not found"
   },
{noformat}

2. Also maked the {{pn}} command handle exceptions.
{noformat}
/content/some/folder> pn
{ jcr:primaryType = cq:Page, jcr:createdBy = silcottl, jcr:created = 
2018-09-20T15:05:43.264Z, :childOrder = [jcr:content, one, two, three, four, 
give], one = { ... }, two = { ... }, three = { ... }, jcr:content = { ... }, 
four = { ERROR on node: 
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
a9d9a87e-e452-452f-a3d3-d89228bef4d6 not found }, five = { ... } }
{noformat}

In both cases it catches on property value access and child node access, which 
at least with SegmentTar are the places where you'd get a 
SegmentNotFoundException if the segment storing that information is missing. 
For handling the getChildNodeEntries() case in the JsonSerializer I had to to a 
bit of a trick with a MemoryChildNodeEntry to get a nice json output with as 
much information as possible, without rewriting more of the JsonSerializer.

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: oak-run-1.8.9-SNAPSHOT.jar

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: OAK-7832.patch

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: (was: OAK-7832.patch)

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: (was: oak-run-1.8.9-SNAPSHOT.jar)

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: oak-run-1.8.9-SNAPSHOT.jar

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Attachment: OAK-7832.patch

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7832:
---
Component/s: store-spi

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run, store-spi
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-7832:
--

Assignee: Alexander Klimetschek

> oak-run console export should handle exceptions such as missing segments
> 
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: run
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> Problem: When trying to look at data using "pn" or running "export" oak-run 
> console currently will choke and abort the traversal on exceptions such as 
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed 
> in the exported json, and just continue. This also ensures all missing 
> segments for a subtree are listed with one command (if it fails right now you 
> just see the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-7832) oak-run console export should handle exceptions such as missing segments

[Alexander Klimetschek (JIRA)]

Alexander Klimetschek created OAK-7832:
--

 Summary: oak-run console export should handle exceptions such as 
missing segments
 Key: OAK-7832
 URL: https://issues.apache.org/jira/browse/OAK-7832
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: run
Reporter: Alexander Klimetschek


Problem: When trying to look at data using "pn" or running "export" oak-run 
console currently will choke and abort the traversal on exceptions such as 
SegmentNotFoundException.

Expected: It would be nice if it would be best-effort, log the issue or embed 
in the exported json, and just continue. This also ensures all missing segments 
for a subtree are listed with one command (if it fails right now you just see 
the first missing segment).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588529#comment-16588529
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/22/18 8:17 AM:
-

[~mreutegg] The javadoc patch is ready in JCR-4355, it could be committed 
(JCR-4355-v3.patch) and a new jackrabbit-api released.


was (Author: alexander.klimetschek):
[~mreutegg] The javadoc patch is ready in JCR-4355, it could be committed and a 
new jackrabbit-api released.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588529#comment-16588529
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/22/18 8:17 AM:
-

[~mreutegg] The javadoc patch is ready in JCR-4355, it could be committed 
(JCR-4355-v3.patch) and a new jackrabbit-api released.


was (Author: alexander.klimetschek):
[~mreutegg] The javadoc patch is ready in JCR-4355, it could be committed 
(JCR-4355-v3.patch) and a new jackrabbit-api released.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588529#comment-16588529
 ] 

Alexander Klimetschek commented on OAK-7569:


[~mreutegg] The javadoc patch is ready in JCR-4355, it could be committed and a 
new jackrabbit-api released.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576908#comment-16576908
 ] 

Alexander Klimetschek commented on OAK-7693:


[~amitjain] Common properties: yes, that would be good. But maybe [~mattvryan] 
had a good reason for them to be different. However, let's look at this later. 
We also have OAK-7702 to tackle where we might add/change config properties.

[~mreutegg] Here is the final doc change in case you want to commit it with 
OAK-7569: 
https://github.com/alexkli/jackrabbit-oak/commit/d633b51c34bcc77865533cd769b52f9682f69895
 (including binary files)
Otherwise let me know when things are in trunk and I'll commit it. When should 
we publish to the documentation site? As soon as the feature is in trunk or 
only when it's in an official oak release? (The doc says it's since 1.10, so it 
should be clear)

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (OAK-7701) Documentation should link to Jackrabbit API

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek resolved OAK-7701.

Resolution: Fixed

Change
 * committed [http://svn.apache.org/viewvc?view=revision&revision=1837836]
 * and published to [https://jackrabbit.apache.org/oak/docs/index.html]

> Documentation should link to Jackrabbit API
> ---
>
> Key: OAK-7701
> URL: https://issues.apache.org/jira/browse/OAK-7701
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: doc
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7701.patch
>
>
> The [Oak documentation|https://jackrabbit.apache.org/oak/docs/] currently 
> links to the "JCR API" and the (internal) "Oak API" in the navigation on the 
> left side, but it is missing the "Jackrabbit API" (JCR api extensions) that 
> are very important for developers using Oak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575289#comment-16575289
 ] 

Alexander Klimetschek edited comment on OAK-7693 at 8/9/18 7:23 PM:


I updated the docs: 
[preview|https://github.com/alexkli/jackrabbit-oak/blob/direct-binary-access-v2-with-partial-value-factory/oak-doc/src/site/markdown/features/direct-binary-access.md]
 * moved underneath "Blob Storage" in the navigation
 * adding a new high level diagram at the top
 * added important info on the transfer acceleration config
 * links to S3, Azure BS, the oak direct binary access wiki page
 * some editorial improvements

The questions above are still open.

BTW, it's probably easiest for me to commit this once approved since it 
involves the diagram images that aren't included in patch files.


was (Author: alexander.klimetschek):
I updated the docs: 
[preview|https://github.com/alexkli/jackrabbit-oak/blob/direct-binary-access-v2-with-partial-value-factory/oak-doc/src/site/markdown/features/direct-binary-access.md]
 * moved underneath "Blob Storage" in the navigation
 * adding a new high level diagram at the top
 * added important info on the transfer acceleration config
 * links to S3, Azure BS, the oak direct binary access wiki page
 * some editorial improvements

The questions above are still open.

 

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575289#comment-16575289
 ] 

Alexander Klimetschek commented on OAK-7693:


I updated the docs: 
[preview|https://github.com/alexkli/jackrabbit-oak/blob/direct-binary-access-v2-with-partial-value-factory/oak-doc/src/site/markdown/features/direct-binary-access.md]
 * moved underneath "Blob Storage" in the navigation
 * adding a new high level diagram at the top
 * added important info on the transfer acceleration config
 * links to S3, Azure BS, the oak direct binary access wiki page
 * some editorial improvements

The questions above are still open.

 

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575277#comment-16575277
 ] 

Alexander Klimetschek edited comment on OAK-7693 at 8/9/18 7:10 PM:


[~amitjain] Are there any documentations for the S3 and Azure DataStores (i.e. 
regarding their configuration etc.)? I would like to link to them in the doc 
and possibly update them with the new configuration options. The [BlobStore 
page|https://jackrabbit.apache.org/oak/docs/plugins/blobstore.html] doesn't 
explain them in detail nor mentions their configurations. 

The best I can find is the [Adobe AEM documentation for 
Oak|https://helpx.adobe.com/experience-manager/6-4/sites/deploying/using/data-store-config.html],
 but I don't think we should link to that from the Oak documentation.


was (Author: alexander.klimetschek):
[~amitjain] Are there any documentations for the S3 and Azure DataStores (i.e. 
regarding their configuration etc.)? I would like to link to them in the doc 
and possibly update them with the new configuration options.

The best I can find is the [Adobe AEM documentation for 
Oak|https://helpx.adobe.com/experience-manager/6-4/sites/deploying/using/data-store-config.html],
 but I don't think we should link to that from the Oak documentation.

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575277#comment-16575277
 ] 

Alexander Klimetschek commented on OAK-7693:


[~amitjain] Are there any documentations for the S3 and Azure DataStores (i.e. 
regarding their configuration etc.)? I would like to link to them in the doc 
and possibly update them with the new configuration options.

The best I can find is the [Adobe AEM documentation for 
Oak|https://helpx.adobe.com/experience-manager/6-4/sites/deploying/using/data-store-config.html],
 but I don't think we should link to that from the Oak documentation.

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575261#comment-16575261
 ] 

Alexander Klimetschek edited comment on OAK-7693 at 8/9/18 6:42 PM:


1.
 [~mreutegg] The doc currently includes this statement:
{quote}Please note that only Binary objects returned from Property.getBinary(), 
Property.getValue().getBinary() or Property.getValues() ... getBinary() will 
support a functional BinaryDownload.
{quote}
Over in OAK-7569 you mentioned
{quote}Direct binary access is now also working for authorizable properties and 
values obtained through events.
{quote}
Should I update the above statement then? If yes, what is a good way to 
describe the different cases, list all of them?

2.
 Will Oak 1.10 be the "official" release this feature is expected to be in? The 
doc states {{@since Oak 1.10}} and I wasn't sure if that can mention the 
unstable releases, e.g. 1.9.x currently.

 


was (Author: alexander.klimetschek):
1.
[~mreutegg] The doc currently includes this statement:

bq. Please note that only Binary objects returned from Property.getBinary(), 
Property.getValue().getBinary() or Property.getValues() ... getBinary() will 
support a functional BinaryDownload.

Over in OAK-7569 you mentioned

bq. Direct binary access is now also working for authorizable properties and 
values obtained through events.

Should I update the above statement then? If yes, what is a good way to 
describe the different cases, list all of them?

2.
Will Oak 1.10 be the "official" release this feature is expected to be in? The 
doc states {{@since Oak 1.10}} and I wasn't sure if that can mention the 
unstable releases, e.g. 1.9.x currently.

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575261#comment-16575261
 ] 

Alexander Klimetschek commented on OAK-7693:


1.
[~mreutegg] The doc currently includes this statement:

bq. Please note that only Binary objects returned from Property.getBinary(), 
Property.getValue().getBinary() or Property.getValues() ... getBinary() will 
support a functional BinaryDownload.

Over in OAK-7569 you mentioned

bq. Direct binary access is now also working for authorizable properties and 
values obtained through events.

Should I update the above statement then? If yes, what is a good way to 
describe the different cases, list all of them?

2.
Will Oak 1.10 be the "official" release this feature is expected to be in? The 
doc states {{@since Oak 1.10}} and I wasn't sure if that can mention the 
unstable releases, e.g. 1.9.x currently.

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575257#comment-16575257
 ] 

Alexander Klimetschek commented on OAK-7692:


Fixed the license header and attached the updated patch:  [^OAK-7692.patch] 

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Fix For: 1.10
>
> Attachments: OAK-7692.patch
>
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Attachment: OAK-7692.patch

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Fix For: 1.10
>
> Attachments: OAK-7692.patch
>
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574030#comment-16574030
 ] 

Alexander Klimetschek edited comment on OAK-7692 at 8/9/18 6:31 AM:


[~mreutegg] Fix including unit test available 
[here|https://github.com/alexkli/jackrabbit-oak/commit/cdc945d89eb74d0be1163bbfb70d8250a0625e83],
 and as [patch 
file|https://github.com/alexkli/jackrabbit-oak/commit/cdc945d89eb74d0be1163bbfb70d8250a0625e83.diff].

Note that I also made all the exception messages for an invalid token the same 
("Invalid upload token") so that possibly attacking clients don't get too much 
information.


was (Author: alexander.klimetschek):
[~mreutegg] Fix including unit test available 
[here|https://github.com/mattvryan/jackrabbit-oak/pull/14], and as [patch 
file|https://patch-diff.githubusercontent.com/raw/mattvryan/jackrabbit-oak/pull/14.diff].

Note that I also made all the exception messages for an invalid token the same 
("Invalid upload token") so that possibly attacking clients don't get too much 
information.

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Fix For: 1.10
>
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (OAK-7695) [DirectBinaryAccess] Fix and improve javadocs of new API

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek resolved OAK-7695.

Resolution: Duplicate

Created JCR-4355

> [DirectBinaryAccess] Fix and improve javadocs of new API
> 
>
> Key: OAK-7695
> URL: https://issues.apache.org/jira/browse/OAK-7695
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: api
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> Here are some changes to the javadocs for the new API: 
> [OAK-7569-api-javadoc-improvements.patch|https://issues.apache.org/jira/secure/attachment/12934364/12934364_OAK-7569-api-javadoc-improvements.patch]
> * more concise descriptions
> * correcting some inaccuracies (clients cannot choose whether to do single or 
> multipart upload, multipart might be strictly required depending on the size)
> * most importantly the upload algorithm (standard partSize calculation was 
> wrong)
> * focus on API users, separated notes to implementors
> * for BinaryDownloadOptions added note from which jcr properties a client 
> would normally take these values from
> * added security considerations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574342#comment-16574342
 ] 

Alexander Klimetschek commented on OAK-7693:


[~mreutegg] Documentation is available 
[here|https://github.com/alexkli/jackrabbit-oak/commit/0d27d31f606347c22a48d9856f3546e62f3dd244],
 and as [patch 
file|https://github.com/alexkli/jackrabbit-oak/commit/0d27d31f606347c22a48d9856f3546e62f3dd244.diff].

This adds a new "Direct Binary Access" page to the docs, see a [preview 
here|https://github.com/alexkli/jackrabbit-oak/blob/direct-binary-access-v2-with-partial-value-factory/oak-doc/src/site/markdown/features/direct-binary-access.md].

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OAK-7693) [DirectBinaryAccess] Documentation

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-7693:
--

Assignee: Alexander Klimetschek

> [DirectBinaryAccess] Documentation
> --
>
> Key: OAK-7693
> URL: https://issues.apache.org/jira/browse/OAK-7693
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: doc
>Reporter: Marcel Reutegger
>Assignee: Alexander Klimetschek
>Priority: Minor
> Fix For: 1.10
>
>
> Add official Oak documentation for the direct binary access feature.
> There's a [wiki 
> page|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access] that 
> already describes the feature, but official documentation should be created 
> in the oak-doc module.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-7702) [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for uploads

[Alexander Klimetschek (JIRA)]

Alexander Klimetschek created OAK-7702:
--

 Summary: [Direct Binary Access] Add CDN support, use S3 Transfer 
Acceleration only for uploads
 Key: OAK-7702
 URL: https://issues.apache.org/jira/browse/OAK-7702
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: blob-cloud, blob-cloud-azure
Reporter: Alexander Klimetschek


Azure Blob Store and S3 support CDNs in front of private containers/buckets, 
which also work with presigned URLs ([S3 
docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html],
 [Azure 
docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). The 
binary access feature should support these for download URLs via a 
configuration on the DataStore.

Currently, the S3 datastore has a config for [transfer 
acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html]
 that if enabled, will make both upload & download URLs use the acceleration 
option. However, this feature only makes sense for uploads, especially if CDN 
is an option for downloads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7701) Documentation should link to Jackrabbit API

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574039#comment-16574039
 ] 

Alexander Klimetschek commented on OAK-7701:


This patch: [^OAK-7701.patch] adds a link to 
[https://jackrabbit.apache.org/jcr/jcr-api.html] with label "Jackrabbit API" 
underneath the "JCR API" link.

> Documentation should link to Jackrabbit API
> ---
>
> Key: OAK-7701
> URL: https://issues.apache.org/jira/browse/OAK-7701
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: doc
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7701.patch
>
>
> The [Oak documentation|https://jackrabbit.apache.org/oak/docs/] currently 
> links to the "JCR API" and the (internal) "Oak API" in the navigation on the 
> left side, but it is missing the "Jackrabbit API" (JCR api extensions) that 
> are very important for developers using Oak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7701) Documentation should link to Jackrabbit API

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7701:
---
Attachment: OAK-7701.patch

> Documentation should link to Jackrabbit API
> ---
>
> Key: OAK-7701
> URL: https://issues.apache.org/jira/browse/OAK-7701
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: doc
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7701.patch
>
>
> The [Oak documentation|https://jackrabbit.apache.org/oak/docs/] currently 
> links to the "JCR API" and the (internal) "Oak API" in the navigation on the 
> left side, but it is missing the "Jackrabbit API" (JCR api extensions) that 
> are very important for developers using Oak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OAK-7701) Documentation should link to Jackrabbit API

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-7701:
--

Assignee: Alexander Klimetschek

> Documentation should link to Jackrabbit API
> ---
>
> Key: OAK-7701
> URL: https://issues.apache.org/jira/browse/OAK-7701
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: doc
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7701.patch
>
>
> The [Oak documentation|https://jackrabbit.apache.org/oak/docs/] currently 
> links to the "JCR API" and the (internal) "Oak API" in the navigation on the 
> left side, but it is missing the "Jackrabbit API" (JCR api extensions) that 
> are very important for developers using Oak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-7701) Documentation should link to Jackrabbit API

[Alexander Klimetschek (JIRA)]

Alexander Klimetschek created OAK-7701:
--

 Summary: Documentation should link to Jackrabbit API
 Key: OAK-7701
 URL: https://issues.apache.org/jira/browse/OAK-7701
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: doc
Reporter: Alexander Klimetschek


The [Oak documentation|https://jackrabbit.apache.org/oak/docs/] currently links 
to the "JCR API" and the (internal) "Oak API" in the navigation on the left 
side, but it is missing the "Jackrabbit API" (JCR api extensions) that are very 
important for developers using Oak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574030#comment-16574030
 ] 

Alexander Klimetschek edited comment on OAK-7692 at 8/8/18 11:33 PM:
-

[~mreutegg] Fix including unit test available 
[here|https://github.com/mattvryan/jackrabbit-oak/pull/14], and as [patch 
file|https://patch-diff.githubusercontent.com/raw/mattvryan/jackrabbit-oak/pull/14.diff].

Note that I also made all the exception messages for an invalid token the same 
("Invalid upload token") so that possibly attacking clients don't get too much 
information.


was (Author: alexander.klimetschek):
[~mreutegg] Fix including unit test available 
[here|https://github.com/mattvryan/jackrabbit-oak/pull/14], and as [patch 
file|https://patch-diff.githubusercontent.com/raw/mattvryan/jackrabbit-oak/pull/14.diff].

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek resolved OAK-7692.

Resolution: Fixed

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574030#comment-16574030
 ] 

Alexander Klimetschek commented on OAK-7692:


[~mreutegg] Fix including unit test available 
[here|https://github.com/mattvryan/jackrabbit-oak/pull/14], and as [patch 
file|https://patch-diff.githubusercontent.com/raw/mattvryan/jackrabbit-oak/pull/14.diff].

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek reassigned OAK-7692:
--

Assignee: Alexander Klimetschek

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7695) [DirectBinaryAccess] Fix and improve javadocs of new API

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7695:
---
Description: 
Here are some changes to the javadocs for the new API: 
[OAK-7569-api-javadoc-improvements.patch|https://issues.apache.org/jira/secure/attachment/12934364/12934364_OAK-7569-api-javadoc-improvements.patch]

* more concise descriptions
* correcting some inaccuracies (clients cannot choose whether to do single or 
multipart upload, multipart might be strictly required depending on the size)
* most importantly the upload algorithm (standard partSize calculation was 
wrong)
* focus on API users, separated notes to implementors
* for BinaryDownloadOptions added note from which jcr properties a client would 
normally take these values from
* added security considerations

  was:
Here are some changes to the javadocs for the new API: 
[^OAK-7569-api-javadoc-improvements.patch]

* more concise descriptions
* correcting some inaccuracies (clients cannot choose whether to do single or 
multipart upload, multipart might be strictly required depending on the size)
* most importantly the upload algorithm (standard partSize calculation was 
wrong)
* focus on API users, separated notes to implementors
* for BinaryDownloadOptions added note from which jcr properties a client would 
normally take these values from
* added security considerations


> [DirectBinaryAccess] Fix and improve javadocs of new API
> 
>
> Key: OAK-7695
> URL: https://issues.apache.org/jira/browse/OAK-7695
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: api
>Reporter: Alexander Klimetschek
>Assignee: Alexander Klimetschek
>Priority: Major
>
> Here are some changes to the javadocs for the new API: 
> [OAK-7569-api-javadoc-improvements.patch|https://issues.apache.org/jira/secure/attachment/12934364/12934364_OAK-7569-api-javadoc-improvements.patch]
> * more concise descriptions
> * correcting some inaccuracies (clients cannot choose whether to do single or 
> multipart upload, multipart might be strictly required depending on the size)
> * most importantly the upload algorithm (standard partSize calculation was 
> wrong)
> * focus on API users, separated notes to implementors
> * for BinaryDownloadOptions added note from which jcr properties a client 
> would normally take these values from
> * added security considerations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-7695) [DirectBinaryAccess] Fix and improve javadocs of new API

[Alexander Klimetschek (JIRA)]

Alexander Klimetschek created OAK-7695:
--

 Summary: [DirectBinaryAccess] Fix and improve javadocs of new API
 Key: OAK-7695
 URL: https://issues.apache.org/jira/browse/OAK-7695
 Project: Jackrabbit Oak
  Issue Type: Technical task
  Components: api
Reporter: Alexander Klimetschek
Assignee: Alexander Klimetschek


Here are some changes to the javadocs for the new API: 
[^OAK-7569-api-javadoc-improvements.patch]

* more concise descriptions
* correcting some inaccuracies (clients cannot choose whether to do single or 
multipart upload, multipart might be strictly required depending on the size)
* most importantly the upload algorithm (standard partSize calculation was 
wrong)
* focus on API users, separated notes to implementors
* for BinaryDownloadOptions added note from which jcr properties a client would 
normally take these values from
* added security considerations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573451#comment-16573451
 ] 

Alexander Klimetschek commented on OAK-7569:


AuthorizablePropertiesImpl: Good to be confirmed. When I followed the call 
hierachy, there were many places that used DEFAULT (IIRC for the 
UserManagerImpl constructor) and it wasn‘t clear which ones are actually used 
at normal runtime (and I skipped any test usages).

Limitation/documentation: fair enough. So the places that we support are 
getProperty().[getValue().]getBinary(), as well as the equivalent multivalue 
methods (if anyone really uses multivalue binary properties ;-). Any other 
„normal“ Binary read case that I missed?

One thought: if we don‘t have a BlobAccessProvider available, could we return a 
Binary that does not implement BinaryDownload? Then it would be clear to 
developers they used an unsupported access path.

We need documentation for the oak docs anyway, not sure if there is anything 
yet. Happy to add something. Where would I add this in the navigation tree?

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Description: 
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}
Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.

  was:
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}
Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.


> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148].
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Description: 
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}
Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.

  was:
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}
Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac for 
a string that can be safely passed around.


> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572552#comment-16572552
 ] 

Alexander Klimetschek commented on OAK-7569:


Added OAK-7692 which was found by a team trying out the feature. While 
[~mattvryan] is out, I can tackle that unless [~amitjain] has time to look into 
this.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Component/s: (was: blob-cloud)
 blob-plugins

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac 
> for a string that can be safely passed around.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Description: 
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}
Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac for 
a string that can be safely passed around.

  was:
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1jYzVmLTk2M2EtNGVmMC1hMjEzLTdlYTJjM2MwYWJkYi0xNTMzNjkxNzA3Nzg0IzIwMTgtMDgtMDhUMDE6Mjg6MjcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}

Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac for 
a string that can be safely passed around.


> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-plugins
>Reporter: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1[...]jcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac 
> for a string that can be safely passed around.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Description: 
The upload token's hmac signature (after the #) is not base64 encoded. This 
might create problems for clients passing that string around if it can contain 
non-ascii characters.

Example:
{noformat}
ZDI4Zi1jYzVmLTk2M2EtNGVmMC1hMjEzLTdlYTJjM2MwYWJkYi0xNTMzNjkxNzA3Nzg0IzIwMTgtMDgtMDhUMDE6Mjg6MjcuNzg3Wg==#i�_�\��?��S��,0:�
{noformat}

Code is 
[here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]

Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac for 
a string that can be safely passed around.

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-cloud
>Reporter: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1jYzVmLTk2M2EtNGVmMC1hMjEzLTdlYTJjM2MwYWJkYi0xNTMzNjkxNzA3Nzg0IzIwMTgtMDgtMDhUMDE6Mjg6MjcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac 
> for a string that can be safely passed around.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Klimetschek updated OAK-7692:
---
Component/s: blob-cloud

> [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded
> ---
>
> Key: OAK-7692
> URL: https://issues.apache.org/jira/browse/OAK-7692
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: blob-cloud
>Reporter: Alexander Klimetschek
>Priority: Major
>
> The upload token's hmac signature (after the #) is not base64 encoded. This 
> might create problems for clients passing that string around if it can 
> contain non-ascii characters.
> Example:
> {noformat}
> ZDI4Zi1jYzVmLTk2M2EtNGVmMC1hMjEzLTdlYTJjM2MwYWJkYi0xNTMzNjkxNzA3Nzg0IzIwMTgtMDgtMDhUMDE6Mjg6MjcuNzg3Wg==#i�_�\��?��S��,0:�
> {noformat}
> Code is 
> [here|https://github.com/mattvryan/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/directaccess/DataRecordUploadToken.java#L147-L148]
> Should probably do a {{Base64.encode()}} of the {{hash}} result of the hmac 
> for a string that can be safely passed around.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-7692) [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

[Alexander Klimetschek (JIRA)]

Alexander Klimetschek created OAK-7692:
--

 Summary: [DirectBinaryAccess] Upload token HMAC signature must be 
base64 encoded
 Key: OAK-7692
 URL: https://issues.apache.org/jira/browse/OAK-7692
 Project: Jackrabbit Oak
  Issue Type: Technical task
Reporter: Alexander Klimetschek






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572284#comment-16572284
 ] 

Alexander Klimetschek commented on OAK-7569:


For reference, [~mreutegg] did the ValueFactoryImpl refactoring mentioned above 
in OAK-7688.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572281#comment-16572281
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/7/18 8:30 PM:


FWIW, I added a plain unit test for oak-jcr: 
https://github.com/mattvryan/jackrabbit-oak/commit/f45e4ecc7823ee8d50484fdb3d076b88fab429d9

This quickly tests the current problem (plus the upload API) without requiring 
S3 or Azure blob storage setup.


was (Author: alexander.klimetschek):
FWIW, I added a plain unit test for oak-jcr: 
https://github.com/mattvryan/jackrabbit-oak/pull/13

This quickly tests the current problem (plus the upload API) without requiring 
S3 or Azure blob storage setup.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572281#comment-16572281
 ] 

Alexander Klimetschek commented on OAK-7569:


FWIW, I added a plain unit test for oak-jcr: 
https://github.com/mattvryan/jackrabbit-oak/pull/13

This quickly tests the current problem (plus the upload API) without requiring 
S3 or Azure blob storage setup.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571976#comment-16571976
 ] 

Alexander Klimetschek commented on OAK-7569:


Ok, the „unlikely“ part is the one I wonder if Oak would be comfortable with. 
Otherwise I am happy if you can handle approach 1.

Regarding NamePathMapper.DEFAULT: I didn‘t say all usages are wrong, and the 
stats was just an indication. But from looking at the code and call 
hierarchies, it seems a lot or most of the NamePathMapper usages is simply 
there to pass it to the static ValueFactoryImpl methods in the end. And it 
seemed impossible for me to reason about it with the varying requirements of 
all these different components. One example: if I have a NAME or PATH type 
property on a user home, will Authorizable.getProperty() handle the right 
namespace mappings?

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570592#comment-16570592
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/6/18 11:22 PM:
-

Option 1 also has 3 different levels if you look at it in detail. From less 
complete (a) to full complete (c):
 * *a)* Fix just the immediate case in oak-jcr and oak-core as done by Matt in 
[this 
branch|https://github.com/mattvryan/jackrabbit-oak/tree/direct-binary-access-v2-with-create-binary-value-fix]
 ** Addresses the normal {{getProperty().getBinary()}} and 
{{getProperty().getValue().getBinary()}} usages plus multi-value binary 
properties.
 ** I would do the changes slightly differently and just replace all static 
method usage outside ValueFactoryImpl, where possible.
 ** Con: Leaves open {{ValueFactoryImpl.createValue()}} static method usage in 
12 files (3 of which are in tests), with unknown impact.
 * *b)* Same as a) but also change the {{createValue()}} usage in all code 
paths that don't actually use a {{NamePathMapper.DEFAULT}} and don't have 
access to a {{SessionContext}} currently
 ** No exact numbers, but {{NamePathMapper}} is used in 104 files, and 62 use 
{{NamePathMapper.DEFAULT}}. Some might be from different uses of this interface.
 ** A trick and arguable a hack for this case could be to repurpose 
{{NamePathMapper}} and add a new interface method {{getInternalValueFactory()}} 
to it, which would return {{ValueFactoryImpl}} (or a new internal interface). 
In some / most cases the {{NamePathMapper}} is actually the {{SessionContext}}, 
and this would be simple to do. The API semantics of {{NamePathMapper}} would 
be a bit confusing, but at least it would prevent one from changing all the 
methods and classes that pass around {{NamePathMapper}} to pass around 
something else.
 ** Con: This leaves open places that end up using {{NamePathMapper.DEFAULT}}.
 * *c)* Rigorously ensure everyone using a {{ValueFactoryImpl}} static method 
must use an instance, and must have access to the current {{SessionContext}} or 
{{SessionImpl}}, and remove use of {{NamePathMapper.DEFAULT}}
 ** Would surely be a useful improvement.
 ** Con: Seems like a massive Oak change as mentioned above.

Note that places (at runtime, not tests) that use {{NamePathMapper.DEFAULT}} 
today would have at least the limitation that when a NAME or PATH property is 
used, and the session has local namespaces, that these aren't respected, 
because the default impl has no access to them.


was (Author: alexander.klimetschek):
Option 1 also has 3 different levels if you look at it in detail. From less 
complete (a) to full complete (c):
 * *a)* Fix just the immediate case in oak-jcr and oak-core as done by Matt in 
[this 
PR|https://github.com/mattvryan/jackrabbit-oak/commit/e022f846c0ff113c95910d26584568004beacb66]
 (linked in my long comment above already)
 ** Addresses the normal {{getProperty().getBinary()}} and 
{{getProperty().getValue().getBinary()}} usages plus multi-value binary 
properties.
 ** I would do the changes slightly differently and just replace all static 
method usage outside ValueFactoryImpl, where possible.
 ** Con: Leaves open {{ValueFactoryImpl.createValue()}} static method usage in 
12 files (3 of which are in tests), with unknown impact.
 * *b)* Same as a) but also change the {{createValue()}} usage in all code 
paths that don't actually use a {{NamePathMapper.DEFAULT}} and don't have 
access to a {{SessionContext}} currently
 ** No exact numbers, but {{NamePathMapper}} is used in 104 files, and 62 use 
{{NamePathMapper.DEFAULT}}. Some might be from different uses of this interface.
 ** A trick and arguable a hack for this case could be to repurpose 
{{NamePathMapper}} and add a new interface method {{getInternalValueFactory()}} 
to it, which would return {{ValueFactoryImpl}} (or a new internal interface). 
In some / most cases the {{NamePathMapper}} is actually the {{SessionContext}}, 
and this would be simple to do. The API semantics of {{NamePathMapper}} would 
be a bit confusing, but at least it would prevent one from changing all the 
methods and classes that pass around {{NamePathMapper}} to pass around 
something else.
 ** Con: This leaves open places that end up using {{NamePathMapper.DEFAULT}}.
 * *c)* Rigorously ensure everyone using a {{ValueFactoryImpl}} static method 
must use an instance, and must have access to the current {{SessionContext}} or 
{{SessionImpl}}, and remove use of {{NamePathMapper.DEFAULT}}
 ** Would surely be a useful improvement.
 ** Con: Seems like a massive Oak change as mentioned above.

Note that places (at runtime, not tests) that use {{NamePathMapper.DEFAULT}} 
today would have at least the limitation that when a NAME or PATH property is 
used, and the session has local namespaces, that these aren't respected, 
because the default impl has no access 

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570592#comment-16570592
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/6/18 6:25 PM:


Option 1 also has 3 different levels if you look at it in detail. From less 
complete (a) to full complete (c):
 * *a)* Fix just the immediate case in oak-jcr and oak-core as done by Matt in 
[this 
PR|https://github.com/mattvryan/jackrabbit-oak/commit/e022f846c0ff113c95910d26584568004beacb66]
 (linked in my long comment above already)
 ** Addresses the normal {{getProperty().getBinary()}} and 
{{getProperty().getValue().getBinary()}} usages plus multi-value binary 
properties.
 ** I would do the changes slightly differently and just replace all static 
method usage outside ValueFactoryImpl, where possible.
 ** Con: Leaves open {{ValueFactoryImpl.createValue()}} static method usage in 
12 files (3 of which are in tests), with unknown impact.
 * *b)* Same as a) but also change the {{createValue()}} usage in all code 
paths that don't actually use a {{NamePathMapper.DEFAULT}} and don't have 
access to a {{SessionContext}} currently
 ** No exact numbers, but {{NamePathMapper}} is used in 104 files, and 62 use 
{{NamePathMapper.DEFAULT}}. Some might be from different uses of this interface.
 ** A trick and arguable a hack for this case could be to repurpose 
{{NamePathMapper}} and add a new interface method {{getInternalValueFactory()}} 
to it, which would return {{ValueFactoryImpl}} (or a new internal interface). 
In some / most cases the {{NamePathMapper}} is actually the {{SessionContext}}, 
and this would be simple to do. The API semantics of {{NamePathMapper}} would 
be a bit confusing, but at least it would prevent one from changing all the 
methods and classes that pass around {{NamePathMapper}} to pass around 
something else.
 ** Con: This leaves open places that end up using {{NamePathMapper.DEFAULT}}.
 * *c)* Rigorously ensure everyone using a {{ValueFactoryImpl}} static method 
must use an instance, and must have access to the current {{SessionContext}} or 
{{SessionImpl}}, and remove use of {{NamePathMapper.DEFAULT}}
 ** Would surely be a useful improvement.
 ** Con: Seems like a massive Oak change as mentioned above.

Note that places (at runtime, not tests) that use {{NamePathMapper.DEFAULT}} 
today would have at least the limitation that when a NAME or PATH property is 
used, and the session has local namespaces, that these aren't respected, 
because the default impl has no access to them.


was (Author: alexander.klimetschek):
Option 1 also has 3 different levels if you look at it in detail. From less 
complete (a) to full complete (c):
 * *a)* Fix just the immediate case in oak-jcr and oak-core as done by Matt in 
[this 
PR|https://github.com/mattvryan/jackrabbit-oak/commit/e022f846c0ff113c95910d26584568004beacb66]
 (linked in my long comment above already)
 ** Addresses the normal {{getProperty().getBinary()}} and 
{{getProperty().getValue().getBinary()}} usages plus multi-value binary 
properties.
 ** I would do the changes slightly differently and just replace all static 
method usage outside ValueFactoryImpl, where possible.
 ** Con: Leaves open {{ValueFactoryImpl.createValue()}} static method usage in 
12 files (3 of which are in tests), with unknown impact.
 * *b)* Same as a) but also change the createValue() usage in all code paths 
that don't start with a NamePathMapper.DEFAULT and don't have access to a 
SessionContext currently
 ** No exact numbers, but NamePathMapper is used in 104 files, and 62 use 
{{NamePathMapper.DEFAULT}}.
 ** A trick and arguable a hack for this case could be to repurpose 
{{NamePathMapper}} and add a new interface method {{getInternalValueFactory()}} 
to it, which would return {{ValueFactoryImpl}} (or a new internal interface). 
In some / most cases the {{NamePathMapper}} is actually the {{SessionContext}}, 
and this would be simple to do. The API semantics of {{NamePathMapper}} would 
be a bit confusing, but at least it would prevent one from changing all the 
methods and classes that pass around {{NamePathMapper}} to pass around 
something else.
 ** Con: This leaves open places that end up using {{NamePathMapper.DEFAULT}}.
 * *c)* Rigorously ensure everyone using a {{ValueFactoryImpl}} static method 
must use an instance, and must have access to the current {{SessionContext}} or 
{{SessionImpl}}, and remove use of {{NamePathMapper.DEFAULT}}
 ** Would surely be a useful improvement.
 ** Con: Seems like a massive Oak change as mentioned above.

Note that places (at runtime, not tests) that use {{NamePathMapper.DEFAULT}} 
today would have at least the limitation that when a NAME or PATH property is 
used, and the session has local namespaces, that these aren't respected, 
because the default impl has no access to them.

> Direct Binary Access
> -

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570592#comment-16570592
 ] 

Alexander Klimetschek commented on OAK-7569:


Option 1 also has 3 different levels if you look at it in detail. From less 
complete (a) to full complete (c):
 * *a)* Fix just the immediate case in oak-jcr and oak-core as done by Matt in 
[this 
PR|https://github.com/mattvryan/jackrabbit-oak/commit/e022f846c0ff113c95910d26584568004beacb66]
 (linked in my long comment above already)
 ** Addresses the normal {{getProperty().getBinary()}} and 
{{getProperty().getValue().getBinary()}} usages plus multi-value binary 
properties.
 ** I would do the changes slightly differently and just replace all static 
method usage outside ValueFactoryImpl, where possible.
 ** Con: Leaves open {{ValueFactoryImpl.createValue()}} static method usage in 
12 files (3 of which are in tests), with unknown impact.
 * *b)* Same as a) but also change the createValue() usage in all code paths 
that don't start with a NamePathMapper.DEFAULT and don't have access to a 
SessionContext currently
 ** No exact numbers, but NamePathMapper is used in 104 files, and 62 use 
{{NamePathMapper.DEFAULT}}.
 ** A trick and arguable a hack for this case could be to repurpose 
{{NamePathMapper}} and add a new interface method {{getInternalValueFactory()}} 
to it, which would return {{ValueFactoryImpl}} (or a new internal interface). 
In some / most cases the {{NamePathMapper}} is actually the {{SessionContext}}, 
and this would be simple to do. The API semantics of {{NamePathMapper}} would 
be a bit confusing, but at least it would prevent one from changing all the 
methods and classes that pass around {{NamePathMapper}} to pass around 
something else.
 ** Con: This leaves open places that end up using {{NamePathMapper.DEFAULT}}.
 * *c)* Rigorously ensure everyone using a {{ValueFactoryImpl}} static method 
must use an instance, and must have access to the current {{SessionContext}} or 
{{SessionImpl}}, and remove use of {{NamePathMapper.DEFAULT}}
 ** Would surely be a useful improvement.
 ** Con: Seems like a massive Oak change as mentioned above.

Note that places (at runtime, not tests) that use {{NamePathMapper.DEFAULT}} 
today would have at least the limitation that when a NAME or PATH property is 
used, and the session has local namespaces, that these aren't respected, 
because the default impl has no access to them.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570288#comment-16570288
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/6/18 2:57 PM:


Sure, but please note that getting the implementation right (getting rid of 
static ValueFactoryImpl usage) affects 100+ files. Searching for NamePathMapper 
gives an indication: 
https://github.com/apache/jackrabbit-oak/search?utf8=%E2%9C%93&q=NamePathMapper&type

This would involve folks from different parts of Oak and possible take weeks 
and introduce a lot of risks.

Matt and me really looked into that last week but had to give up - especially 
when you hit NamePathMapper.DEFAULT. I fully agree it would be good for Oak to 
straighten this, but I guess there is a limit to what amount can be refactored 
and it would be nice to finally get this feature in.


was (Author: alexander.klimetschek):
Sure, but please note that getting the implementation right (getting rid of 
static ValueFactoryImpl usage) affects 100+ files. Searching for NamePathMapper 
gives an indication: 
https://github.com/apache/jackrabbit-oak/search?utf8=%E2%9C%93&q=NamePathMapper&type

This would involve folks from different parts of Oak and possible take weeks 
and introduce a lot of risks.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570288#comment-16570288
 ] 

Alexander Klimetschek commented on OAK-7569:


Sure, but please note that getting the implementation right (getting rid of 
static ValueFactoryImpl usage) affects 100+ files. Searching for NamePathMapper 
gives an indication: 
https://github.com/apache/jackrabbit-oak/search?utf8=%E2%9C%93&q=NamePathMapper&type

This would involve folks from different parts of Oak and possible take weeks 
and introduce a lot of risks.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569821#comment-16569821
 ] 

Alexander Klimetschek commented on OAK-7569:


Ok, maybe then I might not know Oak‘s process for such API releases and how it 
relates to semantic versioning.

Anyway, any preference on the different options? Do you think 1 is ok or the 
„full“ static-less ValueFactoryImpl is feasible for an Oak code expert?

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569802#comment-16569802
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/6/18 7:14 AM:


h4. Problem

Unfortunately, we could not commit the final agreed change this week, because 
[~mattvryan] found a blocking issue: {{BinaryDownload.getURI()}} always returns 
null because the regular code path {{node.getProperty().getBinary()}} does not 
pass the {{BlobAccessProvider}} instance through to the {{ValueFactoryImpl}} 
because it uses a [static method of ValueFactoryImpl in 
PropertyImpl.getValue()|https://github.com/apache/jackrabbit-oak/blob/905c9bd0f716778a035a708bbdb8de634f464e66/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/session/PropertyImpl.java#L252-L253].

Looking further, there are actually many static method usages of 
{{ValueFactoryImpl}} today, that all need to ensure a {{BlobAccessProvider}} is 
available, which is far from trivial.

[~mattvryan] and me think there are 2 solutions:
 # fix the static ValueFactoryImpl issue as much as possible (but ALL cases 
seem to be impossible to fix without a major Oak refactoring)
 # go back to the original API design (2a) or a slight variation thereof (2b) 
as a feasible compromise.

More details below.
h4. Details

The unit tests so far had a shortcut where it reused the {{Binary}} used for 
writing, hence no one saw the issue earlier. This has been fixed in the latest 
changes for the integration tests: 
[https://github.com/mattvryan/jackrabbit-oak/pull/12] (also includes a lot of 
improvements around the test framework itself).

This issue, that did not exist initially, and was tested, was introduced as a 
result of the refactorings during the review - which moved the API to a 
ValueFactory extension. At the same time the unit tests changed a bit, 
accidentally removing the important test case.

It is possible to fix this particular Value access case, since {{PropertyImpl}} 
already has access to the per-session {{ValueFactoryImpl}} instance, and it 
just needs to replace all the static usages left by leveraging 
{{getValueFactory()}}. Same for some other places in oak-jcr and oak-core that 
currently employ a mix of instance and static method access. [~mattvryan] did 
an attempt at this here: 
[https://github.com/mattvryan/jackrabbit-oak/commit/e022f846c0ff113c95910d26584568004beacb66]

Ideally, to ensure the {{blobAccessProvider}} reference in the 
{{ValueFactoryImpl}} is always set to the reference from the whiteboard that 
the {{SessionContext}} has, one would have to eliminate the static methods on 
{{ValueFactoryImpl}} and replace by instance methods. The class isn't doing 
proper encapsulation at this point, which is visible by the fact that 
{{NamePathMapper}}, an implementation detail the ValueFactoryImpl relies on, is 
actually all over the Oak code base.

*However, there are two static {{ValueFactoryImpl.createValue()}} methods 
retrieving a PropertyState and PropertyValue argument respectively (+ 
NamePathMapper) that are used in different places in oak, including 
oak-security-spi.* These are completely unrelated to the binary change. They 
don't have any immediate access to the {{SessionContext}} (which provides both 
the {{ValueFactoryImpl}} and {{BlobAcccessProvider}} instance). They all get a 
{{NamePathMapper}} passed in, which is essentially the same situation that a 
component reference must be available for the {{ValueFactoryImpl}} methods to 
do their work, but if you follow the call hierarchy it literally explodes. 
Meaning a lot of files all over Oak would have to be changed to actually pass 
through the {{ValueFactoryImpl}} instance instead of just {{NamePathMapper}}.

But even then, in many places it does not even get any access to the real 
{{NamePathMapper}} which {{SessionContext}} implements, but a 
{{NamePathMapper.DEFAULT}} is used (which apparently works for some of these 
cases?). It might be that some components were never designed to get access to 
the {{SessionContext}}, at least it seems it would be a massive refactoring to 
ensure that everywhere.
h4. Unsolved Binary cases

How does that affect the direct binary access feature? One example is 
[AuthorizablePropertiesImpl|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/security/user/AuthorizablePropertiesImpl.java#L108].
 This corresponds to the code path of 
{{Authorizable.getProperty("foo").getValue().getBinary()}}, which would never 
be able to return anything from {{getURI()}}. Note that it will implement 
{{BinaryDownload}}, and the API contract is that {{getURI()}} shall return null 
if the feature is not available, but not if the client used the "wrong" way of 
retrieving a {{Binary}}.

Other cases are hard to reason about: it's unclear where exactly they could end 
up being used b

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569819#comment-16569819
 ] 

Alexander Klimetschek commented on OAK-7569:


[~reschke] Yes, if by "here" you mean just talking about the API changes done 
for this feature in 2.17.5. Generally speaking, there cannot be breaking API 
changes (major version updates) in jackrabbit-api without disrupting the entire 
ecosystem. It's important to note the maven-bundle-plugin will not 
differentiate between the two - it will complain about both.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569814#comment-16569814
 ] 

Alexander Klimetschek commented on OAK-7569:


Another approach for 2a) would be to keep 2.17.5 on maven release, but 
completely ignore it (mark it as broken), and for 2.17.6 just ignore following 
the semantic version changes (that are only based on stuff added in 2.17.5).

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569814#comment-16569814
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/6/18 7:10 AM:


Another approach for 2a) would be to keep 2.17.5 on maven central, but 
completely ignore it (mark it as broken), and for 2.17.6 just ignore following 
the semantic version changes (that are only based on stuff added in 2.17.5).


was (Author: alexander.klimetschek):
Another approach for 2a) would be to keep 2.17.5 on maven release, but 
completely ignore it (mark it as broken), and for 2.17.6 just ignore following 
the semantic version changes (that are only based on stuff added in 2.17.5).

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7569) Direct Binary Access

[Alexander Klimetschek (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569813#comment-16569813
 ] 

Alexander Klimetschek edited comment on OAK-7569 at 8/6/18 7:07 AM:


[~reschke] Please have a look at the details of 2a and 2b (just made some 
slight changes). Anything that forces {{org.apache.jackrabbit.api}} to have a 
breaking change and have it's package version go from 2.5.0 to 3.0.0 would be a 
no go as it would make all existing JCR client bundles using that package (and 
that will be most I reckon) incompatible without recompilation. I.e. just the 
upgrade to a newer Oak would break all these bundles.


was (Author: alexander.klimetschek):
[~reschke] Please have a look at the details of 2a and 2b (just made some 
slight changes). Anything that forces {{org.apache.jackrabbit.api}} to have a 
breaking change and have it's package version go from 2.5.0 to 3.0.0 would be a 
no go as it would make all existing JCR client bundles using that package (and 
that will be most I reckon) incompatible without recompilation.

> Direct Binary Access
> 
>
> Key: OAK-7569
> URL: https://issues.apache.org/jira/browse/OAK-7569
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: api, blob-cloud, blob-cloud-azure, blob-plugins
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-7569-api-javadoc-improvements.patch
>
>
> Provide a direct binary access feature to Oak which allows an authenticated 
> client to create or download blobs directly to/from the blob store, assuming 
> the authenticated user has appropriate permission to do so. The primary value 
> of this feature is that the I/O of transferring large binary files to or from 
> the blob store can be offloaded entirely from Oak and performed directly 
> between a client application and the blob store.
> This feature is described in more detail [on the Oak 
> wiki|https://wiki.apache.org/jackrabbit/Direct%20Binary%20Access].
> This feature is similar in functionality to OAK-6575.  It adds the capability 
> to also upload directly to storage via preauthorized URLs in addition to 
> downloading directly from storage via preauthorized URLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)