[jira] [Commented] (OAK-8551) Minimize network calls in cloud data stores (performance optimization)
[
https://issues.apache.org/jira/browse/OAK-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967317#comment-16967317
]
Ian Boston commented on OAK-8551:
-
> For cloud DataStores there will be *a* network call to get the URI from the
>cloud provide.
Creating a Azure SRS or a AWS Signed URL does not need network calls. It is a
HMAC and can/should be performed completely offline. Are you certain that the
client library makes a network API call ?
Or are you referring to some input data into the HMAC that is not available in
the NodeStore ?
> Minimize network calls in cloud data stores (performance optimization)
> --
>
> Key: OAK-8551
> URL: https://issues.apache.org/jira/browse/OAK-8551
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.18.0, 1.10.4
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.20.0
>
>
> Oak cloud data stores (e.g. {{AzureDataStore}}, {{S3DataStore}}) are by
> definition more susceptible to performance degradation due to network issues.
> While we can't do much about the performance of uploading or downloading a
> blob, there are other places within the implementations where we are making
> network calls to the storage service which might be avoidable or minimized.
> One example is the {{exists()}} call to check whether a blob with a
> particular identifier exists in the blob storage. In some places
> {{exists()}} is being called where instead we could simply attempt the
> network access and handle failures elegantly, avoiding making an extra
> network call. In other places perhaps a cache could be used to minimize
> round trips.
> Another example is the higher-level {{getReference()}} call in
> {{DataStoreBlobStore}}. This asks the implementation for a {{DataRecord}}
> and then gets the reference from that, but in truth the data store backend
> can already obtain a reference for an identifier on its own. Asking for the
> {{DataRecord}} however requires a network request to get the blob metadata
> for the record.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI
[ https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916482#comment-16916482 ] Ian Boston commented on OAK-8552: - I have not digested all the details, but IMHO, the Oak NodeStore (SegmentNodeStore or DocumentNodeStore) should be the record of authority for blob existence, length, lastModified, blobID, such that to answer any question about a binary, only the NodeStore should have to be consulted, and no network API calls made. Only when a process actively needs to validate the date in NodeStore or interact directly with the binaries (upload, download, streaming) should it be forced to make network API calls to the blob storage API. IIUC there are edge cases highlight by Alex that abuse this principal (async upload?), but in general the principal holds. Also IIUC to make the Oak NodeStore the record of authority might require some additional properties to be stored (existence flag, length?, lastModified ?, etc?) Sorry if I have oversimplified. > Minimize network calls required when creating a direct download URI > --- > > Key: OAK-8552 > URL: https://issues.apache.org/jira/browse/OAK-8552 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: blob-cloud, blob-cloud-azure >Reporter: Matt Ryan >Assignee: Matt Ryan >Priority: Major > Attachments: OAK-8552_ApiChange.patch > > > We need to isolate and try to optimize network calls required to create a > direct download URI. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (OAK-6911) Provide a way to tune inline size while storing binaries
[ https://issues.apache.org/jira/browse/OAK-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366790#comment-16366790 ] Ian Boston commented on OAK-6911: - +1 to disabling. memory mapped ram is expensive especially when it runs out and triggers paging. The OS is not clever enough to differentiate. Even if the binaries are not put in the DS, they should not be put in files that are memory mapped. They should be put into separate files stored along side the tar files. OS disk cache is a lot safer to use for things that don't absolutely need to be memory mapped. Data that is in a binary file and is not used to back a operational object tree doesn't need to be memory mapped. The only exception might be Lucene files, which should be opened from a local copy. > Provide a way to tune inline size while storing binaries > > > Key: OAK-6911 > URL: https://issues.apache.org/jira/browse/OAK-6911 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Chetan Mehrotra >Priority: Major > Labels: performance, scalability > > SegmentNodeStore currently inlines binaries of size less that 16KB > (Segment.MEDIUM_LIMIT) even if external BlobStore is configured. > Due to this behaviour quite a bit of segment tar storage consist of blob > data. In one setup out of 370 GB segmentstore size 290GB is due to inlined > binary. If most of this binary content is moved to BlobStore then it would > allow same repository to work better in lesser RAM > So it would be useful if some way is provided to disable this default > behaviour and let BlobStore take control of inline size i.e. in presence of > BlobStore no inlining is attempted by SegmentWriter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6841) Revert Changes made in OAK-6575 before 1.7.10 is released.
[ https://issues.apache.org/jira/browse/OAK-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212281#comment-16212281 ] Ian Boston commented on OAK-6841: - Thank you for dealing with this and OAK-6575. Sorry it didnt work out. > Revert Changes made in OAK-6575 before 1.7.10 is released. > -- > > Key: OAK-6841 > URL: https://issues.apache.org/jira/browse/OAK-6841 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: blob, core, jcr >Affects Versions: 1.7.10 >Reporter: Ian Boston >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.8, 1.7.10 > > > Please revert the following commits to avoid the changes made in OAK-6575 > already released reaching 1.7.10 or a stable release while alternative > approaches are being investigated. > The reasons are documented in OAK-6575. > https://github.com/apache/jackrabbit-oak/commit/4870129e21352cef6f075ce26e09b743883a3214 > ( https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1811184) > https://github.com/apache/jackrabbit-oak/commit/8275b9baf1065d2cb977d23c5c64ec77fa6615ee > ( https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1811183 ) > https://github.com/apache/jackrabbit-oak/commit/9a77e0785b1177b33f9d72dfbb9d1d369becc914 > ( https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1811179) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208833#comment-16208833
]
Ian Boston commented on OAK-6575:
-
Blocker subtask OAK-6841 has been created. I hope that is Ok, 3 commits to
revert.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
>Assignee: Amit Jain
> Fix For: 1.8, 1.7.10
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6841) Revert Changes made in OAK-6575 before 1.7.10 is released.
Ian Boston created OAK-6841: --- Summary: Revert Changes made in OAK-6575 before 1.7.10 is released. Key: OAK-6841 URL: https://issues.apache.org/jira/browse/OAK-6841 Project: Jackrabbit Oak Issue Type: Sub-task Components: blob, core, jcr Affects Versions: 1.7.10 Reporter: Ian Boston Assignee: Amit Jain Priority: Blocker Please revert the following commits to avoid the changes made in OAK-6575 already released reaching 1.7.10 or a stable release while alternative approaches are being investigated. The reasons are documented in OAK-6575. https://github.com/apache/jackrabbit-oak/commit/4870129e21352cef6f075ce26e09b743883a3214 ( https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1811184) https://github.com/apache/jackrabbit-oak/commit/8275b9baf1065d2cb977d23c5c64ec77fa6615ee ( https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1811183 ) https://github.com/apache/jackrabbit-oak/commit/9a77e0785b1177b33f9d72dfbb9d1d369becc914 ( https://svn.apache.org/repos/asf/jackrabbit/oak/trunk@1811179) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208827#comment-16208827
]
Ian Boston commented on OAK-6575:
-
Summarising discussions on oak-dev and Sling dev.
Sling can only release using stable versions of oak.
Oak 1.7.x is unstable.
Oak cant backport new features to stable branches.
The Sling patch cant be merged and released until Oak 2.0 is released.
Oak 1.8, soon to be Oak 2.0 both of which are stable is not released.
Once Oak 2.0 is released, it will be too late to test and release the Sling
patch and use it in downstream projects.
Moving the API to Jackarabbit API doesn't address the problem as the versioning
and release model are the same as for Oak itself.
On that basis the patches here need to be re-thought to work with the Oak
development and release model. I will open a new issue, blocking 1.7.10 to
revert the changes made to Oak here, so that these patches don't make it out
into any other versions of Oak. IIUC Oak 1.7 is unstable and so this request is
Ok.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
>Assignee: Amit Jain
> Fix For: 1.8, 1.7.10
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private
[jira] [Comment Edited] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194443#comment-16194443
]
Ian Boston edited comment on OAK-6575 at 10/6/17 10:50 AM:
---
At a bundle level yes, it is incompatable.
1.6 oak-core vs 1.8-SNAPSHOT oak-api
But not at a package level, provided the package versions are the same, which
they are.
Since OSGi at build and runtime only looks at the package versions that are
provided, you can use any bundle that contains the correct package
version. Sling Servlets Get bundle in the patch has oak-core 1.6 using
org.apache.jackrabbit.oak.api.conversion v 1.0.0. So its MANIFEST.MF will
import org.apache.jackrabbit.oak.api.conversion=[1.0.0,2.0.0).
oak-api/1.8-SNAPSHOT also provides that package so Sling Servlets GET released
using oak-core 1.6 will also work in a OSGi container satisfied by
oak-api/1.8-SNAPSHOT
Obviously if Sling Servlets GET depends on a behaviour not expressed in the API
or binds to some other service or package only available in Oak core 1.6, then
it wont work. but for this patch, and the ones in Sling it uses the api and
a common package version.
I hope that makes sense.
was (Author: ianeboston):
At a bundle level yes.
1.6 oak-core vs 1.8-SNAPSHOT oak-api
But not at a package level, provided the package versions are the same, which
they are.
Since OSGi at build and runtime only looks at the package versions that are
provided, you can use any bundle that contains the correct package
version. Sling Servlets Get bundle in the patch has oak-core 1.6 using
org.apache.jackrabbit.oak.api.conversion v 1.0.0. So its MANIFEST.MF will
import org.apache.jackrabbit.oak.api.conversion=[1.0.0,2.0.0).
oak-api/1.8-SNAPSHOT also provides that package so Sling Servlets GET released
using oak-core 1.6 will also work in a OSGi container satisfied by
oak-api/1.8-SNAPSHOT
Obviously if Sling Servlets GET depends on a behaviour not expressed in the API
or binds to some other service or package only available in Oak core 1.6, then
it wont work. but for this patch, and the ones in Sling it uses the api and
a common package version.
I hope that makes sense.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
>Assignee: Amit Jain
> Fix For: 1.8, 1.7.9
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if (
[jira] [Comment Edited] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194443#comment-16194443
]
Ian Boston edited comment on OAK-6575 at 10/6/17 10:50 AM:
---
At a bundle level yes, it is incompatible.
1.6 oak-core vs 1.8-SNAPSHOT oak-api
But not at a package level, provided the package versions are the same, which
they are.
Since OSGi at build and runtime only looks at the package versions that are
provided, you can use any bundle that contains the correct package
version. Sling Servlets Get bundle in the patch has oak-core 1.6 using
org.apache.jackrabbit.oak.api.conversion v 1.0.0. So its MANIFEST.MF will
import org.apache.jackrabbit.oak.api.conversion=[1.0.0,2.0.0).
oak-api/1.8-SNAPSHOT also provides that package so Sling Servlets GET released
using oak-core 1.6 will also work in a OSGi container satisfied by
oak-api/1.8-SNAPSHOT
Obviously if Sling Servlets GET depends on a behaviour not expressed in the API
or binds to some other service or package only available in Oak core 1.6, then
it wont work. but for this patch, and the ones in Sling it uses the api and
a common package version.
I hope that makes sense.
was (Author: ianeboston):
At a bundle level yes, it is incompatable.
1.6 oak-core vs 1.8-SNAPSHOT oak-api
But not at a package level, provided the package versions are the same, which
they are.
Since OSGi at build and runtime only looks at the package versions that are
provided, you can use any bundle that contains the correct package
version. Sling Servlets Get bundle in the patch has oak-core 1.6 using
org.apache.jackrabbit.oak.api.conversion v 1.0.0. So its MANIFEST.MF will
import org.apache.jackrabbit.oak.api.conversion=[1.0.0,2.0.0).
oak-api/1.8-SNAPSHOT also provides that package so Sling Servlets GET released
using oak-core 1.6 will also work in a OSGi container satisfied by
oak-api/1.8-SNAPSHOT
Obviously if Sling Servlets GET depends on a behaviour not expressed in the API
or binds to some other service or package only available in Oak core 1.6, then
it wont work. but for this patch, and the ones in Sling it uses the api and
a common package version.
I hope that makes sense.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
>Assignee: Amit Jain
> Fix For: 1.8, 1.7.9
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
>
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194443#comment-16194443
]
Ian Boston commented on OAK-6575:
-
At a bundle level yes.
1.6 oak-core vs 1.8-SNAPSHOT oak-api
But not at a package level, provided the package versions are the same, which
they are.
Since OSGi at build and runtime only looks at the package versions that are
provided, you can use any bundle that contains the correct package
version. Sling Servlets Get bundle in the patch has oak-core 1.6 using
org.apache.jackrabbit.oak.api.conversion v 1.0.0. So its MANIFEST.MF will
import org.apache.jackrabbit.oak.api.conversion=[1.0.0,2.0.0).
oak-api/1.8-SNAPSHOT also provides that package so Sling Servlets GET released
using oak-core 1.6 will also work in a OSGi container satisfied by
oak-api/1.8-SNAPSHOT
Obviously if Sling Servlets GET depends on a behaviour not expressed in the API
or binds to some other service or package only available in Oak core 1.6, then
it wont work. but for this patch, and the ones in Sling it uses the api and
a common package version.
I hope that makes sense.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
>Assignee: Amit Jain
> Fix For: 1.8, 1.7.9
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194387#comment-16194387
]
Ian Boston commented on OAK-6575:
-
Doc: Anyone deploying will find it hard to do so without the instructions in
the README.md file
https://gist.github.com/ieb/f9e044e4033b8f810238fe9a27eeaa78#file-readme-md
Could the 1.6 pull request also be merged please ?
Reasoning:
1.6: Sling will remain blocked until there is 1.6 release. Patch was backported
to unblock Sling and allow a release of the components there which require the
new API package.
The API in 1.6 is part of oak-core, in 1.8-SNAPSHOT it is separate.
IIUC Sling won't bind to an unstable (1.7.x) release of Oak so without a 1.6
release will remain blocked until there is a 1.8.0 release of Oak, which will
all continue to block downstream work.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
>Assignee: Amit Jain
> Fix For: 1.8, 1.7.9
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Boston updated OAK-6575:
Fix Version/s: 1.6.6
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8, 1.6.6
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Boston updated OAK-6575:
Flags: Patch
Any chance the patch could be merged to unblock downstream work ?
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185732#comment-16185732
]
Ian Boston commented on OAK-6575:
-
Configuration information
https://gist.github.com/ieb/f9e044e4033b8f810238fe9a27eeaa78
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185710#comment-16185710
]
Ian Boston commented on OAK-6575:
-
Pull requests:
Oak Trunk https://github.com/apache/jackrabbit-oak/pull/69
Oak 1.6 Branch https://github.com/apache/jackrabbit-oak/pull/70
Downstream the patch to Sling is
https://github.com/apache/sling/pull/252
Documentation on configuration to follow in a Gist.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185665#comment-16185665
]
Ian Boston commented on OAK-6575:
-
Tested against Sling Trunk using a patch backported to Oak 1.6. The patches are
https://github.com/apache/jackrabbit-oak/compare/1.6...ieb:OAK-6575-1.6-3?expand=1
and in Sling
https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2?expand=1
There were 2 fixes to the Oak patch.
Its not viable to put a 4096 bit PKCS8 encoded private key into a OSGi
property. The Sling provisioning model truncates it and its hard to manage, so
this has been changed to a file. The file can be relative, but must be
resolvable from the working directory of the JVM.
In addition a configurable minimum size limit has been added so that small
files, can be streamed via the JVM if desired.
Also added is a log line to allow audit of all external urls. This might want
to be at debug level, but for the moment it is at info level.
This changes need to be forward ported to Oak Trunk.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173382#comment-16173382
]
Ian Boston commented on OAK-6575:
-
The pull request identified some issues from the community and I need to do
some further testing, so its been closed for now.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171977#comment-16171977
]
Ian Boston commented on OAK-6575:
-
Opened SLING-7140 to track downstream work.
Patch has been discussed on Sling dev pending a suitable release of Oak.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171962#comment-16171962
]
Ian Boston commented on OAK-6575:
-
Pull request created,
https://github.com/apache/jackrabbit-oak/pull/67
Based on discussions on list this is ready to go.
The commits should be squashed.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr, security
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166284#comment-16166284
]
Ian Boston commented on OAK-6575:
-
Thanks for the corrections. Made the changes and updated the branch.
I have discussed the private key with deployers at Adobe and the view was that
a password OSGi property would be better than a file system file as it would
hide the key from view in the UI and leave any protection of that key upto the
OSGi framework. There was also concern that using an filename might cause
deployment problems is the location was not consistent over all instances. I
dont have any evidence to support that concern, but accepted it as stated.
If there is consensus that this should be a file on the filesystem, then I can
easily change the patch.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156722#comment-16156722
]
Ian Boston commented on OAK-6575:
-
Based on discussions on oak-dev and URIProvider proposal from [~mduerig]
https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-3?expand=1
This removes the AdapterFactory from Oak completely
which will require a new bundle + patch in Sling
https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3?expand=1
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153541#comment-16153541
]
Ian Boston commented on OAK-6575:
-
After discussion on oak-dev, 2 more alternative patches.
https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-1?expand=1
This drops the OSGi AdapterManager/AdapterFactory in favour of a non OSGi
static pattern. Implementations of the AdapterFactory self register rather than
rely on OSGi doing the wiring. This is probably an IoC anti pattern, but does
avoid exposing the AdapterFactory/AdapterManager outside Oak.
https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-2?expand=1
This drops the AdapterManager concept completely and attempts to get from Value
to URI using mix in interfaces and instanceof. I cant be certain it manages to
do this as there is a disconnect between Blob, Blobstore and DataStore
implementations with no guarantee that a BlobStore as seen by the Blob
implementation actually implements DataStore, or the Blob that is exposed in
the JCR Value (implemented by OakValue) actually connects to the correct
DataStore of it it connects to a FileDatastore cache on local disk. I could
only wire this as far as I did with API changes. I may have broken some of the
new multi node store and multi datastore code used for 0DT in the process. An
Oak committer with global knowledge will probably be able to do better.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152298#comment-16152298
]
Ian Boston commented on OAK-6575:
-
Changes made and pull request created.
https://github.com/apache/jackrabbit-oak/pull/66
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152217#comment-16152217
]
Ian Boston commented on OAK-6575:
-
I will make the changes (CloudFrontS3SignedUrlAdapterFactory, 1.11.24) to the
patch in a moment and report back here when done.
I thought long and hard about URI vs PrivateURI vs PublicURI before suggesting
PrivateURI. Here is my logic.
This feature potentially leaks direct access to protected content managed by
Oak, arguably bypassing Oak security. Therefore it must as secure by default as
possible and minimise the risk of mistakes. The proposed usage of a uri obtains
by adapting to PublicURI or PrivateURI is clear. A Private URI is something
that is private and should not be sent out to the world. A PublicURI is public
and can be sent out to the world. Its unlikely a programmer in Sling or AEM is
going to make a mistake. The URI class has a big benefit in that it is an
existing API part of the JRE and so doesn't require any special treatment from
a dependency point of view. However, adapting to URI gives no indication of
where it is safe to use. If the URI is public, then its safe to use everywhere.
If the URI is private, then it is not. For that reason using a URI class to
indicate a private uri risks the developer in Sling or AEM using it, without
any knowledge, as a public uri breaching the security promises of this feature.
For these reasons I suggested keeping URI to represent the public URI, since it
can be used anywhere, and introducing a new PrivateURI so that private uris
were only generated by developers that had added the dependency to the API and
were using it without doubt over its intended purpose.
I hope my reasoning makes sense. If you dont agree then I think we need to
introduce new APIs and not use URI or URL at all. This will certainly cause
problems downstream as this feature will require a new dependency, directly on
Oak, to be utilized. I have no doubt that will generate a RTC and delay as it
will require changes to the pom.xml.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
>
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150373#comment-16150373
]
Ian Boston commented on OAK-6575:
-
In which case, is the patch ok for inclusion ?
Should I do a pull request ?
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150196#comment-16150196
]
Ian Boston commented on OAK-6575:
-
bq. One usecase would be in workflow processing where some processing service
needs to access the blob content. Such access need not go via cdn and would be
better of with s3 access via signed uri.
This use case is valid if the workflow processing service is not running inside
an Oak JVM. There is a security argument that since Oak cant differentiate
between how a URI exposed outside the Oak JVM might be used, it should make all
URIs equal. Putting that to one side.
This needs a new Oak API in the oak-api bundle,
org.apache.jackrabbit.oak.api.PrivateURI. Unless the API is in a dedicated
package, it would mean versioning the Oak API package. Each DS that supported
PrivateURIs would need a seperate AdapterFactory that emits an implementation
of this class. For S3 it would emit a S3 Signed URL.
This issue was to provide a external URL, not a private URL. Do you want to add
provision of a private URL here or leave it to a later issue ?
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[ https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150168#comment-16150168 ] Ian Boston commented on OAK-6575: - Could you share the use case that creates a requirement for Sling or AEM to programmatically decide between a CloudFrontSignedBinary and a S3SignedBinary and explicitly request one or the other from Oak ? Background: The initial PoC introduced a new SignedBinary API to do the same as you sugest. After discussion internally, rapidly moved to oak-dev it was proposed that URI or URL was used. The reasoning behind this proposal was not fully exposed so I will share my view here: 1. Exposing APIs specific to each DS implementation would require that code in Sling or AEM binds to all possible DS implementations which is undesirable. 2. Not adding new APIs became a requirement for this patch. 3. The DS implementation bundle should decide decide what type of signed URL, if any, can be emitted on a binary by binary basis. 4. A deployer should be able to select which signed URL implementations are active, and possibly which signed URLs are emitted based on path or properties of the binary. 5. The class that is emitted should cover all types of URI, not just URLs. For this reason the discussions on oak-dev reached consensus on emitting a URI rather than a new API interface implementation. Initially I thought that was not the right decision, but the more I think about it, the code in Sling or AEM only needs a URI and should not be forced to learn about the implementation details of the selected datastore. The current patch allows a DS implementation to implement many AdapterFactories, each one capable of being configured to be active, and capable of emitting a signed URL on a case by case basis. Code in Sling or AEM does not need to know what type of singed URL is available, and must not be allowed to override a deployer in making that decision. For instance, a deployer using CloudFront will typically not allow direct access to the S3 buckets when using CloudFront shielding the S3 buckets from public access. Hence a deployer would only configure the CloudFront AdapterFactory to be active ensuring that Oak could not be coerced by Sling or AEM code to emit the wrong signed URL potentially deployment driven security policy. There is one alternative implementation pattern that was briefly discussed and rejected. Each DS implementation implement its own ServletFilter that when active performs a http redirect for request to Binary values that should be redirected. This would be completely internal to the DS implementation with no need for any OakConversionService or AdapterFactory pattern. This was rejected as it would require teh DS implementations to bind to the Sling Resource API in order to correctly resolve the OakValue. So to recap. the URI pattern emitted from a configurable AdapterFactory implementation which may emit a null is adaption to its type of signed url supports the use case where there may be many implementations of an external URI, signed or otherwise, by many DS implementations. It does this without introducing new APIs and without exposing implementation details of any DS. If you think there is some aspect of the use case that has not been considered, could you detail that use case for consideration. > Provide a secure external URL to a DataStore binary. > > > Key: OAK-6575 > URL: https://issues.apache.org/jira/browse/OAK-6575 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: blob, core, jcr >Reporter: Ian Boston > Fix For: 1.8 > > > Where the DataStore is a DataStore that may be accessed over an independent > API it would be advantageous for Oak to provide a secure URL to allow direct, > read only access to the current immutable instance of that binary. The term > "secure" needs to be defined, but typically it would a URL that is valid for > a appropriately short length of time to ensure that the risk of the URL being > used by a user that it was not intended for, is minimised. It should also > ensure that anyone in possession of the URL could not use the information in > the url to create a valid URL or a valid URL to a different binary. > One example of such a URL might be a AWS Signed URL as used by AWS CloudFront > to access private content. The signed url being signed by a private key known > only to the Oak instance and the the CloudFront or S3 instance. The signed > url having a significantly low ttl so that a redirect by the same client > would work. > Oak should only emit these URLs to sessions that could otherwise read the > binary directly from Oak, and Oak should be in complete control of the nature > of the url and the security mechanisms applied to the URL.
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148884#comment-16148884
]
Ian Boston commented on OAK-6575:
-
Now using the AWS SDK.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148813#comment-16148813
]
Ian Boston commented on OAK-6575:
-
The patch opens the way to allow other providers including direct to S3 and
even Azure Blobs. You could split it into multiple patches each one depending
on the other, and then the oak team could discuss each one in turn, or use this
patch to get all the features in to support the concept and discuss it in one
go with a concrete example of usage. I did the patch this way as concrete usage
normally keeps code simple and to the point rather than adding features just in
case.
The Official AWS CloudFront API could be used. It adds a minimum of 0.5MB of
extra Jar files which is why I didn't use it in the patch.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147137#comment-16147137
]
Ian Boston commented on OAK-6575:
-
Path now contains test coverage,
Ready for review.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145718#comment-16145718
]
Ian Boston commented on OAK-6575:
-
Patch has been updated based on pointers.
No unit test coverage as yet.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145004#comment-16145004
]
Ian Boston commented on OAK-6575:
-
Can do, but will need some architectural style guidance as I dont know Oak that
well.
Where does Oak stand on OSGi and how does it want to configure things like the
DataStore private key ?
In Sling I would use OSGi components throughout, but IIUC Oak needs to run
outside OSGi (oak-run)
I used a static factory pattern for the AdapterManager, and for the
S3SignedUrlAdapterFactory component I provided 2 constructors, one for OSGi the
other to be called with a full set or properties. In both cases the component
registered itself with the AdapterManager. AFAIK, this is an OSGi anti pattern.
In OSGi the AdapterManager should be a component with bind and unbind methods
to allow OSGi to register implementations of an AdapterFactory service. It
should use OSGi properties to get the list of classes the AdapterFactory can
adapt to and it should use a property for the priority. I could do that, but
then it wouldnt work for oak-run which has no OSGi container.
In OSGi all configuration would come via OSGi configuration properties. In
oak-run there is none.
Perhaps none of this is relevant for oak-run since it wont emit URLs, in which
case it should be all OSGi, in which case where should the AdapterManager be
run from, as oak core has no OSGi dependencies.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140199#comment-16140199
]
Ian Boston commented on OAK-6575:
-
There is a very rough PoC at
https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-6575?expand=1
for further discussion. I am sure there is lots wrong with it.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is present, and Oak decides no
> URL is appropriate, then no-op.
> Only if the Oak DS implementation being used supports the external URL and
> Oak decides it is appropriate, will a url be available and a redirect
> performed.
> I have used AWS S3 URLs as an example, however the approach should be
> applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6575) Provide a secure external URL to a DataStore binary.
Ian Boston created OAK-6575:
---
Summary: Provide a secure external URL to a DataStore binary.
Key: OAK-6575
URL: https://issues.apache.org/jira/browse/OAK-6575
Project: Jackrabbit Oak
Issue Type: Bug
Reporter: Ian Boston
Where the DataStore is a DataStore that may be accessed over an independent API
it would be advantageous for Oak to provide a secure URL to allow direct, read
only access to the current immutable instance of that binary. The term
"secure" needs to be defined, but typically it would a URL that is valid for a
appropriately short length of time to ensure that the risk of the URL being
used by a user that it was not intended for, is minimised. It should also
ensure that anyone in possession of the URL could not use the information in
the url to create a valid URL or a valid URL to a different binary.
One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
to access private content. The signed url being signed by a private key known
only to the Oak instance and the the CloudFront or S3 instance. The signed url
having a significantly low ttl so that a redirect by the same client would
work.
Oak should only emit these URLs to sessions that could otherwise read the
binary directly from Oak, and Oak should be in complete control of the nature
of the url and the security mechanisms applied to the URL.
The viability of the approach has been investigated showing that given a JCR
Binary it is possible to get the Oak Blob Content Identifier using
ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
knowing the way in which the DataStore implementation transforms that into a
pointer into the datastore implementation form a URL to be made secure.
To achieve the above, internal implementation details specific to the Oak
DataStore implementation are required, hence this request to implement as a
part of Oak rather than to reverse engineer in some external project.
Since API changes are often significant using the Sling AdapaterFactory
approach would allow a ServletFilter to selectively use the URL in a redirect,
avoiding any new API methods to existing Oak APIs. A new interface might be
required, in the example below that interface is SignedBinaryURL.
{code}
public void doFilter(ServletRequest servletRequest, ServletResponse
servletResponse, FilterChain filterChain) throws IOException, ServletException {
if ( servletRequest instanceof SlingHttpServletRequest &&
servletResponse instanceof SlingHttpServletResponse) {
if ("GET".equals(((SlingHttpServletRequest)
servletRequest).getMethod())){
Resource resource = ((SlingHttpServletRequest)
servletRequest).getResource();
SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
if (url != null) {
((SlingHttpServletResponse)
servletResponse).sendRedirect(url.getURL());
return;
}
}
}
filterChain.doFilter(servletRequest, servletResponse);
}
{code}
If the AdapterFactory to go from Binary to SingedBinaryURL is not present then
url will always be null, and no-op. If it is present, and Oak decides no URL is
appropriate, then no-op.
Only if the Oak DS implementation being used supports the external URL and Oak
decides it is appropriate, will a url be available and a redirect performed.
I have used AWS S3 URLs as an example, however the approach should be
applicable (and pluggable) to most REST based APIs to private binary content.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6554) Update Dropwizard Metrics bundles to 3.2 to eliminate sun.misc dependency.
Ian Boston created OAK-6554: --- Summary: Update Dropwizard Metrics bundles to 3.2 to eliminate sun.misc dependency. Key: OAK-6554 URL: https://issues.apache.org/jira/browse/OAK-6554 Project: Jackrabbit Oak Issue Type: Task Reporter: Ian Boston Also mentioned in SLING-7047. Not certain which project needs to change. Possibly both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-5978) Allow deployers to configure a query time limit
[ https://issues.apache.org/jira/browse/OAK-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940424#comment-15940424 ] Ian Boston commented on OAK-5978: - It's different for the reasons you stated. When a query causes the working set of an active repository to increase beyond the resources that are available, everything slows down. That shows up as page faulting in TarMK, page faults with MongoMK on MMAPv1 and saturated IO in MongoMK WT. Counting the number of results or looking for traversal is no longer a good way of detecting a query that should be canceled. Starting a timer at the start of the query and checking the time elapsed at convenient points throughout the query execution would be a better way of protecting normal operation from roque queries. That suggestion is based on my very limited knowledge of how Oak works internally. There are probably much better ways of doing it based on elapsed time. Perhaps it can't be implemented in Oak. The current limits don't appear to protect against outage. > Allow deployers to configure a query time limit > --- > > Key: OAK-5978 > URL: https://issues.apache.org/jira/browse/OAK-5978 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: query >Affects Versions: 1.6.1 >Reporter: Ian Boston > > Currently when a query takes a long time to complete, Oak allows it to > continue to completion, which can under certain circumstances result in > resource exhaustion and slow performance for all operations depending on Oak. > This feature request is to apply a deployer configurable time limit to all > queries. If the query exceeds that time limit, it is canceled with a suitable > exception, so that it consumes no more resources and the code or user that > submitted the query is notified. > Ideally this needs to work while the query is executing inside Oak rather > than waiting to return via the Oak API before being canceled. Cancelation > should result in minimal resource usage. > At this stage, this is an idea. It may not be possible to implement. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OAK-5978) Allow deployers to configure a query time limit
Ian Boston created OAK-5978: --- Summary: Allow deployers to configure a query time limit Key: OAK-5978 URL: https://issues.apache.org/jira/browse/OAK-5978 Project: Jackrabbit Oak Issue Type: New Feature Components: query Affects Versions: 1.6.1 Reporter: Ian Boston Currently when a query takes a long time to complete, Oak allows it to continue to completion, which can under certain circumstances result in resource exhaustion and slow performance for all operations depending on Oak. This feature request is to apply a deployer configurable time limit to all queries. If the query exceeds that time limit, it is canceled with a suitable exception, so that it consumes no more resources and the code or user that submitted the query is notified. Ideally this needs to work while the query is executing inside Oak rather than waiting to return via the Oak API before being canceled. Cancelation should result in minimal resource usage. At this stage, this is an idea. It may not be possible to implement. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OAK-5588) Improve Session stats.
Ian Boston created OAK-5588: --- Summary: Improve Session stats. Key: OAK-5588 URL: https://issues.apache.org/jira/browse/OAK-5588 Project: Jackrabbit Oak Issue Type: Wish Reporter: Ian Boston Currently each session has a SessionsStats MBean. Omongst other things it records the total number or refresh operations. It also records the rate of refresh operations, although this number in its current form is not usefull as the rate is the number of refresh operations/session lifetime. It would be much better to have a set of stats related to classes of users that recorded proper metrics in a consistent way. eg 1 metric set per service-user, 1 for the admin user and perhaps 1 for all normal users. Each would record m1,m5,m15 rates, total count, p50,p75,p95,p99,p999 durations with mean and stdev then 2 sets of metrics could be compared and monitored without having to look at the code to work out how the metric was calculated. Oak has metrics support to do this, minimal code would be required. I dont think it would be viable to have 1 metric per unique session (too much overhead, too much data, good for devs but bad for production), and in fact having 1 JMX MBean per unique session is likely to cause problems with everything connected to JMX even the ManagementServer can cope. Same goes for the other proliferation of MBeans in the Oak 1.6. Perhaps a review of JMX in Oak is due. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (OAK-4726) Add Metrics coverage to the Lucene Indexer
[ https://issues.apache.org/jira/browse/OAK-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695074#comment-15695074 ] Ian Boston edited comment on OAK-4726 at 11/25/16 7:29 AM: --- New patch lgtm. The names need to be unique and understandable without looking in documentation and globally unique. Perhaps prefix with oak-lucene rather than lucene, eg oak-lucene-- to allow some possible future where sling-lucene-- corp-lucene-- also exist. One observation from dealing with a production system with 1000s of metrics: Classnames are good because they very quickly identify the location in code where the metric is. They are bad because they are long. With 1000s of metrics, short names like lucene-- are very time consuming to locate. There is no clue as to which git repo the metric comes from or the class. They are bad because they are long. But that's Java for you. To avoid this Changing org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter-:data-close to o.a.j.o.p.index.lucene.writer.DefaultIndexWriter-:data-close Would be enough was (Author: ianeboston): New patch lgtm. The names need to be unique and understandable without looking in documentation and globally unique. Perhaps prefix with oak-lucene rather than lucene, eg oak-lucene-- to allow some possible future where sling-lucene-- corp-lucene-- also exist. > Add Metrics coverage to the Lucene Indexer > -- > > Key: OAK-4726 > URL: https://issues.apache.org/jira/browse/OAK-4726 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.5.8 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.6, 1.5.17 > > Attachments: OAK-4726.patch > > > Although there are some stats surrounding the Lucene indexing processes it > would be useful to have metrics style stats available. > Looking at the code, the implementations IndexWriter look like the right > place to add the metrics. These could be global aggregate metrics, ie one set > of metrics covering all IndexWriter implementations, or there could be > individual metrics for each Lucene index definition. The latter would be more > useful as it will allow detailed stats on individual metrics. > These metrics will only give information on the writer operations, and not > any Tokenizing operations. > Patch in the form of a pull request will flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4726) Add Metrics coverage to the Lucene Indexer
[ https://issues.apache.org/jira/browse/OAK-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695074#comment-15695074 ] Ian Boston commented on OAK-4726: - New patch lgtm. The names need to be unique and understandable without looking in documentation and globally unique. Perhaps prefix with oak-lucene rather than lucene, eg oak-lucene-- to allow some possible future where sling-lucene-- corp-lucene-- also exist. > Add Metrics coverage to the Lucene Indexer > -- > > Key: OAK-4726 > URL: https://issues.apache.org/jira/browse/OAK-4726 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.5.8 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.6, 1.5.17 > > Attachments: OAK-4726.patch > > > Although there are some stats surrounding the Lucene indexing processes it > would be useful to have metrics style stats available. > Looking at the code, the implementations IndexWriter look like the right > place to add the metrics. These could be global aggregate metrics, ie one set > of metrics covering all IndexWriter implementations, or there could be > individual metrics for each Lucene index definition. The latter would be more > useful as it will allow detailed stats on individual metrics. > These metrics will only give information on the writer operations, and not > any Tokenizing operations. > Patch in the form of a pull request will flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
[ https://issues.apache.org/jira/browse/OAK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653769#comment-15653769 ] Ian Boston commented on OAK-4943: - The WebConsole to analyse the Segments can be found here https://github.com/ieb/oakui > Keep Lucene Commits so that if the Segments file gets corrupted recovery can > be attempted. > -- > > Key: OAK-4943 > URL: https://issues.apache.org/jira/browse/OAK-4943 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.6 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.6 > > > Currently, Lucene deletes all previous generations of the segments files as > it has it uses the default IndexDeletionPolicy. Changing this to an > IndexDeletionPolicy that keeps a number of previous segments files will allow > an operator to find the most recent valid commit when the current segments > file reports corruption. The patch found at > https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits > keeps 10 previous commits. > A more sophisticated policy could be used to save commits non-linearly over > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3512) Synchronous index rebuilds can be performed by all commits once the reindex propery is true
[ https://issues.apache.org/jira/browse/OAK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved OAK-3512. - Resolution: Won't Fix No more occurrences of this issue reported, so closing. > Synchronous index rebuilds can be performed by all commits once the reindex > propery is true > --- > > Key: OAK-3512 > URL: https://issues.apache.org/jira/browse/OAK-3512 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.4 >Reporter: Ian Boston > > If reindex=true gets persisted to the database, then every commit triggers a > rebuild of the index. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
[ https://issues.apache.org/jira/browse/OAK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615248#comment-15615248 ] Ian Boston edited comment on OAK-4943 at 10/28/16 12:30 PM: FYI: I am working on a Sling WebConsole UI to provide detail on all commits, including the age of each commit and functionality to analyse each commit, validate, rollback and download the commit. This is FYI as it doesn't fit inside the Oak project and may be better in Sling or a contrib repo. was (Author: ianeboston): FYI: I am working on a Sling WebConsole UI to provide detail on all commits, including age of each commit and functionality to analyse each commit, validate, rollback and download the commit. > Keep Lucene Commits so that if the Segments file gets corrupted recovery can > be attempted. > -- > > Key: OAK-4943 > URL: https://issues.apache.org/jira/browse/OAK-4943 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.6 >Reporter: Ian Boston > > Currently, Lucene deletes all previous generations of the segments files as > it has it uses the default IndexDeletionPolicy. Changing this to an > IndexDeletionPolicy that keeps a number of previous segments files will allow > an operator to find the most recent valid commit when the current segments > file reports corruption. The patch found at > https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits > keeps 10 previous commits. > A more sophisticated policy could be used to save commits non-linearly over > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
[ https://issues.apache.org/jira/browse/OAK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615248#comment-15615248 ] Ian Boston commented on OAK-4943: - FYI: I am working on a Sling WebConsole UI to provide detail on all commits, including age of each commit and functionality to analyse each commit, validate, rollback and download the commit. > Keep Lucene Commits so that if the Segments file gets corrupted recovery can > be attempted. > -- > > Key: OAK-4943 > URL: https://issues.apache.org/jira/browse/OAK-4943 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.6 >Reporter: Ian Boston > > Currently, Lucene deletes all previous generations of the segments files as > it has it uses the default IndexDeletionPolicy. Changing this to an > IndexDeletionPolicy that keeps a number of previous segments files will allow > an operator to find the most recent valid commit when the current segments > file reports corruption. The patch found at > https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits > keeps 10 previous commits. > A more sophisticated policy could be used to save commits non-linearly over > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
[ https://issues.apache.org/jira/browse/OAK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615246#comment-15615246 ] Ian Boston commented on OAK-4943: - OAK-4943 is a better solution to the problem > Keep Lucene Commits so that if the Segments file gets corrupted recovery can > be attempted. > -- > > Key: OAK-4943 > URL: https://issues.apache.org/jira/browse/OAK-4943 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.6 >Reporter: Ian Boston > > Currently, Lucene deletes all previous generations of the segments files as > it has it uses the default IndexDeletionPolicy. Changing this to an > IndexDeletionPolicy that keeps a number of previous segments files will allow > an operator to find the most recent valid commit when the current segments > file reports corruption. The patch found at > https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits > keeps 10 previous commits. > A more sophisticated policy could be used to save commits non-linearly over > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581731#comment-15581731 ] Ian Boston commented on OAK-3547: - Using Lucene directly to manage generations of the segments file is covered in OAK-4943 > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
Ian Boston created OAK-4943: --- Summary: Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted. Key: OAK-4943 URL: https://issues.apache.org/jira/browse/OAK-4943 Project: Jackrabbit Oak Issue Type: Improvement Components: lucene Affects Versions: 1.6 Reporter: Ian Boston Currently, Lucene deletes all previous generations of the segments files as it has it uses the default IndexDeletionPolicy. Changing this to an IndexDeletionPolicy that keeps a number of previous segments files will allow an operator to find the most recent valid commit when the current segments file reports corruption. The patch found at https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits keeps 10 previous commits. A more sophisticated policy could be used to save commits non-linearly over time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved OAK-3547. - Resolution: Won't Fix Fix Version/s: (was: 1.6) Will be dealt with under OAK-4805 and other issues. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551486#comment-15551486 ] Ian Boston commented on OAK-3547: - The original patch was written such a long time ago, I don't think IndexCopier was present, or at least the deployment the patch was targeting did not have writeOnCopy etc enabled or possibly available. The impl of OakIndexFile is suboptimal for Lucene usage, as it loads chunks of the index into memory as byte[] to perform seek, whereas FSDirectory uses OS level native code to seek, hence it makes no sense to use OakDirectory any more. FSDirectory should be used by whatever means necessary. Might be an idea to delete or deprecate OakDirectory, so its not used for opening lucene indexes. The patch is in a state where it should not be applied or used. It can't efficiently determine corruption without direct access to the underlying file, which is abstracted by Oak. With the benefit of hindsight, the patch should be in IndexCopier to prevent a bad segments.gen file failing the index. We should close this issue as the patch isn't valid any more. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > Fix For: 1.6 > > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529963#comment-15529963 ] Ian Boston commented on OAK-3547: - Some inspection of all the metrics captured indicates that the method [1] is the main cause of differences in times taken to open, sync and close the directory index, as each operation must generate a fresh SHA1 from all the index files. If it was possible to rely on some other mechanism for checking the integrity of each file, this checksum could be replaced with something much simpler, like file length which would avoid generating a sha1 on each operation. This would then rely on Oak Lucene managing the recovery rather than the Oak Directory listing being self healing. To achieve this, a drop() method on the OakDirectory might be required to drop the current generation of the listing on demand. 1 https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-3547#diff-28ec89220db72ab858b9eb25927c2a29R1026 > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > Fix For: 1.6 > > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[
https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529581#comment-15529581
]
Ian Boston commented on OAK-3547:
-
I have tested the patch in Sling and AEM 6.2+Oak 1.6, no functional regressions
or errors were seen.
There is an indication that opening the index is significantly slower with the
patch as, in order to verify all the files in the index the files are read and
a sha1 is generated from those files to ensure that the files are not damaged.
Instrumenting the OakDirectory constructor call, which in the generational
version validates the contents of the directory before opening, reveals.
WIthout OAK-3547 patch
{code}
t count max meanmin stddev p50 p75 p95 p98
p99 p999mean_rate m1_rate m5_rate m15_raterate_unit
duration_unit
1475066389 40 10.221184 0.1488390.022663
0.8821450.0722110.0722110.0784190.15831 0.15831
10.221184 0.07476 0.00111 0.2274860.687796calls/second
milliseconds
{code}
With OAK-3547 patch
{code}
t count max meanmin stddev p50 p75 p95 p98
p99 p999mean_rate m1_rate m5_rate m15_raterate_unit
duration_unit
1475063657 40 571.811075 378.76475 0.125733
209.479519 492.429012 492.429012 492.429012 571.811075
571.811075 571.811075 0.0823290.0026450.269424
0.727275calls/secondmilliseconds
{code}
How much the difference is will depend on the size of the index files. The
patch may also transfer the IO read operation on the index from outside the
OakDirectory constructor to inside the OakDirectory constructor, so these
readings may or may not be significant.
If they prove to be significant, then the SHA1 on files could be dropped on
every open directory open and some other check be performed. Other checks won't
be as robust as a full SHA1 check.
> Improve ability of the OakDirectory to recover from unexpected file errors
> --
>
> Key: OAK-3547
> URL: https://issues.apache.org/jira/browse/OAK-3547
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.4
>Reporter: Ian Boston
> Fix For: 1.6
>
>
> Currently if the OakDirectory finds that a file is missing or in some way
> damaged, and exception is thrown which impacts all queries using that index,
> at times making the index unavailable. This improvement aims to make the
> OakDirectory recover to a previously ok state by storing which files were
> involved in previous states, and giving the code some way of checking if they
> are valid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516665#comment-15516665 ] Ian Boston commented on OAK-3547: - There is now unit test coverage to validate that the correct previous generation of the directory will be opened in the event the underlying files are lost or damaged. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > Fix For: 1.6 > > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513619#comment-15513619 ] Ian Boston commented on OAK-3547: - The branch at [1] has been updated to work with Trunk at r1761930 and passes all unit tests in the bundle build. Doing full build now to verify no regressions. 1 https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-3547 cc: [~chetanm] > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > Fix For: 1.6 > > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4726) Add Metrics coverage to the Lucene Indexer
[ https://issues.apache.org/jira/browse/OAK-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454970#comment-15454970 ] Ian Boston commented on OAK-4726: - For existing instances, there is a config to instrument the same methods as the patch instruments using Sling metrics at https://github.com/ieb/slingmetrics/blob/master/config/metrics_indexing.yaml. This wont generate one metric set per index, as the patch does, but will allow earlier versions of Oak, with the same class/method names to generate metrics on their indexing behaviour. See the README.md in that repo to see how to use Sling Metrics. > Add Metrics coverage to the Lucene Indexer > -- > > Key: OAK-4726 > URL: https://issues.apache.org/jira/browse/OAK-4726 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.5.8 >Reporter: Ian Boston > > Although there are some stats surrounding the Lucene indexing processes it > would be useful to have metrics style stats available. > Looking at the code, the implementations IndexWriter look like the right > place to add the metrics. These could be global aggregate metrics, ie one set > of metrics covering all IndexWriter implementations, or there could be > individual metrics for each Lucene index definition. The latter would be more > useful as it will allow detailed stats on individual metrics. > These metrics will only give information on the writer operations, and not > any Tokenizing operations. > Patch in the form of a pull request will flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4726) Add Metrics coverage to the Lucene Indexer
[ https://issues.apache.org/jira/browse/OAK-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454948#comment-15454948 ] Ian Boston commented on OAK-4726: - Added metrics for the tokenizing operations. > Add Metrics coverage to the Lucene Indexer > -- > > Key: OAK-4726 > URL: https://issues.apache.org/jira/browse/OAK-4726 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.5.8 >Reporter: Ian Boston > > Although there are some stats surrounding the Lucene indexing processes it > would be useful to have metrics style stats available. > Looking at the code, the implementations IndexWriter look like the right > place to add the metrics. These could be global aggregate metrics, ie one set > of metrics covering all IndexWriter implementations, or there could be > individual metrics for each Lucene index definition. The latter would be more > useful as it will allow detailed stats on individual metrics. > These metrics will only give information on the writer operations, and not > any Tokenizing operations. > Patch in the form of a pull request will flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4726) Add Metrics coverage to the Lucene Indexer
[ https://issues.apache.org/jira/browse/OAK-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454825#comment-15454825 ] Ian Boston commented on OAK-4726: - Patch that instruments the IndexWriter can be found at https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-4726 It generates a set of metrics for each index based on the name of the index. [~chetanm] would this help ? Should the Tokenizer also have metrics ? > Add Metrics coverage to the Lucene Indexer > -- > > Key: OAK-4726 > URL: https://issues.apache.org/jira/browse/OAK-4726 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.5.8 >Reporter: Ian Boston > > Although there are some stats surrounding the Lucene indexing processes it > would be useful to have metrics style stats available. > Looking at the code, the implementations IndexWriter look like the right > place to add the metrics. These could be global aggregate metrics, ie one set > of metrics covering all IndexWriter implementations, or there could be > individual metrics for each Lucene index definition. The latter would be more > useful as it will allow detailed stats on individual metrics. > These metrics will only give information on the writer operations, and not > any Tokenizing operations. > Patch in the form of a pull request will flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4726) Add Metrics coverage to the Lucene Indexer
Ian Boston created OAK-4726: --- Summary: Add Metrics coverage to the Lucene Indexer Key: OAK-4726 URL: https://issues.apache.org/jira/browse/OAK-4726 Project: Jackrabbit Oak Issue Type: Improvement Components: lucene Affects Versions: 1.5.8 Reporter: Ian Boston Although there are some stats surrounding the Lucene indexing processes it would be useful to have metrics style stats available. Looking at the code, the implementations IndexWriter look like the right place to add the metrics. These could be global aggregate metrics, ie one set of metrics covering all IndexWriter implementations, or there could be individual metrics for each Lucene index definition. The latter would be more useful as it will allow detailed stats on individual metrics. These metrics will only give information on the writer operations, and not any Tokenizing operations. Patch in the form of a pull request will flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4412) Lucene hybrid index
[
https://issues.apache.org/jira/browse/OAK-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395191#comment-15395191
]
Ian Boston commented on OAK-4412:
-
I assume you mean using the NRT search feature only for the local index and not
the index that is committed into Oak. IIRC from experiments in 2011, Lucene's
NRT feature is not durable and without a separate WAL looses data. Also, with
the same index on every cluster member, the index gains no scalability benefits
from the cluster, but that is a bigger issue which would need a significant
rethink.
> Lucene hybrid index
> ---
>
> Key: OAK-4412
> URL: https://issues.apache.org/jira/browse/OAK-4412
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.6
>
> Attachments: OAK-4412.patch
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic.
> The asynchronous index would be an answer here (as the index update won't be
> made in the client request thread), but the AEM requires the updates to be
> visible immediately in order to work properly.
> The idea here is to enhance the existing asynchronous Lucene index with a
> synchronous, locally-stored counterpart that will persist only the data since
> the last Lucene background reindexing job.
> The new index can be stored in memory or (if necessary) in MMAPed local
> files. Once the "main" Lucene index is being updated, the local index will be
> purged.
> Queries will use an union of results from the {{lucene}} and
> {{lucene-memory}} indexes.
> The {{lucene-memory}} index, as a local stored entity, will be updated using
> an observer, so it'll get both local and remote changes.
> The original idea has been suggested by [~chetanm] in the discussion for the
> OAK-4233.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387264#comment-15387264
]
Ian Boston commented on OAK-4581:
-
Only consider my suggestions for a WAL if the existing Journal implementation
is unsuitable. It seems the existing journal is suitable, so ignore WAL
references.
---
Assuming the persisted journal capacity is only bounded by available disk
space, the need to reference a remote journal to conserve queue capacity in
exchange for additional computation or network is probably unnecessary. I
assume "remote" means remote to the VM where the JVM is running and not some
other interpretation of remote.
---
Events lagging too far behind (data latency) is:
1. something that will (I assume) be caused by the commit traffic an
application is subjecting Oak to, rather than caused by Oak internal commits.
2. something an application will care about.
3. given sufficient and appropriate feedback (queue latency metric) something
an application developer will be keen to act on during dev and QE before
production.
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
> Fix For: 1.6
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis.
> This would allow each Listener to "pull" content changes and generate diff as
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2]
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3]
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4529) DocumentNodeStore does not have a repository software version range check.
[ https://issues.apache.org/jira/browse/OAK-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361577#comment-15361577 ] Ian Boston commented on OAK-4529: - The default behavior should be safe and do no damage, as a user of oak-run or other utilities will not know they are about to corrupt the repository, and when they do, it will be too late. ie safe by default. It should be possible to override the default behaviour to allow an upgrade to be performed. > DocumentNodeStore does not have a repository software version range check. > -- > > Key: OAK-4529 > URL: https://issues.apache.org/jira/browse/OAK-4529 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.2.14, 1.0.31, 1.5.4, 1.4.4 >Reporter: Ian Boston > > DocumentNodeStore does not currently check which software version the > persisted repository it is connecting to was created with or last updated. > There is a risk that if the versions are incompatible the repository may be > damaged. > Somewhere in the repository, the version of the software that created it, and > the versions that written to it should be stored. In the case of TarMK this > information could be on local disk near the TarMK files. In the case of a > DocumentMK implementation, the information should be stored in the "database" > itself. > When a DocumentNodeStore instance connects it should: check the versions > stored in the repository then check the versions are within a compatible > range and refuse to start if not. > When a DocumentNodeStore writes to a repository, it should add its version to > the list of versions that have updated the repository. > This check behaviour should be active in full Oak or any utilities (eg > oak-run). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4529) DocumentNodeStore does not have a repository software version range check.
[ https://issues.apache.org/jira/browse/OAK-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361159#comment-15361159 ] Ian Boston commented on OAK-4529: - The situation that this issue is trying to protect against is something like this: 1. Oak production running at 1.0.x 2. Production JVM stopped 3. oak-run 1.4 mistakenly used to perform some maintenance. (should have been oak-run 1.0.x) 4. Production JVM wont start. bq. So, I think what's more important is for DNS instance to be initialized with the fact that the repository needs to be accessible by an older version of oak after any writes that this one does. I think that will cause the issue, not prevent it ? imho, Oak should refuse to write to an older repository, where those writes would break that repository for older versions of Oak listed as in use in the repository, unless some form of override is provided eg (eg -Doak.allowRepositoryUpgrade=true), or the versions are removed from the active list in the repository. If later versions of Oak cant read older repositories, they should refuse to start, hence the suggestion that DocumentNodeStore should use a range of version to determine what is safe. BTW. Perhaps its best not to use DNS when talking about DocumentNodeStore as DNS is widely accepted to mean https://en.wikipedia.org/wiki/Domain_Name_System and could cause confusion in without context. > DocumentNodeStore does not have a repository software version range check. > -- > > Key: OAK-4529 > URL: https://issues.apache.org/jira/browse/OAK-4529 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.2.14, 1.0.31, 1.5.4, 1.4.4 >Reporter: Ian Boston > > DocumentNodeStore does not currently check which software version the > persisted repository it is connecting to was created with or last updated. > There is a risk that if the versions are incompatible the repository may be > damaged. > Somewhere in the repository, the version of the software that created it, and > the versions that written to it should be stored. In the case of TarMK this > information could be on local disk near the TarMK files. In the case of a > DocumentMK implementation, the information should be stored in the "database" > itself. > When a DocumentNodeStore instance connects it should: check the versions > stored in the repository then check the versions are within a compatible > range and refuse to start if not. > When a DocumentNodeStore writes to a repository, it should add its version to > the list of versions that have updated the repository. > This check behaviour should be active in full Oak or any utilities (eg > oak-run). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4529) DocumentNodeStore does not have a repository software version range check.
Ian Boston created OAK-4529: --- Summary: DocumentNodeStore does not have a repository software version range check. Key: OAK-4529 URL: https://issues.apache.org/jira/browse/OAK-4529 Project: Jackrabbit Oak Issue Type: Bug Components: core Affects Versions: 1.4.4, 1.5.4, 1.0.31, 1.2.14 Reporter: Ian Boston DocumentNodeStore does not currently check which software version the persisted repository it is connecting to was created with or last updated. There is a risk that if the versions are incompatible the repository may be damaged. Somewhere in the repository, the version of the software that created it, and the versions that written to it should be stored. In the case of TarMK this information could be on local disk near the TarMK files. In the case of a DocumentMK implementation, the information should be stored in the "database" itself. When a DocumentNodeStore instance connects it should: check the versions stored in the repository then check the versions are within a compatible range and refuse to start if not. When a DocumentNodeStore writes to a repository, it should add its version to the list of versions that have updated the repository. This check behaviour should be active in full Oak or any utilities (eg oak-run). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4412) Lucene hybrid index
[
https://issues.apache.org/jira/browse/OAK-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350844#comment-15350844
]
Ian Boston commented on OAK-4412:
-
bq. Note that in Oak the target usecase is not a generic search server.
Agreed. All the techniques described are appropriate to any application using
Lucene that requires low data latency and scalability, not just a generic
search server. ElasticSearch, for instance, is frequently for metrics
aggregation and much less frequently used as a generic search server.
bq. By design code running on specific cluster node can only see changes as per
its head latest revision. So there is no notion of "cluster consistent index"
for very recent changes.
Does that imply that, regardless of the indexing mechanism used, indexes will
always have a data latency of a best the rate of commit of the root node in the
repository (IIUC each instances currently syncs its copy of the root node once
a second, since the root node is a cluster singleton), and indexing will always
have to be tied to an Oak revision ?
If there is a hard requirement that indexes are tied to the root node revision,
then there seems to be no point in following the work done by others in
lowering the data latency, increasing throughput and scalability. (ie ignore my
previous observations and sorry for the distraction).
> Lucene hybrid index
> ---
>
> Key: OAK-4412
> URL: https://issues.apache.org/jira/browse/OAK-4412
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.6
>
> Attachments: OAK-4412.patch
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic.
> The asynchronous index would be an answer here (as the index update won't be
> made in the client request thread), but the AEM requires the updates to be
> visible immediately in order to work properly.
> The idea here is to enhance the existing asynchronous Lucene index with a
> synchronous, locally-stored counterpart that will persist only the data since
> the last Lucene background reindexing job.
> The new index can be stored in memory or (if necessary) in MMAPed local
> files. Once the "main" Lucene index is being updated, the local index will be
> purged.
> Queries will use an union of results from the {{lucene}} and
> {{lucene-memory}} indexes.
> The {{lucene-memory}} index, as a local stored entity, will be updated using
> an observer, so it'll get both local and remote changes.
> The original idea has been suggested by [~chetanm] in the discussion for the
> OAK-4233.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4412) Lucene hybrid index
[
https://issues.apache.org/jira/browse/OAK-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350785#comment-15350785
]
Ian Boston commented on OAK-4412:
-
Thank you for the explanation of the detail. It confirms what I understood from
the description, when I posted the information.
The approach here is very similar to what others have tried in the past. The
problems they documented were:
1. No increase in index update throughput or reduction in data latency for a
cluster consistent index. Still single threaded cluster sequential.
2. Reduction in data latency for changes isolated to one node, mitigated by
sticky http sessions, which are/were considered an indication of a non scalable
platform.
3. Application issues for searches performed by request threads expecting
uniform data latency across the cluster.
4. Application issues for searches performed by non request threads seeing
local index changes inconsistently across the cluster.
5. Added complexity resulting in lower reliability.
For those that tried, not delivering any improvement in data latency for a
cluster consistent index was the biggest reason they abandoned the approach.
Some reported 3,4,5 to be blockers requiring applications to do special things
(eg querying the DB layer for results unexpectedly missing).
Unfortunately, the detailed reports referenced above were from 5 years ago, so
I am unlikely to be able to find them. Those reports mirror the experiences
shared by Shay Banon in some of his early tech talks. eg Berlin Buzz 2011
https://vimeo.com/26710663 where he described in detail many different methods
to implementing distributed Lucene. Current oak is described at 18:34. I think
Solr Cloud is described at 22:49, but I could be wrong about its commit
mechanism (IIRC from Lucene4 it ships segments but might not require hard
commits). This enhancement is a form of distributed directory described at
06:37. I share the link because Shay Banon has successfully built a technology
business over perhaps 15 years based on distributed real time lucene search,
not because I am suggesting you should directly use ElasticSearch.
HTH
> Lucene hybrid index
> ---
>
> Key: OAK-4412
> URL: https://issues.apache.org/jira/browse/OAK-4412
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.6
>
> Attachments: OAK-4412.patch
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic.
> The asynchronous index would be an answer here (as the index update won't be
> made in the client request thread), but the AEM requires the updates to be
> visible immediately in order to work properly.
> The idea here is to enhance the existing asynchronous Lucene index with a
> synchronous, locally-stored counterpart that will persist only the data since
> the last Lucene background reindexing job.
> The new index can be stored in memory or (if necessary) in MMAPed local
> files. Once the "main" Lucene index is being updated, the local index will be
> purged.
> Queries will use an union of results from the {{lucene}} and
> {{lucene-memory}} indexes.
> The {{lucene-memory}} index, as a local stored entity, will be updated using
> an observer, so it'll get both local and remote changes.
> The original idea has been suggested by [~chetanm] in the discussion for the
> OAK-4233.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4412) Lucene hybrid index
[
https://issues.apache.org/jira/browse/OAK-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344644#comment-15344644
]
Ian Boston commented on OAK-4412:
-
Does this work in a cluster ?
To clarify that question a little.
If the http requests hitting the cluster don't have sticky state will all
requests generate the same set of results for data just changed, or will that
depend on which instances the requests hit ?
Ie the servers in the cluster now have transient conversational state while the
changes in the in-memory index are hard committed to the existing async lucene
index and propagate to all instances in the cluster.
> Lucene hybrid index
> ---
>
> Key: OAK-4412
> URL: https://issues.apache.org/jira/browse/OAK-4412
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.6
>
> Attachments: OAK-4412.patch
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic.
> The asynchronous index would be an answer here (as the index update won't be
> made in the client request thread), but the AEM requires the updates to be
> visible immediately in order to work properly.
> The idea here is to enhance the existing asynchronous Lucene index with a
> synchronous, locally-stored counterpart that will persist only the data since
> the last Lucene background reindexing job.
> The new index can be stored in memory or (if necessary) in MMAPed local
> files. Once the "main" Lucene index is being updated, the local index will be
> purged.
> Queries will use an union of results from the {{lucene}} and
> {{lucene-memory}} indexes.
> The {{lucene-memory}} index, as a local stored entity, will be updated using
> an observer, so it'll get both local and remote changes.
> The original idea has been suggested by [~chetanm] in the discussion for the
> OAK-4233.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4471) More compact storage format for Documents
[ https://issues.apache.org/jira/browse/OAK-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344143#comment-15344143 ] Ian Boston commented on OAK-4471: - bq. However we would still benefit with size of stuff that goes over the wire. Wire protocol compression may be available for free in the drivers currently used by Oak. https://jira.mongodb.org/browse/SERVER-3018?focusedCommentId=897375=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-897375 compression can be enabled if OpenSSL is compiled with compression enabled, in which case zlib cann be used by the driver (not checked if the Java Driver supports this, but the server does and mongos does). The ticket is open as some dont want to use SSL based compression even with a null cypher. Most common JDBC drivers already support wire protocol compression. bq. One of the driver for me was Dynamo limit of 4k as 1 read unit Does Oak deploy on Dynamo ? AFAIK it has RDBMK and MongoMK support, but no Dynamo or Cassandra support. bq. OAK-1312 - Bundle multiple nodes in single Document agreed. bq. Hybrid lucene index etc Implementing a real time index in a cluster using a shadow local index has been tried many times by others and abandoned due to production experience with reliability and stability. I guess Oak might succeed where many others have failed. Most abandoned the model based on shipping segments in favor of sharded indexes, with replication at the index update level coupled with a write ahead log to cover resilience and real time high volume throughput. Eliminating the mix of semi transactional in database property indexes, non real time lucene indexes and offloaded Solr indexes with a performant, resilient, real time index capability for all properties would fix some aspects of the data record explosion that exists in the DocumentMK variants. OAK-1312 should address most of the other aspects. > More compact storage format for Documents > - > > Key: OAK-4471 > URL: https://issues.apache.org/jira/browse/OAK-4471 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: documentmk >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > Attachments: node-doc-size2.png > > > Aim of this task is to evaluate storage cost of current approach for various > Documents in DocumentNodeStore. And then evaluate possible alternative to see > if we can get a significant reduction in storage size. > Possible areas of improvement > # NodeDocument > ## Use binary encoding for property values - Currently property values are > stored in JSON encoding i.e. arrays and single values are encoded in json > along with there type > ## Use binary encoding for Revision values - In a given document Revision > instances are a major part of storage size. A binary encoding might provide > more compact storage > # Journal - The journal entries can be stored in compressed form > Any new approach should support working with existing setups i.e. provide > gradual change in storage format. > *Possible Benefits* > More compact storage would help in following ways > # Low memory footprint of Document in Mongo and RDB > # Low memory footprint for in memory NodeDocument instances - For e.g. > property values when stored in binary format would consume less memory > # Reduction in IO over wire - That should reduce the latency in say > distributed deployments where Oak has to talk to remote primary > Note that before doing any such change we must analyze the gains. Any change > in encoding would make interpreting stored data harder and also represents > significant change in stored data where we need to be careful to not > introduce any bug! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2787) Faster multi threaded indexing for binary content
[ https://issues.apache.org/jira/browse/OAK-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341657#comment-15341657 ] Ian Boston commented on OAK-2787: - Using a hidden text property on the Oak Document removes most of the benefit of the approach as originally proposed. The original proposal intended that any transcoding or tokenisation performed on a binary object, was performed once only, regardless of how many times that binary object was referenced throughout the repository. > Faster multi threaded indexing for binary content > - > > Key: OAK-2787 > URL: https://issues.apache.org/jira/browse/OAK-2787 > Project: Jackrabbit Oak > Issue Type: Wish > Components: lucene >Reporter: Chetan Mehrotra > > With Lucene based indexing the indexing process is single threaded. This > hamper the indexing of binary content as on a multi processor system only > single thread can be used to perform the indexing > [~ianeboston] Suggested a possible approach [1] involving a 2 phase indexing > # In first phase detect the nodes to be indexed and start the full text > extraction of the binary content. Post extraction save the binary token > stream back to the node as a hidden data. In this phase the node properties > can still be indexed and a marker field would be added to indicate the > fulltext index is still pending > # Later in 2nd phase look for all such Lucene docs and then update them with > the saved token stream > This would allow the text extraction logic to be decouple from Lucene > indexing logic > [1] http://markmail.org/thread/2w5o4bwqsosb6esu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4471) More compact storage format for Documents
[ https://issues.apache.org/jira/browse/OAK-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341626#comment-15341626 ] Ian Boston commented on OAK-4471: - Compressing will obfuscate readability which will make it harder for deployers to fix issues directly. MongoDB already stores json as BSON. MongoDB 3.2 recommends using the WiredTiger storage engine which has disk and memory compression. Any benefits in compressing individual documents might be minimal. Minimising ratios between Application Domain Object -> JCR Node -> Oak Document will have the largest impact. This current stands at 2 orders of magnitude for an average application. (105x to be precise). > More compact storage format for Documents > - > > Key: OAK-4471 > URL: https://issues.apache.org/jira/browse/OAK-4471 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: documentmk >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > Attachments: node-doc-size2.png > > > Aim of this task is to evaluate storage cost of current approach for various > Documents in DocumentNodeStore. And then evaluate possible alternative to see > if we can get a significant reduction in storage size. > Possible areas of improvement > # NodeDocument > ## Use binary encoding for property values - Currently property values are > stored in JSON encoding i.e. arrays and single values are encoded in json > along with there type > ## Use binary encoding for Revision values - In a given document Revision > instances are a major part of storage size. A binary encoding might provide > more compact storage > # Journal - The journal entries can be stored in compressed form > Any new approach should support working with existing setups i.e. provide > gradual change in storage format. > *Possible Benefits* > More compact storage would help in following ways > # Low memory footprint of Document in Mongo and RDB > # Low memory footprint for in memory NodeDocument instances - For e.g. > property values when stored in binary format would consume less memory > # Reduction in IO over wire - That should reduce the latency in say > distributed deployments where Oak has to talk to remote primary > Note that before doing any such change we must analyze the gains. Any change > in encoding would make interpreting stored data harder and also represents > significant change in stored data where we need to be careful to not > introduce any bug! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4440) Async upload in S3DataStore can lead to intermittent binary missing exception in cluster
[
https://issues.apache.org/jira/browse/OAK-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329203#comment-15329203
]
Ian Boston commented on OAK-4440:
-
OAK-3547 would allow other cluster nodes to fall back to earlier versions of
the index.
> Async upload in S3DataStore can lead to intermittent binary missing exception
> in cluster
>
>
> Key: OAK-4440
> URL: https://issues.apache.org/jira/browse/OAK-4440
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob
>Reporter: Chetan Mehrotra
> Fix For: 1.6
>
>
> {{S3DataStore}} supports async upload of binaries. This feature was added to
> speedup performance for cases where S3 is remote (i.e. Oak/JR2 app was
> running on premise) (JCR-3733)
> This feature would cause issue in cluster deployment where binaries get
> uploaded from any cluster node because its possible that binary is not
> uploaded to S3 (present in local cache of one of the cluster node) and some
> code in other cluster node tries to read the binary.
> This feature also poses problem in backup and restore where we would need
> backup the local S3 cache on all the cluster nodes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3512) Synchronous index rebuilds can be performed by all commits once the reindex propery is true
[ https://issues.apache.org/jira/browse/OAK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248487#comment-15248487 ] Ian Boston commented on OAK-3512: - [~joerghoh] IIRC there was a fix by the Oak team before the issue was re-opened, but after the fix was applied there were some later instances where reindex=true appeared committed to the DB. IIRC the system in question was running RBMK, but the root cause of why reindex=true appeared in the database was not found, or at least, I didn't hear what it was. I don't think this has been seen again since. There may be other fixed already in the latest version of Oak. If there are no more occurrences of the issue, or the root cause is known and fixed elsewhere it can probably be closed. > Synchronous index rebuilds can be performed by all commits once the reindex > propery is true > --- > > Key: OAK-3512 > URL: https://issues.apache.org/jira/browse/OAK-3512 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.4 >Reporter: Ian Boston > > If reindex=true gets persisted to the database, then every commit triggers a > rebuild of the index. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2920) RDBDocumentStore: fail init when database config seems to be inadequate
[ https://issues.apache.org/jira/browse/OAK-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168925#comment-15168925 ] Ian Boston commented on OAK-2920: - If the DB config is broken, then content that expects UTF8 in the Path will fail to import as the IDs will be rejected as duplicates. For instance any application that stores i18n content in the repository and needs to work with any language that has double byte characters (eg German) will fail. ID duplicates are easy to detect. Much harder to detect is data corruption within JCR properties as a user using Oak via a WebUI could suspect any of the links between the Browser and the DB as the source of UTF8 corruption. Taking mySQL as an example. Without utf8, Characters in common use in EU countries cant be stored as JCR properties. http://www.periodni.com/unicode_utf-8_encoding.html. Without utf8mb4, supplementary UTF8 characters can't be stored as JCR properties. http://www.i18nguy.com/unicode/supplementary-test.html For those reasons, any database or JDBC connection that is misconfigured is likely to cause considerable problems in production and probably won't work with most modern applications that have been internationalised or need to mention the Euro. € One approach to detect this is to write a row to the nodes table containing supplementary UTF8 characters, commit the row, and then read the same row back, verifying that the data survived the round trip. Finally delete the row. The ID of the row can be something that Oak would never use with a low probability of collision with other Oak instances in the same cluster. (ie ms timestamp eg 21313412313:utf8test). If there is a concern about tables other than the nodes table, then those can be tested as well. A switch should be provided to allow those who have managed to run Oak in production with a misconfigured database to at least keep running in production while they correct the issue. For mySQL this might be as simple as correcting the JDBC url to include utf8mb4 encoding. > RDBDocumentStore: fail init when database config seems to be inadequate > --- > > Key: OAK-2920 > URL: https://issues.apache.org/jira/browse/OAK-2920 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: rdbmk >Reporter: Julian Reschke >Priority: Minor > Labels: resilience > > It has been suggested that the implementation should fail to start (rather > than warn) when it detects a DB configuration that is likely to cause > problems (such as wrt character encoding or collation sequences) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3654) Integrate with Metrics for various stats collection
[ https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025052#comment-15025052 ] Ian Boston commented on OAK-3654: - LvGTM. You might want to share the pointer with the Metrics project as it does have a significant impact on the overhead of a Meter, bringing it closer to the original AtomicLong overhead. > Integrate with Metrics for various stats collection > > > Key: OAK-3654 > URL: https://issues.apache.org/jira/browse/OAK-3654 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.3.12 > > Attachments: OAK-3654-fast-clock.patch, OAK-3654-v1.patch, > query-stats.png > > > As suggested by [~ianeboston] in OAK-3478 current approach of collecting > TimeSeries data is not easily consumable by other monitoring systems. Also > just extracting the last moment data and exposing it in simple form would not > be useful. > Instead of that we should look into using Metrics library [1] for collecting > metrics. To avoid having dependency on Metrics API all over in Oak we can > come up with minimal interfaces which can be used in Oak and then provide an > implementation backed by Metric. > This task is meant to explore that aspect and come up with proposed changes > to see if its feasible to make this change > * metrics-core ~100KB in size with no dependency > * ASL Licensee > [1] http://metrics.dropwizard.io -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3654) Integrate with Metrics for various stats collection
[
https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024219#comment-15024219
]
Ian Boston commented on OAK-3654:
-
The metrics stats show 2 things.
{code}
min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0
duration_unit=microseconds
{code}
The section being timed is too small for System.nanoTime() to detect a
duration. (conclusive)
{code}
mean_rate=2268390.1680024294
{code}
The block gets called at least every 4.4e-7s assuming no other operations.
(indicative, as there are other operations)
Both of which are a strong indication that the block is too quick to be timed
with a timer of the resolution available in the JDK, even if the timing
mechanism had no overhead.
Adding some other sort of instrumentation (meter, counter) might provide
something of value when monitoring the rates longitudinally, but based on the
above it's probable the block should not be instrumented at all.
> Integrate with Metrics for various stats collection
>
>
> Key: OAK-3654
> URL: https://issues.apache.org/jira/browse/OAK-3654
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.3.12
>
> Attachments: OAK-3654-v1.patch, query-stats.png
>
>
> As suggested by [~ianeboston] in OAK-3478 current approach of collecting
> TimeSeries data is not easily consumable by other monitoring systems. Also
> just extracting the last moment data and exposing it in simple form would not
> be useful.
> Instead of that we should look into using Metrics library [1] for collecting
> metrics. To avoid having dependency on Metrics API all over in Oak we can
> come up with minimal interfaces which can be used in Oak and then provide an
> implementation backed by Metric.
> This task is meant to explore that aspect and come up with proposed changes
> to see if its feasible to make this change
> * metrics-core ~100KB in size with no dependency
> * ASL Licensee
> [1] http://metrics.dropwizard.io
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3654) Integrate with Metrics for various stats collection
[
https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024367#comment-15024367
]
Ian Boston edited comment on OAK-3654 at 11/24/15 12:24 PM:
Metrics provides "bad stats" because its being given "bad data", have a look at
the patch to the methods, copied here.
{code}
@@ -622,14 +623,14 @@ public class SessionDelegate {
if (op.isUpdate()) {
sessionCounters.writeTime = t0;
sessionCounters.writeCount++;
-writeCounter.incrementAndGet();
-writeDuration.addAndGet(dt);
+writeCounter.mark();
+writeDuration.update(dt, TimeUnit.MILLISECONDS);
updateCount++;
} else {
sessionCounters.readTime = t0;
sessionCounters.readCount++;
-readCounter.incrementAndGet();
-readDuration.addAndGet(dt);
+readCounter.mark();
+readDuration.update(dt, TimeUnit.MILLISECONDS);
}
{code}
It should be using
{code}
Context c = metric.time();
c.stop();
{code}
IIRC if done that way Metrics uses nanoseconds.
Internally Metics will probably use nanoseconds even it it reports in
microseconds.
As mentioned earlier, using the context.stop() pattern avoids inadvertent bugs.
BTW, according to the output, it was given 136M (136121718) samples, which
should be enough samples to generate something statistically relevant.
was (Author: ianeboston):
Metrics provides "bad stats" because its being given "bad data", have a look at
the patch to the methods, copied here.
{code}
@@ -622,14 +623,14 @@ public class SessionDelegate {
if (op.isUpdate()) {
sessionCounters.writeTime = t0;
sessionCounters.writeCount++;
-writeCounter.incrementAndGet();
-writeDuration.addAndGet(dt);
+writeCounter.mark();
+writeDuration.update(dt, TimeUnit.MILLISECONDS);
updateCount++;
} else {
sessionCounters.readTime = t0;
sessionCounters.readCount++;
-readCounter.incrementAndGet();
-readDuration.addAndGet(dt);
+readCounter.mark();
+readDuration.update(dt, TimeUnit.MILLISECONDS);
}
{code}
It should be using
{code}
Context c = metric.time();
c.stop();
{code}
IIRC if done that way Metrics uses nanoseconds.
Internally Metics will probably use nanoseconds even it it reports in
microseconds.
As mentioned earlier, using the context.stop() pattern avoids inadvertent bugs.
> Integrate with Metrics for various stats collection
>
>
> Key: OAK-3654
> URL: https://issues.apache.org/jira/browse/OAK-3654
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.3.12
>
> Attachments: OAK-3654-v1.patch, query-stats.png
>
>
> As suggested by [~ianeboston] in OAK-3478 current approach of collecting
> TimeSeries data is not easily consumable by other monitoring systems. Also
> just extracting the last moment data and exposing it in simple form would not
> be useful.
> Instead of that we should look into using Metrics library [1] for collecting
> metrics. To avoid having dependency on Metrics API all over in Oak we can
> come up with minimal interfaces which can be used in Oak and then provide an
> implementation backed by Metric.
> This task is meant to explore that aspect and come up with proposed changes
> to see if its feasible to make this change
> * metrics-core ~100KB in size with no dependency
> * ASL Licensee
> [1] http://metrics.dropwizard.io
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3654) Integrate with Metrics for various stats collection
[
https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024367#comment-15024367
]
Ian Boston edited comment on OAK-3654 at 11/24/15 12:30 PM:
Metrics provides "bad stats" because its being given "bad data", have a look at
the patch to the methods, copied here.
(I am assuming this is the code being tested and reported in the tests)
{code}
@@ -622,14 +623,14 @@ public class SessionDelegate {
if (op.isUpdate()) {
sessionCounters.writeTime = t0;
sessionCounters.writeCount++;
-writeCounter.incrementAndGet();
-writeDuration.addAndGet(dt);
+writeCounter.mark();
+writeDuration.update(dt, TimeUnit.MILLISECONDS);
updateCount++;
} else {
sessionCounters.readTime = t0;
sessionCounters.readCount++;
-readCounter.incrementAndGet();
-readDuration.addAndGet(dt);
+readCounter.mark();
+readDuration.update(dt, TimeUnit.MILLISECONDS);
}
{code}
It should be using
{code}
Context c = metric.time();
c.stop();
{code}
IIRC if done that way Metrics uses nanoseconds.
Internally Metics will probably use nanoseconds even it it reports in
microseconds.
As mentioned earlier, using the context.stop() pattern avoids inadvertent bugs.
BTW, according to the output, it was given 136M (136121718) samples, which
should be enough samples to generate something statistically relevant.
was (Author: ianeboston):
Metrics provides "bad stats" because its being given "bad data", have a look at
the patch to the methods, copied here.
{code}
@@ -622,14 +623,14 @@ public class SessionDelegate {
if (op.isUpdate()) {
sessionCounters.writeTime = t0;
sessionCounters.writeCount++;
-writeCounter.incrementAndGet();
-writeDuration.addAndGet(dt);
+writeCounter.mark();
+writeDuration.update(dt, TimeUnit.MILLISECONDS);
updateCount++;
} else {
sessionCounters.readTime = t0;
sessionCounters.readCount++;
-readCounter.incrementAndGet();
-readDuration.addAndGet(dt);
+readCounter.mark();
+readDuration.update(dt, TimeUnit.MILLISECONDS);
}
{code}
It should be using
{code}
Context c = metric.time();
c.stop();
{code}
IIRC if done that way Metrics uses nanoseconds.
Internally Metics will probably use nanoseconds even it it reports in
microseconds.
As mentioned earlier, using the context.stop() pattern avoids inadvertent bugs.
BTW, according to the output, it was given 136M (136121718) samples, which
should be enough samples to generate something statistically relevant.
> Integrate with Metrics for various stats collection
>
>
> Key: OAK-3654
> URL: https://issues.apache.org/jira/browse/OAK-3654
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.3.12
>
> Attachments: OAK-3654-v1.patch, query-stats.png
>
>
> As suggested by [~ianeboston] in OAK-3478 current approach of collecting
> TimeSeries data is not easily consumable by other monitoring systems. Also
> just extracting the last moment data and exposing it in simple form would not
> be useful.
> Instead of that we should look into using Metrics library [1] for collecting
> metrics. To avoid having dependency on Metrics API all over in Oak we can
> come up with minimal interfaces which can be used in Oak and then provide an
> implementation backed by Metric.
> This task is meant to explore that aspect and come up with proposed changes
> to see if its feasible to make this change
> * metrics-core ~100KB in size with no dependency
> * ASL Licensee
> [1] http://metrics.dropwizard.io
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3654) Integrate with Metrics for various stats collection
[
https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024367#comment-15024367
]
Ian Boston commented on OAK-3654:
-
Metrics provides "bad stats" because its being given "bad data", have a look at
the patch to the methods, copied here.
{code}
@@ -622,14 +623,14 @@ public class SessionDelegate {
if (op.isUpdate()) {
sessionCounters.writeTime = t0;
sessionCounters.writeCount++;
-writeCounter.incrementAndGet();
-writeDuration.addAndGet(dt);
+writeCounter.mark();
+writeDuration.update(dt, TimeUnit.MILLISECONDS);
updateCount++;
} else {
sessionCounters.readTime = t0;
sessionCounters.readCount++;
-readCounter.incrementAndGet();
-readDuration.addAndGet(dt);
+readCounter.mark();
+readDuration.update(dt, TimeUnit.MILLISECONDS);
}
{code}
It should be using
{code}
Context c = metric.time();
c.stop();
{code}
IIRC if done that way Metrics uses nanoseconds.
Internally Metics will probably use nanoseconds even it it reports in
microseconds.
As mentioned earlier, using the context.stop() pattern avoids inadvertent bugs.
> Integrate with Metrics for various stats collection
>
>
> Key: OAK-3654
> URL: https://issues.apache.org/jira/browse/OAK-3654
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.3.12
>
> Attachments: OAK-3654-v1.patch, query-stats.png
>
>
> As suggested by [~ianeboston] in OAK-3478 current approach of collecting
> TimeSeries data is not easily consumable by other monitoring systems. Also
> just extracting the last moment data and exposing it in simple form would not
> be useful.
> Instead of that we should look into using Metrics library [1] for collecting
> metrics. To avoid having dependency on Metrics API all over in Oak we can
> come up with minimal interfaces which can be used in Oak and then provide an
> implementation backed by Metric.
> This task is meant to explore that aspect and come up with proposed changes
> to see if its feasible to make this change
> * metrics-core ~100KB in size with no dependency
> * ASL Licensee
> [1] http://metrics.dropwizard.io
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3654) Integrate with Metrics for various stats collection
[ https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022031#comment-15022031 ] Ian Boston commented on OAK-3654: - Clock.Fast has a maximum resolution of 1ms and when getTime is called it makes a call to System.nanoTime() and a call to System.getCurrentTimeMillis(), hence when ACCURATE.getTime() is called it will generally be slower than just calling System.nanoTime(). The result from Clock.Fast.getTime() is of little use in the Metrics use case. Reading the background I think the requirements from Oak for Clock.Fast are different from the requirements from a Metrics use case. A metrics use case must have nano second time resolution and should not require more than 1 call to System.nanoTime() to achieve that. IIUC Oak needed ms time resolution. I don't believe there is any requirement to deal with long term drift in time (ie 1% or less) over a period of > 10s. If the OS/JDK really is very slow for System.nanoTime() calls then it's probably a JDK bug. I see the testes were done on JDK 7u45 which is very old compared to the current release. If there is a issue with non linearity in time or steps in time then its likely the OS ntpd is misconfigured. If you read https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks it explains why System.getCurrentTimeMillis() is faster than System.nanoTime() (measured at 13ns vs 18ns). It also explains why the 2 drift, and strongly recommends using System.nanoTime() to measure intervals, as that call generally binds to a crystal oscillator with sub 1% drift. The blog post also warns about mixing the 2 sources of time. There are some notes specific to the Windows impl where System.getCurrentTimeMillis() has a 10ms resolution for certain versions for the JDK. So in answer to your question: Metrics has considerable production experience. I would avoid making fundamental changes unilaterally. If there is a real problem with the Metrics code and how they measure time, I would open an issue on their bug tracker to validate the issue is real. > Integrate with Metrics for various stats collection > > > Key: OAK-3654 > URL: https://issues.apache.org/jira/browse/OAK-3654 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.4 > > Attachments: OAK-3654-v1.patch, query-stats.png > > > As suggested by [~ianeboston] in OAK-3478 current approach of collecting > TimeSeries data is not easily consumable by other monitoring systems. Also > just extracting the last moment data and exposing it in simple form would not > be useful. > Instead of that we should look into using Metrics library [1] for collecting > metrics. To avoid having dependency on Metrics API all over in Oak we can > come up with minimal interfaces which can be used in Oak and then provide an > implementation backed by Metric. > This task is meant to explore that aspect and come up with proposed changes > to see if its feasible to make this change > * metrics-core ~100KB in size with no dependency > * ASL Licensee > [1] http://metrics.dropwizard.io -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3654) Integrate with Metrics for various stats collection
[ https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023912#comment-15023912 ] Ian Boston commented on OAK-3654: - My tests showed that timing measurements of operations that took less than 1E-7s contained a 10% overhead which concurs with the observation that System.nanoTime() takes order 1E-8s to complete. If you need to measure metrics at that level, then using a Timer wrapping an individual call is the wrong approach as you will need a timer with an overhead of more like 1E-9 or 1E-10s and an accuracy of 1E-8 or 1E-9s. Realistically, continuous and routine timing at that level is not particularly useful in production and best left to attaching a profiler with sampling support. > Integrate with Metrics for various stats collection > > > Key: OAK-3654 > URL: https://issues.apache.org/jira/browse/OAK-3654 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.4 > > Attachments: OAK-3654-v1.patch, query-stats.png > > > As suggested by [~ianeboston] in OAK-3478 current approach of collecting > TimeSeries data is not easily consumable by other monitoring systems. Also > just extracting the last moment data and exposing it in simple form would not > be useful. > Instead of that we should look into using Metrics library [1] for collecting > metrics. To avoid having dependency on Metrics API all over in Oak we can > come up with minimal interfaces which can be used in Oak and then provide an > implementation backed by Metric. > This task is meant to explore that aspect and come up with proposed changes > to see if its feasible to make this change > * metrics-core ~100KB in size with no dependency > * ASL Licensee > [1] http://metrics.dropwizard.io -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3654) Integrate with Metrics for various stats collection
[
https://issues.apache.org/jira/browse/OAK-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015528#comment-15015528
]
Ian Boston commented on OAK-3654:
-
Patch LGTM, some comments.
The Timer.update(...) isn't very clean compared to the other patterns that
leave performing the timing operation to the implementation.
eg (Metrics notation).
{code}
Context c = timerMetric.time();
Do work
c.stop();
{code}
is cleaner than, and potentially less buggy than:
{code}
long nanos = System.nanoTime();
Do work
timerMetric.update(System.nanoTime()-nanos, TimeUnit.NANOS);
{code}
Perhaps consider exposing a wrapper for Context in the TimerStats interface. It
would reduce the volume of code require to instrument a section/method and
trigger HotSpot earlier on that code. It is marginal, but could save some bugs.
(incorrect sum, wrong time units, wrong start time).
nanoTimer and performance. I have measured the overhead of the metrics library
to be about 10% timing calls of about 1E-7s. (ie 1E-8s). Timing to operations
that take less than 1E-7s tends to be counter productive. I haven't tested
counting. Testing was done using JDK7, OSX 10.10 on a 2.53 GHz Intel Core 2
Duo, late 2008 model.
-
If we had detailed reliable metrics on more things (eg queues, rates of
indexing, query rates, etc) not just database calls we would be able to get a
snapshot of the entire stack operation in a single json file. Seeing which
queries are slow does help, but seeing a 99 percentile response time is 5
orders of magnitude greater than the median is a sure indicator of trouble that
needs investigation, wherever it is in the stack. Metrics are not that useful
if they only cover a very small area of the stack.
MongoDB already logs slow queries its log files, with good OS tools to analyse.
Having that information has helped in the past, but often issues doen show up
in that way (Oak commit queue overflow or stall).
> Integrate with Metrics for various stats collection
>
>
> Key: OAK-3654
> URL: https://issues.apache.org/jira/browse/OAK-3654
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
> Attachments: OAK-3654-v1.patch, query-stats.png
>
>
> As suggested by [~ianeboston] in OAK-3478 current approach of collecting
> TimeSeries data is not easily consumable by other monitoring systems. Also
> just extracting the last moment data and exposing it in simple form would not
> be useful.
> Instead of that we should look into using Metrics library [1] for collecting
> metrics. To avoid having dependency on Metrics API all over in Oak we can
> come up with minimal interfaces which can be used in Oak and then provide an
> implementation backed by Metric.
> This task is meant to explore that aspect and come up with proposed changes
> to see if its feasible to make this change
> * metrics-core ~100KB in size with no dependency
> * ASL Licensee
> [1] http://metrics.dropwizard.io
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[ https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010908#comment-15010908 ] Ian Boston commented on OAK-3478: - @Chetan Mehrotra The bean "Oak Repository Statistics-simple" produces the right type of stats, but I think the the "per second" approach wont work as the sequence of numbers from the existing TimeSeries follows the pattern 0,0,0,0,0,0,0,76100,0,0,0,0,0,0,0 etc. If the monitoring tool happens to query either side of the 71600 value, then it get 0. To get 76100 it has to query at exactly the correct time. You could use the per minute value but it would be better not to re-invent the research in this area and look at what others have already proven to work in production. Most Metrics gathering systems use some form of moving average eg http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average or a pure counter to let the monitoring tool do the work. Pure counters are not very good as they tend to hit overflow problems, Moving averages of some form are better, provided the Windows or reservoirs can be implemented efficiently. Since averages are not always that useful when identifying performance most metrics tools also provide live histogram of the metric to produce 50%, 99.99%. 99.99% becomes important if the operation is performed thousands of times where each operation correlates with others, as a slow operation 0.001 % of the time will impact all operations with absolute certainty. Oak has plenty of places where this is a characteristic, starting with queues. To produce those sort of stats you may need to implement http://www.cs.umd.edu/~samir/498/vitter.pdf. The link was taken from Codehale Metrics which has various implementations testing in production. If Oak is not prepared to use a third party library for metrics support it should learn from what others have used successfully in production. > Provide JMX Beans for Oak that can be monitored by external tooling. > > > Key: OAK-3478 > URL: https://issues.apache.org/jira/browse/OAK-3478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, jcr, lucene, query >Affects Versions: 1.3.7 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.3.11 > > Attachments: OAK-3478-v1.patch > > > The Current JMX beans, while ok in the Sling Web Console are hard if not > impossible to monitor with external tooling, as external tooling will poll > for current values, ideally from named attributes containing primitive types. > Those values containing timers, counters or gauges. Timers timing an > operation. Counters counting an operation. Guages measuring an instantaneous > value. > The request is to provide a small number of JMX beans that can be configured > into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk > etc etc, which in turn will provide long term time series and statistics. > Primitive values of this form can also be graphed with ease in JConsole, > VisualVM etc. A improvement for the Sling Web Console might be to add a > Console that can maintain a TimeSeries graph of any JMX bean by object name > in the same way Ganglia, AppDynamics does, however that may be duplicating > existing functionality. > The Metrics Library could be considered to provide the above functionality > for all JMX beans and monitoring, although its footprint at 111K might be > considered too big as an additional dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[ https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010755#comment-15010755 ] Ian Boston commented on OAK-3478: - @Michael Dürig SLING-4849 might do what you need if you want to decouple monitoring. > Provide JMX Beans for Oak that can be monitored by external tooling. > > > Key: OAK-3478 > URL: https://issues.apache.org/jira/browse/OAK-3478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, jcr, lucene, query >Affects Versions: 1.3.7 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.3.11 > > Attachments: OAK-3478-v1.patch > > > The Current JMX beans, while ok in the Sling Web Console are hard if not > impossible to monitor with external tooling, as external tooling will poll > for current values, ideally from named attributes containing primitive types. > Those values containing timers, counters or gauges. Timers timing an > operation. Counters counting an operation. Guages measuring an instantaneous > value. > The request is to provide a small number of JMX beans that can be configured > into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk > etc etc, which in turn will provide long term time series and statistics. > Primitive values of this form can also be graphed with ease in JConsole, > VisualVM etc. A improvement for the Sling Web Console might be to add a > Console that can maintain a TimeSeries graph of any JMX bean by object name > in the same way Ganglia, AppDynamics does, however that may be duplicating > existing functionality. > The Metrics Library could be considered to provide the above functionality > for all JMX beans and monitoring, although its footprint at 111K might be > considered too big as an additional dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[
https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008260#comment-15008260
]
Ian Boston commented on OAK-3478:
-
[~chetanm]
Any bean that uses org.apache.jackrabbit.api.stats.TimeSeries is expressing
time series data in a way that can't be consumed.
Possible classes include
{code}
./oak-core/src/main/java/org/apache/jackrabbit/oak/stats/RepositoryStats.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/AsyncIndexUpdate.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStoreGCMonitor.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/stats/StatisticManager.java
./oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/IndexStatsMBean.java
{code}
Most beans that expose their data via javax.management.openmbean.CompositeData
Only those beans that expose data containing information that needs to be
monitored are an issue. Although most JMX tools are slow and hard to use when
confronted with deeply nested data inside Composite or Tabular data, a human
can deal with them.
Potential classes are below
{code}
/oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/IndexStatsMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/RepositoryManagementMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/RepositoryStatsMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/CheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/ConsolidatedCacheStatsMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/RepositoryManagementMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/cache/ConsolidatedCacheStats.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/management/ManagementOperation.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/management/RepositoryManager.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/backup/FileStoreBackupRestore.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/backup/FileStoreBackupRestoreMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/BlobGC.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/BlobGCMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/migration/BlobMigration.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/migration/BlobMigrationMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentCheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentCheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/AsyncIndexUpdate.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/property/jmx/PropertyIndexAsyncReindex.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/property/jmx/PropertyIndexAsyncReindexMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStoreGCMonitor.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/GCMonitorMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentCheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/spi/state/RevisionGC.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/spi/state/RevisionGCMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/stats/RepositoryStats.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/util/AbstractCheckpointMBean.java
./oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ConsolidatedListenerMBeanImpl.java
./oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/IndexCopier.java
./oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexMBeanImpl.java
{code}
> Provide JMX Beans for Oak that can be monitored by external tooling.
>
>
> Key: OAK-3478
> URL: https://issues.apache.org/jira/browse/OAK-3478
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, jcr, lucene, query
>Affects Versions: 1.3.7
>Reporter: Ian Boston
>Assignee: Chetan Mehrotra
>
> The Current JMX beans, while ok in the Sling Web Console are hard if not
> impossible to monitor with external tooling, as external tooling will poll
> for current values, ideally from named attributes containing primitive types.
> Those values containing timers, counters or gauges. Timers timing an
> operation. Counters counting an operation. Guages measuring an instantaneous
> value.
> The request is to provide a small number of JMX beans that can be configured
> into an external monitoring tool like
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993557#comment-14993557 ] Ian Boston commented on OAK-3547: - With the latest commit to the branch, segments.gen is now stored in segments.gen_ and is immutable making it possible to correctly open previous generations. I think the patch is complete, unless a more sophisticated recovery mechanism is required. (ie flagging the index as requiring a rebuild, without actually doing it). Obviously needs extensive testing to see what happens when real repo corruption happens in a live cluster. At present only tested with a single instance on MongoMK. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991349#comment-14991349 ] Ian Boston edited comment on OAK-3547 at 11/5/15 9:01 AM: -- [~mreutegg] If an earlier version of the index is used by the writer, there will be holes in the index and items will be missing. There are several options. a) flag the issue to alert admins the index is not healthy, but continue to index using an index that will open. b) Fail the index write and stop indexing completely. c) Fail the index write and start re-indexing automatically. Of those I think option a will deliver the best continuity. Option b risks wide scale application level issues, option c risks both application level issues and potential unavailability caused by the load or rebuilding an index from scratch. There is no easy answer. Now that there are checksums in place I have been seeing more frequent race conditions between the writer and the readers which occasionally open older versions. I think this is because the OakDirectory checks all the files when its opened by computing a checksum of everything referenced. I think that Lucene delays checking the file or checking the internals of a file until its needed, hence any errors are more visible than before. Lucene already has a concept of committing the index by syncing the segment_xx and segment.gen files. I am writing the listing node on sync of either of these or close of the index which has reduced the number of generations. The result appears to be very stable. I have also introduced the concept of mutability as some of the file types are mutable. .del is mutable, so the length and checksum are not checked. If a .del from a later generation is used, that will only delete the lucene docs that were deleted in that later generation. No damage. segments.gen is also mutable. This is more of a problem. It is supposed to be a fallback file with segment_xx used in preference, however if segment.gen is used it will be from the wrong generation and will define the wrong set of segment files for the index. I need to check if segment.gen is ever read. If it is, then I think the OakDirectory needs to map segment.gen to a generational version of the same (ie segment.gen_) so that only .del files are mutable. That should make the OakDirectory recoverable. was (Author: ianeboston): [~mreutegg] If an earlier version of the index is used by the writer, there will he holes in the index and items will be missing. There are several options. a) flag the issue to alert admins the index is not healthy, but continue to index using an index that will open. b) Fail the index write and stop indexing completely. c) Fail the index write and start re-indexing automatically. Of those I think option a will deliver the best continuity. Option b risks wide scale application level issues, option c risks both application level issues and potential unavailability caused by the load or rebuilding an index from scratch. There is no easy answer. Now that there are checksums in place I have been seeing more frequent race conditions between the writer and the readers which occasionally open older versions. I think this is because the OakDirectory checks all the files when its opened by computing a checksum of everything referenced. I think that Lucene delays checking the file or checking the internals of a file until its needed, hence any errors are more visible than before. Lucene already has a concept of committing the index by syncing the segment_xx and segment.gen files. I am writing the listing node on sync of either of these or close of the index which has reduced the number of generations. The result appears to be very stable. I have also introduced the concept of mutability as some of the file types are mutable. .del is mutable, so the length and checksum are not checked. If a .del from a later generation is used, that will only delete the lucene docs that were deleted in that later generation. No damage. segments.gen is also mutable. This is more of a problem. It is supposed to be a fallback file with segment_xx used in preference, however if segment.gen is used it will be from the wrong generation and will define the wrong set of segment files for the index. I need to check if segment.gen is ever read. If it is, then I think the OakDirectory needs to map segment.gen to a generational version of the same (ie segment.gen_) so that only .del files are mutable. That should make the OakDirectory recoverable. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type:
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991349#comment-14991349 ] Ian Boston commented on OAK-3547: - [~mreutegg] If an earlier version of the index is used by the writer, there will he holes in the index and items will be missing. There are several options. a) flag the issue to alert admins the index is not healthy, but continue to index using an index that will open. b) Fail the index write and stop indexing completely. c) Fail the index write and start re-indexing automatically. Of those I think option a will deliver the best continuity. Option b risks wide scale application level issues, option c risks both application level issues and potential unavailability caused by the load or rebuilding an index from scratch. There is no easy answer. Now that there are checksums in place I have been seeing more frequent race conditions between the writer and the readers which occasionally open older versions. I think this is because the OakDirectory checks all the files when its opened by computing a checksum of everything referenced. I think that Lucene delays checking the file or checking the internals of a file until its needed, hence any errors are more visible than before. Lucene already has a concept of committing the index by syncing the segment_xx and segment.gen files. I am writing the listing node on sync of either of these or close of the index which has reduced the number of generations. The result appears to be very stable. I have also introduced the concept of mutability as some of the file types are mutable. .del is mutable, so the length and checksum are not checked. If a .del from a later generation is used, that will only delete the lucene docs that were deleted in that later generation. No damage. segments.gen is also mutable. This is more of a problem. It is supposed to be a fallback file with segment_xx used in preference, however if segment.gen is used it will be from the wrong generation and will define the wrong set of segment files for the index. I need to check if segment.gen is ever read. If it is, then I think the OakDirectory needs to map segment.gen to a generational version of the same (ie segment.gen_) so that only .del files are mutable. That should make the OakDirectory recoverable. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991734#comment-14991734 ] Ian Boston commented on OAK-3547: - segment.gen is opened by Lucene as well as segment_xx which means that its mutable and used. To allow previous generations to be used, the name will need to be transformed inside the listing, if that is possible. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989600#comment-14989600 ] Ian Boston commented on OAK-3547: - [~mreutegg] Currently every call to OakDirectory.sync(...) and OakDirectory.close(...) where the OakDirectory is not a read only oak directory, causes a list of files with size and sha1 hash to be written to a new node with a name of the form /oak:index//:state/l_. When the current Oak session commits, that is committed to the Oak repo. When the OakDirectory is loaded, it tries upto 100 l_ nodes in order, newest first, checking that the contents are present and have matching length+sha1. The first valid listing found is loaded. If no valid matches are found then the code reverts to earlier behaviour, using all the non deleted files in the /oak:index//:dir folder. If the bundle is deployed to an existing repository it will fall back to the old behaviour. I have assumed a call to either OakDirectory.sync(...) or OakDirectory.close(...) indicates a checkpoint of the Lucene indexing process. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989147#comment-14989147 ] Ian Boston commented on OAK-3547: - No issues seen running for 12h. Need to look into improving recovery capabilities. Currently any change to any file referenced in a listing causes rejection of the listing. Some lucene index files are mutable (the delete file) and so checking must be relaxed for those files as the file will change. Need to analyse the changes in the log files to check what really is mutable and what is immutable in a listing. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[
https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989329#comment-14989329
]
Ian Boston commented on OAK-3547:
-
Attempts to break the index operation have produced recovery behaviour that
appears stable.
{code}
04.11.2015 10:02:20.391 *INFO* [aysnc-index-update-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Saving Listing state to l_1446631340351 as
_3.si,229,1a1c2c541a3cb5087d2a3c60ae77b6b05461410d;segments_3,117,a94bece6be54b2690a3b11925ed9bbc80e914d5d;segments_4,117,a74de5ae54af2abbb2423573cd0ad93b950ec18a;_2.cfe,224,dd2d758773e57e172933ff3d3fc3a4908af59dc4;_0_1.del,36,906f2506ff277e28716cc19eb8b55f289e34c53c;_0.si,229,57e0616d14993ad2a9680f55d4151d440cad8255;_2.si,229,ceddd18aa9343666e78f6330d0e261e96474717b;segments.gen,20,395c2b9ba7f05f4debb52b0a7cea8ac56ad671a2;_3.cfe,224,2596860e7bcdd550e221488708afda8729689107;_0.cfs,1431868,a7368bb6e2398a5952eddbc062a498f100a29865;_0.cfe,266,62127a60fe2224e32e3720eb15b2bd9f34d4670a;_3.cfs,1210,1c7764b0713716c53d1b0ed21c0063c5606aad49;_2.cfs,1188,1cd8b738619350e6998b77c2142d5a3748e861f1;
04.11.2015 10:02:20.391 *INFO* [aysnc-index-update-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Saving due to close.
04.11.2015 10:02:20.427 *INFO* [aysnc-index-update-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Saving Listing state to l_1446631340391 as
_3.si,229,1a1c2c541a3cb5087d2a3c60ae77b6b05461410d;segments_4,117,a74de5ae54af2abbb2423573cd0ad93b950ec18a;_0_1.del,36,906f2506ff277e28716cc19eb8b55f289e34c53c;_0.si,229,57e0616d14993ad2a9680f55d4151d440cad8255;segments.gen,20,395c2b9ba7f05f4debb52b0a7cea8ac56ad671a2;_3.cfe,224,2596860e7bcdd550e221488708afda8729689107;_0.cfs,1431868,a7368bb6e2398a5952eddbc062a498f100a29865;_0.cfe,266,62127a60fe2224e32e3720eb15b2bd9f34d4670a;_3.cfs,1210,1c7764b0713716c53d1b0ed21c0063c5606aad49;
04.11.2015 10:02:23.375 *WARN* [aysnc-index-update-fulltext-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
IO Exception reading index file
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing.getIndexFileMetaData(OakDirectory.java:965)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing.validateListing(OakDirectory.java:844)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing.load(OakDirectory.java:878)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing.(OakDirectory.java:750)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing.(OakDirectory.java:728)
04.11.2015 10:02:23.377 *WARN* [aysnc-index-update-fulltext-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Index File and Oak Version and not the same Name: _xi.fdt, length:14816174,
checksum:ff821a1bde2330f1389c782a47677206c685 CheckSum:
ff821a1bde2330f1389c782a47677206c685 != Unable to generate checksum
java.lang.RuntimeException: failed to read block from backend, id
b535214bddc090c74a426acaeeb5654140c1be52d4af824f2b759113c8a7bdc6@0,
04.11.2015 10:02:23.377 *WARN* [aysnc-index-update-fulltext-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Rejected directory listing l_1446631317105
04.11.2015 10:02:24.104 *INFO* [aysnc-index-update-fulltext-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Accepted directory listing l_1446631315966, using
04.11.2015 10:02:24.129 *INFO* [oak-lucene-0]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Accepted directory listing l_1446631317105, using
04.11.2015 10:02:25.034 *INFO* [aysnc-index-update-fulltext-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Saving Listing state to l_1446631344105 as
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987582#comment-14987582 ] Ian Boston commented on OAK-3547: - Added GC on index files and listing files. At present the code will keep a max of 100 directory listing files files and wont delete any index files referenced in those directory listing files. When there are 10 or more directory listing files to delete, they are deleted and the index files are GC'd. The number of directory listing files can be changed. The check happens every time a non read only OakDirectory is opened. its quick to perform the check. Code just pushed, testing using AEM6.1 on MongoMK overnight. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[
https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985293#comment-14985293
]
Ian Boston commented on OAK-3547:
-
Version just pushed works in AEM6.2 MongoMK with each save of the directory
list represented as list of files with file size and sha1 as below
{code}
02.11.2015 14:29:00.344 *INFO* [aysnc-index-update-fulltext-async]
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$GenerationalDirectoryListing
Saving Listing state to l_1446474540217 as
_6.si,229,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.fdx,1143274,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.pos,2256835,da39a3ee5e6b4b0d3255bfef95601890afd80709;_23.cfs,1592,da39a3ee5e6b4b0d3255bfef95601890afd80709;_6.cfe,224,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.fdt,12261250,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.tip,156795,da39a3ee5e6b4b0d3255bfef95601890afd80709;segments_1x,154,da39a3ee5e6b4b0d3255bfef95601890afd80709;_23.si,232,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.tim,6207095,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.si,316,da39a3ee5e6b4b0d3255bfef95601890afd80709;_23.cfe,224,da39a3ee5e6b4b0d3255bfef95601890afd80709;segments.gen,20,da39a3ee5e6b4b0d3255bfef95601890afd80709;_6.cfs,5403594,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.fnm,61229,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.doc,2073360,da39a3ee5e6b4b0d3255bfef95601890afd80709;_6_1.del,51,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b_1.del,38,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.nvd,142931,da39a3ee5e6b4b0d3255bfef95601890afd80709;_b.nvm,46,da39a3ee5e6b4b0d3255bfef95601890afd80709;
{code}
Since the save happens every few seconds, still need to do something to prune
the listings and to delete files no longer referenced. Probably best done with
code that runs every few hours.
Slightly concerned at the frequency of close or sync operations performed on
the OakDirectory
> Improve ability of the OakDirectory to recover from unexpected file errors
> --
>
> Key: OAK-3547
> URL: https://issues.apache.org/jira/browse/OAK-3547
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.4
>Reporter: Ian Boston
>
> Currently if the OakDirectory finds that a file is missing or in some way
> damaged, and exception is thrown which impacts all queries using that index,
> at times making the index unavailable. This improvement aims to make the
> OakDirectory recover to a previously ok state by storing which files were
> involved in previous states, and giving the code some way of checking if they
> are valid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3554) Use write concern of w:majority when connected to a replica set
[ https://issues.apache.org/jira/browse/OAK-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982198#comment-14982198 ] Ian Boston commented on OAK-3554: - [~chetanm] as discussed offline. While putting w=2 onto the query that updates the root commit should ensure that the data is replicated to sufficient secondaries to avoid data loss in the event of a primary election, you may still need w=1 to ensure that the data from previous queries is written to the journal and hence present in the oplog before the w=2 command hits. If you use w=0 or default, there is probably a risk that the data from previous queries wont have reached the oplog. I am basing that on a) experiments performed by others showing that MongoDB with w=0 and w=1 looses data in a replica set and b) and assumption that MongoDB is not completely single threaded. Also, the current advice is not to use a MK based datastore (performance & resilience) for production. In which case the performance impact of w=2 is considerably less, as the volume of traffic passing via the oplog is also considerably less. You may find that the difference between a global w=2 and a specific w=2 is negligible, as the oplog is sequential, and with any w=2 command, they will block until all the data is transferred to a secondary. If you think there is going to be benefit, it would be worth proving that under real load in a real cluster with more than 1 AEM instance connected to a 3 way replica set. > Use write concern of w:majority when connected to a replica set > --- > > Key: OAK-3554 > URL: https://issues.apache.org/jira/browse/OAK-3554 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Chetan Mehrotra > Fix For: 1.3.10 > > > Currently while connecting to Mongo MongoDocumentStore relies on default > write concern provided as part of mongouri. > Recently some issues were seen where Mongo based Oak was connecting to 3 > member replica set and there were frequent replica state changes due to use > of VM for Mongo. This caused data loss and corruption of data in Oak. > To avoid such situation Oak should default to write concern of majority by > default. If some write concern is specified as part of mongouri then that > should take precedence. This would allow system admin to take the call of > tweaking write concern if required and at same time allows Oak to use the > safe write concern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3554) Use write concern of w:majority when connected to a replica set
[ https://issues.apache.org/jira/browse/OAK-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982349#comment-14982349 ] Ian Boston commented on OAK-3554: - +1, looks good and safe to me. > Use write concern of w:majority when connected to a replica set > --- > > Key: OAK-3554 > URL: https://issues.apache.org/jira/browse/OAK-3554 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Chetan Mehrotra > Fix For: 1.3.10 > > > Currently while connecting to Mongo MongoDocumentStore relies on default > write concern provided as part of mongouri. > Recently some issues were seen where Mongo based Oak was connecting to 3 > member replica set and there were frequent replica state changes due to use > of VM for Mongo. This caused data loss and corruption of data in Oak. > To avoid such situation Oak should default to write concern of majority by > default. If some write concern is specified as part of mongouri then that > should take precedence. This would allow system admin to take the call of > tweaking write concern if required and at same time allows Oak to use the > safe write concern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982734#comment-14982734 ] Ian Boston commented on OAK-3547: - Patch in branch now passes build unit tests with the GenerationalDirectoryListing enabled. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971191#comment-14971191 ] Ian Boston edited comment on OAK-3547 at 10/24/15 7:27 AM: --- https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-3547?expand=1 Currently the patch makes no change, but puts the current behaviour behind an interface, and provides 2 implementations. A SimpleDirectoryListing that uses the current implementation and a GenerationalDirectoryListing that writes a new version of the node every time the listing is changed, as well as checking length and UUID of the file when the listing opens. A Checksum is not implemented, as that looked too expensive to achieve given the blob structure. The lifecycle of the GenerationalDirectoryListing is also wrong at present, as the length will have been calculated before writing is complete. was (Author: ianeboston): https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-3547?expand=1 Currently the patch makes no change, but puts the current behaviour behind an interface, and provides 2 implementations. A SimpleDirectoryListing that uses the current implementation and a GenerationalDirectoryListing that writes a new version of the node every time the listing is changed, as well as checking length and UUID of the file when the listing opens. A Checksum is not implemented, as that looked too expensive to achieve given the blob structure. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
Ian Boston created OAK-3547: --- Summary: Improve ability of the OakDirectory to recover from unexpected file errors Key: OAK-3547 URL: https://issues.apache.org/jira/browse/OAK-3547 Project: Jackrabbit Oak Issue Type: Improvement Components: lucene Affects Versions: 1.4 Reporter: Ian Boston Currently if the OakDirectory finds that a file is missing or in some way damaged, and exception is thrown which impacts all queries using that index, at times making the index unavailable. This improvement aims to make the OakDirectory recover to a previously ok state by storing which files were involved in previous states, and giving the code some way of checking if they are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3547) Improve ability of the OakDirectory to recover from unexpected file errors
[ https://issues.apache.org/jira/browse/OAK-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971191#comment-14971191 ] Ian Boston commented on OAK-3547: - https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-3547?expand=1 Currently the patch makes no change, but puts the current behaviour behind an interface, and provides 2 implementations. A SimpleDirectoryListing that uses the current implementation and a GenerationalDirectoryListing that writes a new version of the node every time the listing is changed, as well as checking length and UUID of the file when the listing opens. A Checksum is not implemented, as that looked too expensive to achieve given the blob structure. > Improve ability of the OakDirectory to recover from unexpected file errors > -- > > Key: OAK-3547 > URL: https://issues.apache.org/jira/browse/OAK-3547 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.4 >Reporter: Ian Boston > > Currently if the OakDirectory finds that a file is missing or in some way > damaged, and exception is thrown which impacts all queries using that index, > at times making the index unavailable. This improvement aims to make the > OakDirectory recover to a previously ok state by storing which files were > involved in previous states, and giving the code some way of checking if they > are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (OAK-3512) Synchronous index rebuilds can be performed by all commits once the reindex propery is true
[ https://issues.apache.org/jira/browse/OAK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reopened OAK-3512: - Reopening as more possible instances of the reindex=true property in the repository have been found. > Synchronous index rebuilds can be performed by all commits once the reindex > propery is true > --- > > Key: OAK-3512 > URL: https://issues.apache.org/jira/browse/OAK-3512 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.4 >Reporter: Ian Boston > > If reindex=true gets persisted to the database, then every commit triggers a > rebuild of the index. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3513) Session save going through despite repository being shutdown causing reindex flag to reset
[ https://issues.apache.org/jira/browse/OAK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956483#comment-14956483 ] Ian Boston commented on OAK-3513: - The patch fixes this issue, but does not eliminate the impact of the reindex flag if, for whatever reason, it is persisted into the repository. Since that flag being persisted to the repository, makes the repository unavailable, perhaps that should be eliminated ? It is valid to argue that the flag will never be persisted to the repository if the risk of that happening is absolutely zero, but that was argued before this issue was opened, evidenced by the code that set the flag. Should the commit hook that intercepts the flag and performs reindexing ever fail for whatever reason, the flag will get committed. imvho it would be better to check the NodeBuilder before and after to make certain the flag when true, is not true in the repository, as that is afaict the real desired behaviour of re-indexing. > Session save going through despite repository being shutdown causing reindex > flag to reset > -- > > Key: OAK-3513 > URL: https://issues.apache.org/jira/browse/OAK-3513 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Reporter: Chetan Mehrotra >Priority: Minor > Fix For: 1.3.9, 1.0.23, 1.2.8 > > Attachments: OAK-3513-1.patch > > > In some scenarios it can happen then repository instance has been shutdown > and removed from OSGi service registry but NodeStore is still active. In such > case if any thread has access to a JCR session instance then any commit done > via that session would still go through. > This causes issues as upon repository shutdown the whiteboard based trackers > would have lost any editor, validators and hence such commit can lead to > invalid data being committed. > Also in some cases it can lead to reindex flag for various sync indexes > getting reset to true as backing index editors are not present during the > commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3505) Provide an optionally stricter policy for missing synchronous index editor providers
[ https://issues.apache.org/jira/browse/OAK-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956460#comment-14956460 ] Ian Boston commented on OAK-3505: - Patch LGTM. Imho a good improvement. Thanks. > Provide an optionally stricter policy for missing synchronous index editor > providers > > > Key: OAK-3505 > URL: https://issues.apache.org/jira/browse/OAK-3505 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > Attachments: OAK-3505-v2.patch, OAK-3505.patch > > > Currently a missing synchronous index editor provider will mark the specific > index type with a reindex flag (OAK-3366). I'd like to provide the option to > setup a strategy to fail the commit until a specific impl comes online. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
