[jira] [Commented] (OAK-10814) Tar file recovery still fails with SNFE in some cases.

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847250#comment-17847250
 ] 

Axel Hanikel commented on OAK-10814:


Proposed patch: https://github.com/apache/jackrabbit-oak/pull/1468

Putting recovered segments in the SegmentCache makes them available during 
remaining recovery operations. In the case of the blobId map, this works as 
long as the blobId is stored in a data segment and not in a raw segment (raw 
segment's aren't cached). But the blobIds shouldn't become that large in 
practice.

> Tar file recovery still fails with SNFE in some cases.
> --
>
> Key: OAK-10814
> URL: https://issues.apache.org/jira/browse/OAK-10814
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: 1.62.0
>Reporter: Axel Hanikel
>Priority: Major
>
> {noformat}
> 13.05.2024 06:59:56.357   WARN[FelixStartLevel] 
> org.apache.jackrabbit.oak.segment.file.tar.SegmentTarReader Unable to load 
> index of file data21604a.tar: Unrecognized magic number
> 13.05.2024 06:59:56.369   INFO[FelixStartLevel] 
> org.apache.jackrabbit.oak.segment.file.tar.TarReader Backing up 
> data21604a.tar to data21604a.tar.bak
> 13.05.2024 06:59:56.372   INFO[FelixStartLevel] 
> org.apache.jackrabbit.oak.segment.file.tar.TarReader Regenerating tar file 
> data21604a.tar
> 13.05.2024 06:59:56.439   ERROR   [FelixStartLevel] 
> org.apache.jackrabbit.oak.segment.SegmentNotFoundExceptionListener Segment 
> not found: a4ad6e4b-93dc-4e09-ad0d-ab99a7f93652. SegmentId age=18ms
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
> a4ad6e4b-93dc-4e09-ad0d-ab99a7f93652 not found
>   at 
> org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:297)
>  [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:512)
>  [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163)
>  [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache.get(LocalCache.java:4019) 
> [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160)
>  [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:512)
>  [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) 
> [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71) 
> [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.ListRecord.getEntries(ListRecord.java:99) 
> [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.ListRecord.getEntries(ListRecord.java:92) 
> [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.SegmentStream.read(SegmentStream.java:166) 
> [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.io.ByteStreams.read(ByteStreams.java:934) 
> [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.io.ByteStreams.readFully(ByteStreams.java:802)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.guava.common.io.ByteStreams.readFully(ByteStreams.java:786)
>  [org.apache.jackrabbit.guava:1.62.0]
>   at 
> org.apache.jackrabbit.oak.segment.SegmentStream.getString(SegmentStream.java:105)
>  [org.apache.jackrabbit.oak-segment-tar:1.62.0]
>   at 
> 

[jira] [Created] (OAK-10814) Tar file recovery still fails with SNFE in some cases.

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10814:
--

 Summary: Tar file recovery still fails with SNFE in some cases.
 Key: OAK-10814
 URL: https://issues.apache.org/jira/browse/OAK-10814
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Affects Versions: 1.62.0
Reporter: Axel Hanikel


{noformat}
13.05.2024 06:59:56.357 WARN[FelixStartLevel] 
org.apache.jackrabbit.oak.segment.file.tar.SegmentTarReader Unable to load 
index of file data21604a.tar: Unrecognized magic number
13.05.2024 06:59:56.369 INFO[FelixStartLevel] 
org.apache.jackrabbit.oak.segment.file.tar.TarReader Backing up data21604a.tar 
to data21604a.tar.bak
13.05.2024 06:59:56.372 INFO[FelixStartLevel] 
org.apache.jackrabbit.oak.segment.file.tar.TarReader Regenerating tar file 
data21604a.tar
13.05.2024 06:59:56.439 ERROR   [FelixStartLevel] 
org.apache.jackrabbit.oak.segment.SegmentNotFoundExceptionListener Segment not 
found: a4ad6e4b-93dc-4e09-ad0d-ab99a7f93652. SegmentId age=18ms
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
a4ad6e4b-93dc-4e09-ad0d-ab99a7f93652 not found
at 
org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:297)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:512)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache.get(LocalCache.java:4019) 
[org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:512)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) 
[org.apache.jackrabbit.oak-segment-tar:1.62.0]
at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71) 
[org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.ListRecord.getEntries(ListRecord.java:99) 
[org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.ListRecord.getEntries(ListRecord.java:92) 
[org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.SegmentStream.read(SegmentStream.java:166) 
[org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.guava.common.io.ByteStreams.read(ByteStreams.java:934) 
[org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.io.ByteStreams.readFully(ByteStreams.java:802)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.guava.common.io.ByteStreams.readFully(ByteStreams.java:786)
 [org.apache.jackrabbit.guava:1.62.0]
at 
org.apache.jackrabbit.oak.segment.SegmentStream.getString(SegmentStream.java:105)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.Segment.readString(Segment.java:467) 
[org.apache.jackrabbit.oak-segment-tar:1.62.0]
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(SegmentBlob.java:248)
 [org.apache.jackrabbit.oak-segment-tar:1.62.0]
{noformat}
The reason seems to be that the long blobId string is stored behind a list 
record, where the elements are stored in yet a different segment, which is then 
loaded in the normal way, i.e. bypassing the recovery map added in OAK-10636.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10771) Add disk cache size stats and issue warning if evicted segment has zero length

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838961#comment-17838961
 ] 

Axel Hanikel commented on OAK-10771:


I've created a PR at https://github.com/apache/jackrabbit-oak/pull/1427

> Add disk cache size stats and issue warning if evicted segment has zero length
> --
>
> Key: OAK-10771
> URL: https://issues.apache.org/jira/browse/OAK-10771
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-azure
>Affects Versions: 1.62.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.64.0
>
>
> Note: The component is actually {{segment-remote}} but that's not on the list.
> The persistent disk cache computes its size internally by adding/subtracting 
> the size of added/purged segments. We would like to be able to see if that 
> computation is correct, by having both the computed size and the effective 
> size on disk in the metrics.
> The background for this is that in a few rare instances, the disk cache 
> evicted too many items (down to almost zero) instead of the expected 65% of 
> its max size.
> Also we want to make sure that no evictions happen on items that have a 
> length of zero. We're not sure if that happens or not, so the change is to 
> refuse eviction in that case and log a warning instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OAK-10771) Add disk cache size stats and issue warning if evicted segment has zero length

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10771:
--

 Summary: Add disk cache size stats and issue warning if evicted 
segment has zero length
 Key: OAK-10771
 URL: https://issues.apache.org/jira/browse/OAK-10771
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: segment-azure
Affects Versions: 1.62.0
Reporter: Axel Hanikel
 Fix For: 1.64.0


Note: The component is actually {{segment-remote}} but that's not on the list.

The persistent disk cache computes its size internally by adding/subtracting 
the size of added/purged segments. We would like to be able to see if that 
computation is correct, by having both the computed size and the effective size 
on disk in the metrics.

The background for this is that in a few rare instances, the disk cache evicted 
too many items (down to almost zero) instead of the expected 65% of its max 
size.

Also we want to make sure that no evictions happen on items that have a length 
of zero. We're not sure if that happens or not, so the change is to refuse 
eviction in that case and log a warning instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (OAK-10511) Get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel closed OAK-10511.
--

> Get blobId without additional network roundtrip
> ---
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There is another opportunity to save a network roundtrip to the blobstore, 
> please see: https://github.com/apache/jackrabbit-oak/pull/1171
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (OAK-10511) Get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel resolved OAK-10511.

Resolution: Fixed

> Get blobId without additional network roundtrip
> ---
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There is another opportunity to save a network roundtrip to the blobstore, 
> please see: https://github.com/apache/jackrabbit-oak/pull/1171
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10511) Get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10511:
---
Summary: Get blobId without additional network roundtrip  (was: Examine if 
we can get blobId without additional network roundtrip)

> Get blobId without additional network roundtrip
> ---
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There is another opportunity to save a network roundtrip to the blobstore, 
> please see: https://github.com/apache/jackrabbit-oak/pull/1171
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10511) Examine if we can get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10511:
---
Description: 
There is another opportunity to save a network roundtrip to the blobstore, 
please see: https://github.com/apache/jackrabbit-oak/pull/1171

/cc: [~miroslav] [~adulceanu] [~jelmini]

  was:
There might be another opportunity to save a network roundtrip to the 
blobstore, please see: https://github.com/apache/jackrabbit-oak/pull/1171

I haven't done any measurements yet, however.

/cc: [~miroslav] [~adulceanu] [~jelmini]


> Examine if we can get blobId without additional network roundtrip
> -
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There is another opportunity to save a network roundtrip to the blobstore, 
> please see: https://github.com/apache/jackrabbit-oak/pull/1171
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10511) Examine if we can get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781051#comment-17781051
 ] 

Axel Hanikel commented on OAK-10511:


And the same test *with the improvement*:

{noformat}
Imported 8 items in 2.406267986 seconds.
Imported 362 items in 96.624155226 seconds.
Imported 500 items in 128.375742179 seconds.
Imported 500 items in 126.733083455 seconds.
Imported 500 items in 108.835078016 seconds.
Imported 500 items in 124.852927464 seconds.
Imported 500 items in 129.963522259 seconds.
Imported 500 items in 136.40892435 seconds.
Imported 500 items in 115.523319625 seconds.
Imported 500 items in 112.908585932 seconds.
Imported 500 items in 136.981784626 seconds.
Imported 500 items in 137.053709139 seconds.
Imported 500 items in 120.102729396 seconds.
Imported 500 items in 131.966644238 seconds.
Imported 500 items in 104.596664242 seconds.
Imported 500 items in 115.902249698 seconds.
Imported 500 items in 135.871082709 seconds.
Imported 500 items in 114.618794848 seconds.
Imported 500 items in 141.146436755 seconds.
Imported 310 items in 76.644289048 seconds.
Imported 500 items in 88.396141203 seconds.
Imported 15 items in 3.446359968 seconds.
Imported 500 items in 86.040457931 seconds.
Imported 379 items in 68.886928061 seconds.
Imported 500 items in 131.189662681 seconds.
Imported 15 items in 1.691897343 seconds.
Imported 500 items in 127.275667817 seconds.
Imported 500 items in 143.110948869 seconds.
Imported 500 items in 94.625821854 seconds.
Imported 500 items in 143.871533409 seconds.
Imported 500 items in 140.969148967 seconds.
{noformat}
The average over 25 packages of 500 paths is here 123.093 secs. So the 
improvement looks about the same as the one for OAK-10494, which makes sense, 
because we're also avoiding superfluous {{downloadAttributes}} calls, just from 
a different angle.


> Examine if we can get blobId without additional network roundtrip
> -
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There might be another opportunity to save a network roundtrip to the 
> blobstore, please see: https://github.com/apache/jackrabbit-oak/pull/1171
> I haven't done any measurements yet, however.
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10511) Examine if we can get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781047#comment-17781047
 ] 

Axel Hanikel commented on OAK-10511:


Here are the numbers for importing replication packages on my test environment 
*without the improvement*:
{noformat}
Imported 8 items in 2.689354065 seconds.
Imported 362 items in 126.434390509 seconds.
Imported 500 items in 162.388030113 seconds.
Imported 500 items in 171.972125179 seconds.
Imported 500 items in 147.724703405 seconds.
Imported 500 items in 185.282611537 seconds.
Imported 500 items in 191.230882393 seconds.
Imported 500 items in 154.013032414 seconds.
Imported 500 items in 175.278139947 seconds.
Imported 500 items in 163.083704919 seconds.
Imported 500 items in 218.790697031 seconds.
Imported 500 items in 191.419393131 seconds.
Imported 500 items in 171.245269147 seconds.
Imported 500 items in 186.521936934 seconds.
Imported 500 items in 171.314902497 seconds.
Imported 500 items in 171.286761286 seconds.
Imported 500 items in 183.214408321 seconds.
Imported 500 items in 168.150800912 seconds.
Imported 500 items in 208.644648633 seconds.
Imported 500 items in 143.299461505 seconds.
Imported 310 items in 110.502035138 seconds.
Imported 15 items in 5.409807246 seconds.
Imported 500 items in 199.843748389 seconds.
Imported 500 items in 202.875469024 seconds.
Imported 379 items in 109.665630085 seconds.
Imported 500 items in 138.86516638 seconds.
Imported 500 items in 193.344911073 seconds.
Imported 15 items in 2.86283995 seconds.
Imported 500 items in 178.022908122 seconds.
Imported 500 items in 136.00731131 seconds.
Imported 500 items in 180.168462573 seconds.
Imported 500 items in 190.672982867 seconds.
{noformat}
That's an average of 176.333 secs over 26 packages when considering only the 
500-path packages.

> Examine if we can get blobId without additional network roundtrip
> -
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There might be another opportunity to save a network roundtrip to the 
> blobstore, please see: https://github.com/apache/jackrabbit-oak/pull/1171
> I haven't done any measurements yet, however.
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10511) Examine if we can get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10511:
---
Description: 
There might be another opportunity to save a network roundtrip to the 
blobstore, please see: https://github.com/apache/jackrabbit-oak/pull/1171

I haven't done any measurements yet, however.

/cc: [~miroslav] [~adulceanu] [~jelmini]

> Examine if we can get blobId without additional network roundtrip
> -
>
> Key: OAK-10511
> URL: https://issues.apache.org/jira/browse/OAK-10511
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
>
> There might be another opportunity to save a network roundtrip to the 
> blobstore, please see: https://github.com/apache/jackrabbit-oak/pull/1171
> I haven't done any measurements yet, however.
> /cc: [~miroslav] [~adulceanu] [~jelmini]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OAK-10511) Examine if we can get blobId without additional network roundtrip

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10511:
--

 Summary: Examine if we can get blobId without additional network 
roundtrip
 Key: OAK-10511
 URL: https://issues.apache.org/jira/browse/OAK-10511
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: segment-tar
Reporter: Axel Hanikel






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775733#comment-17775733
 ] 

Axel Hanikel edited comment on OAK-10494 at 10/16/23 12:17 PM:
---

{{getRecordIfStored()}} is called roughly 13600 times per replication package 
of 500 paths, so caching 13600 / 3 = 4600 items should be sufficient and 1 
is a safe bet.


was (Author: ahanikel):
{{getRecordIfStored()}} is called roughly 13600 times per replication package 
of 500 paths, so caching 4600 items should be sufficient and 1 is a safe 
bet.

> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: Measurements.md
>
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775733#comment-17775733
 ] 

Axel Hanikel edited comment on OAK-10494 at 10/16/23 12:15 PM:
---

{{getRecordIfStored()}} is called roughly 13600 times per replication package 
of 500 paths, so caching 4600 items should be sufficient and 1 is a safe 
bet.


was (Author: ahanikel):
{{getRecordIfStored()}} is called roughly 136000 times per replication package 
of 500 paths, so caching 46000 items should be sufficient and 10 is a safe 
bet.

> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: Measurements.md
>
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775733#comment-17775733
 ] 

Axel Hanikel commented on OAK-10494:


{{getRecordIfStored()}} is called roughly 136000 times per replication package 
of 500 paths, so caching 46000 items should be sufficient and 10 is a safe 
bet.

> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: Measurements.md
>
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775712#comment-17775712
 ] 

Axel Hanikel edited comment on OAK-10494 at 10/16/23 11:31 AM:
---

[~amitjain] Do you see any potential issues with this?
/cc: [~miroslav] [~adulceanu]


was (Author: ahanikel):
[~amitjain] Do you see any potential issues with this?

> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: Measurements.md
>
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775712#comment-17775712
 ] 

Axel Hanikel commented on OAK-10494:


[~amitjain] Do you see any potential issues with this?

> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: Measurements.md
>
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10494:
---
Attachment: Measurements.md

> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: Measurements.md
>
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10494:
---
Description: 
h1. Problem: Metadata are requested more than once for each blob

Setting a breakpoint in {{AbstractSharedCachingDataStore.getRecordIfStored()}} 
and logging the dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 
times for the same {{dataIdentifier}} when a replication package is installed 
by vault. The reason seems to be that during commits, every {{CommitHook}} runs 
its own {{compareAgainstBaseState}} and, because the implementation avoids 
fetching the blob if it only needs the metadata, the request to the existing 
blob cache is always a miss.

h1. Proposed solution: Cache {{backend.getRecord()}} calls
Manual testing has shown that caching {{backend.getRecord()}} calls reduces the 
time spent in {{.getRecordIfStored()}} by between 12 and 35% when installing 
replication packages containing 500 paths.

The PR is at https://github.com/apache/jackrabbit-oak/pull/1155

  was:
h1. Problem: Metadata are requested more than once for each blob

Setting a breakpoint in {{AbstractSharedCachingDataStore.getRecordIfStored()}} 
and logging the dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 
times for the same {{dataIdentifier}} when a replication package is installed 
by vault. The reason seems to be that during commits, every {{CommitHook}} runs 
its own {{compareAgainstBaseState}} and, because the implementation avoids 
fetching the blob if it only needs the metadata, the request to the existing 
blob cache is always a miss.

h1. Proposed solution: Cache {{backend.getRecord()}} calls
Manual testing has shown that caching {{backend.getRecord()}} calls reduces the 
time spent in {{.getRecordIfStored()}} by between 12 and 35% when installing 
replication packages containing 500 paths.


> Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() 
> over the network
> ---
>
> Key: OAK-10494
> URL: https://issues.apache.org/jira/browse/OAK-10494
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Reporter: Axel Hanikel
>Priority: Major
>
> h1. Problem: Metadata are requested more than once for each blob
> Setting a breakpoint in 
> {{AbstractSharedCachingDataStore.getRecordIfStored()}} and logging the 
> dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 times for the 
> same {{dataIdentifier}} when a replication package is installed by vault. The 
> reason seems to be that during commits, every {{CommitHook}} runs its own 
> {{compareAgainstBaseState}} and, because the implementation avoids fetching 
> the blob if it only needs the metadata, the request to the existing blob 
> cache is always a miss.
> h1. Proposed solution: Cache {{backend.getRecord()}} calls
> Manual testing has shown that caching {{backend.getRecord()}} calls reduces 
> the time spent in {{.getRecordIfStored()}} by between 12 and 35% when 
> installing replication packages containing 500 paths.
> The PR is at https://github.com/apache/jackrabbit-oak/pull/1155



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OAK-10494) Cache backend.getRecord() calls to minimise CloudBlob.downloadAttributes() over the network

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10494:
--

 Summary: Cache backend.getRecord() calls to minimise 
CloudBlob.downloadAttributes() over the network
 Key: OAK-10494
 URL: https://issues.apache.org/jira/browse/OAK-10494
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: blob-plugins
Reporter: Axel Hanikel


h1. Problem: Metadata are requested more than once for each blob

Setting a breakpoint in {{AbstractSharedCachingDataStore.getRecordIfStored()}} 
and logging the dataIdentifiers, we see that it calls {{backend.getRecord()}} 3 
times for the same {{dataIdentifier}} when a replication package is installed 
by vault. The reason seems to be that during commits, every {{CommitHook}} runs 
its own {{compareAgainstBaseState}} and, because the implementation avoids 
fetching the blob if it only needs the metadata, the request to the existing 
blob cache is always a miss.

h1. Proposed solution: Cache {{backend.getRecord()}} calls
Manual testing has shown that caching {{backend.getRecord()}} calls reduces the 
time spent in {{.getRecordIfStored()}} by between 12 and 35% when installing 
replication packages containing 500 paths.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (OAK-10345) Add debug logger for bytewise comparison of big blobs

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel resolved OAK-10345.

Resolution: Fixed

PR is merged.

> Add debug logger for bytewise comparison of big blobs
> -
>
> Key: OAK-10345
> URL: https://issues.apache.org/jira/browse/OAK-10345
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: store-spi
>Affects Versions: 1.52.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.54.0
>
>
> We would like to be able to get a debug log entry whenever AbstractBlob#equal 
> falls back to a byte-wise comparison of blobs larger than 100 MB.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10345) Add debug logger for bytewise comparison of big blobs

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10345:
---
Description: We would like to be able to get a debug log entry whenever 
AbstractBlob#equal falls back to a byte-wise comparison of blobs larger than 
100 MB.  (was: We would like to be able to get a debug log entry whenever 
AbstractBlob#equal falls back to a byte-wise comparison of blobs larger than 1 
MB.)

> Add debug logger for bytewise comparison of big blobs
> -
>
> Key: OAK-10345
> URL: https://issues.apache.org/jira/browse/OAK-10345
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: store-spi
>Affects Versions: 1.52.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.54.0
>
>
> We would like to be able to get a debug log entry whenever AbstractBlob#equal 
> falls back to a byte-wise comparison of blobs larger than 100 MB.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10345) Add debug logger for bytewise comparison of big blobs

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742787#comment-17742787
 ] 

Axel Hanikel commented on OAK-10345:


[~adulceanu] [~miroslav] I've created a PR for this at 
https://github.com/apache/jackrabbit-oak/pull/1024

> Add debug logger for bytewise comparison of big blobs
> -
>
> Key: OAK-10345
> URL: https://issues.apache.org/jira/browse/OAK-10345
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: store-spi
>Affects Versions: 1.52.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.54.0
>
>
> We would like to be able to get a debug log entry whenever AbstractBlob#equal 
> falls back to a byte-wise comparison of blobs larger than 1 MB.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OAK-10345) Add debug logger for bytewise comparison of big blobs

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10345:
--

 Summary: Add debug logger for bytewise comparison of big blobs
 Key: OAK-10345
 URL: https://issues.apache.org/jira/browse/OAK-10345
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: store-spi
Affects Versions: 1.52.0
Reporter: Axel Hanikel
 Fix For: 1.54.0


We would like to be able to get a debug log entry whenever AbstractBlob#equal 
falls back to a byte-wise comparison of blobs larger than 1 MB.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OAK-10311) Optimize SegmentBlob#equals for segment blobs that originate from the same blob store

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-10311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737046#comment-17737046
 ] 

Axel Hanikel commented on OAK-10311:


It looks like doing what's in the description is not possible: there can be 
more than one blobId pointing to the same blob (esp when using direct binary 
upload), so different blobIds do not mean that the blobs are different.

I'm still keeping the issue open for a bit, perhaps we get some other ideas for 
optimisation in the course of the discussion.

> Optimize SegmentBlob#equals for segment blobs that originate from the same 
> blob store
> -
>
> Key: OAK-10311
> URL: https://issues.apache.org/jira/browse/OAK-10311
> Project: Jackrabbit Oak
>  Issue Type: Story
>  Components: segment-tar
>Affects Versions: 1.52.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.54.0
>
>
> [SegmentBlob#equals|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.52.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java]
>  can be optimized for blobs originating from the same external blob store: In 
> that case, SegmentBlob#getBlobId() can be used to determine if the blobs are 
> equal or not. That change should avoid falling back to a byte-by-byte 
> comparison if the two blobs have the same size, which can be very expensive 
> for larger blobs.
> The optimization should be placed behind a code/feature toggle or system 
> property and disabled by default. The reason is that the contract of 
> [Blob#getContentIdentity|https://github.com/apache/jackrabbit-oak/blob/cc0521e1cf8dc10ad7a8d41a9f2d3fd2905e5c9b/oak-api/src/main/java/org/apache/jackrabbit/oak/api/Blob.java#L80-L82]
>  does not mention the special case where Blobs reside in the same blob store. 
> As part of this story, integration and benchmark tests should be created to 
> demonstrate gains in performance and correct behavior. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10311) Optimize SegmentBlob#equals for segment blobs that originate from the same blob store

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10311:
---
Description: 
[SegmentBlob#equals|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.52.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java]
 can be optimized for blobs originating from the same external blob store: In 
that case, SegmentBlob#getBlobId() can be used to determine if the blobs are 
equal or not. That change should avoid falling back to a byte-by-byte 
comparison if the two blobs have the same size, which can be very expensive for 
larger blobs.

The optimization should be placed behind a code/feature toggle or system 
property and disabled by default. The reason is that the contract of 
[Blob#getContentIdentity|https://github.com/apache/jackrabbit-oak/blob/cc0521e1cf8dc10ad7a8d41a9f2d3fd2905e5c9b/oak-api/src/main/java/org/apache/jackrabbit/oak/api/Blob.java#L80-L82]
 does not mention the special case where Blobs reside in the same blob store. 

As part of this story, integration and benchmark tests should be created to 
demonstrate gains in performance and correct behavior. 


  was:
[SegmentBlob#equals|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.52.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java]
 can be optimized for blobs originating from the same external blob store: In 
that case, SegmentBlob#getBlobId() can be used to determine if the blobs are 
equal or not. That change should avoid falling back to a byte-by-byte 
comparison if the two blobs aren't equal and therefore don't have the same 
ContentIdentity, which can be very expensive for larger blobs.

The optimization should be placed behind a code/feature toggle or system 
property and disabled by default. The reason is that the contract of 
[Blob#getContentIdentity|https://github.com/apache/jackrabbit-oak/blob/cc0521e1cf8dc10ad7a8d41a9f2d3fd2905e5c9b/oak-api/src/main/java/org/apache/jackrabbit/oak/api/Blob.java#L80-L82]
 does not mention the special case where Blobs reside in the same blob store. 

As part of this story, integration and benchmark tests should be created to 
demonstrate gains in performance and correct behavior. 



> Optimize SegmentBlob#equals for segment blobs that originate from the same 
> blob store
> -
>
> Key: OAK-10311
> URL: https://issues.apache.org/jira/browse/OAK-10311
> Project: Jackrabbit Oak
>  Issue Type: Story
>  Components: segment-tar
>Affects Versions: 1.52.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.54.0
>
>
> [SegmentBlob#equals|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.52.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java]
>  can be optimized for blobs originating from the same external blob store: In 
> that case, SegmentBlob#getBlobId() can be used to determine if the blobs are 
> equal or not. That change should avoid falling back to a byte-by-byte 
> comparison if the two blobs have the same size, which can be very expensive 
> for larger blobs.
> The optimization should be placed behind a code/feature toggle or system 
> property and disabled by default. The reason is that the contract of 
> [Blob#getContentIdentity|https://github.com/apache/jackrabbit-oak/blob/cc0521e1cf8dc10ad7a8d41a9f2d3fd2905e5c9b/oak-api/src/main/java/org/apache/jackrabbit/oak/api/Blob.java#L80-L82]
>  does not mention the special case where Blobs reside in the same blob store. 
> As part of this story, integration and benchmark tests should be created to 
> demonstrate gains in performance and correct behavior. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OAK-10311) Optimize SegmentBlob#equals for segment blobs that originate from the same blob store

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10311:
--

 Summary: Optimize SegmentBlob#equals for segment blobs that 
originate from the same blob store
 Key: OAK-10311
 URL: https://issues.apache.org/jira/browse/OAK-10311
 Project: Jackrabbit Oak
  Issue Type: Story
  Components: segment-tar
Affects Versions: 1.52.0
Reporter: Axel Hanikel
 Fix For: 1.54.0


[SegmentBlob#equals|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.52.0/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java]
 can be optimized for blobs originating from the same external blob store: In 
that case, SegmentBlob#getBlobId() can be used to determine if the blobs are 
equal or not. That change should avoid falling back to a byte-by-byte 
comparison if the two blobs aren't equal and therefore don't have the same 
ContentIdentity, which can be very expensive for larger blobs.

The optimization should be placed behind a code/feature toggle or system 
property and disabled by default. The reason is that the contract of 
[Blob#getContentIdentity|https://github.com/apache/jackrabbit-oak/blob/cc0521e1cf8dc10ad7a8d41a9f2d3fd2905e5c9b/oak-api/src/main/java/org/apache/jackrabbit/oak/api/Blob.java#L80-L82]
 does not mention the special case where Blobs reside in the same blob store. 

As part of this story, integration and benchmark tests should be created to 
demonstrate gains in performance and correct behavior. 




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (OAK-10049) Enable access to the secondary Azure blobstore service endpoint in Oak blobstore

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-10049:
---
Summary: Enable access to the secondary Azure blobstore service endpoint in 
Oak blobstore  (was: Enable access to the secondary Azure blobstore service 
endpoint in Oak datastore)

> Enable access to the secondary Azure blobstore service endpoint in Oak 
> blobstore
> 
>
> Key: OAK-10049
> URL: https://issues.apache.org/jira/browse/OAK-10049
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud-azure
>Affects Versions: 1.46.0
>Reporter: Axel Hanikel
>Priority: Major
>
> Create a property for the azure blobstore to enable fallback to the secondary 
> location 
> ({{requestOptions.setLocationMode(LocationMode.PRIMARY_THEN_SECONDARY);}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OAK-10049) Enable access to the secondary Azure blobstore service endpoint in Oak datastore

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-10049:
--

 Summary: Enable access to the secondary Azure blobstore service 
endpoint in Oak datastore
 Key: OAK-10049
 URL: https://issues.apache.org/jira/browse/OAK-10049
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: blob-cloud-azure
Affects Versions: 1.46.0
Reporter: Axel Hanikel


Create a property for the azure blobstore to enable fallback to the secondary 
location 
({{requestOptions.setLocationMode(LocationMode.PRIMARY_THEN_SECONDARY);}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel resolved OAK-7932.
---
Resolution: Abandoned

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> *The first idea* was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> *A second implementation*, was a simple
> nodestore implementation which was kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy (compared with the segment store with its fine-grained record 
> structure) wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  *A third implementation*, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is based 
> on the second one but uses a more compact serialisation format, which records 
> the differences between a new nodestate and its base state, similar to what 
> {{.compareAgainstBaseState()}} produces in code. It looks like this:
> {noformat}
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 
> 823f2252-db37-b0ca-3f7e-09cd073b530a ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ root 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ checkpoints 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ blobs 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 journal golden 
> 823f2252-db37-b0ca-3f7e-09cd073b530a
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-151 checkpoints golden 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 
> 026cb997-d3f7-2c9f-a5ad-d7396b86c267 ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 p+ :clusterId  = 
> 

[jira] [Commented] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17647932#comment-17647932
 ] 

Axel Hanikel commented on OAK-7932:
---

h2. Final status

The implementation has changed quite a bit from what's described here. 
[https://ahanikel.github.io/jackrabbit-oak] has an up-to-date description. 
Although there are some issues left, this concludes the PoC.

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> *The first idea* was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> *A second implementation*, was a simple
> nodestore implementation which was kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy (compared with the segment store with its fine-grained record 
> structure) wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  *A third implementation*, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is based 
> on the second one but uses a more compact serialisation format, which records 
> the differences between a new nodestate and its base state, similar to what 
> {{.compareAgainstBaseState()}} produces in code. It looks like this:
> {noformat}
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 
> 823f2252-db37-b0ca-3f7e-09cd073b530a ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ root 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ checkpoints 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ blobs 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 journal golden 
> 823f2252-db37-b0ca-3f7e-09cd073b530a
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-151 checkpoints 

[jira] [Comment Edited] (OAK-9473) [Cold Standby] Add configuration option for SSL key decryption password

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368942#comment-17368942
 ] 

Axel Hanikel edited comment on OAK-9473 at 6/24/21, 3:54 PM:
-

Attaching a patch for adding the option to configure a password in OSGi which 
can decrypt encrypted SSL keys.

Additionally I've renamed the configuration options {{sslServerSubjectPattern}} 
and {{sslClientSubjectPattern}} into a single {{sslSubjectPattern}} because we 
never need both in the same instance. I hope it's still ok to do that change 
given that we haven't had an Oak release since these were introduced.

I've also created a PR at https://github.com/apache/jackrabbit-oak/pull/301

[~adulceanu] Would you like to have a look? Thanks!



was (Author: ahanikel):
Attaching a patch for adding the option to configure a password in OSGi which 
can decrypt encrypted SSL keys.

Additionally I've renamed the configuration options {sslServerSubjectPattern} 
and {sslClientSubjectPattern} into a single {sslSubjectPattern} because we 
never need both in the same instance. I hope it's still ok to do that change 
given that we haven't had an Oak release since these were introduced.

I've also created a PR at https://github.com/apache/jackrabbit-oak/pull/301

[~adulceanu] Would you like to have a look? Thanks!


> [Cold Standby] Add configuration option for SSL key decryption password
> ---
>
> Key: OAK-9473
> URL: https://issues.apache.org/jira/browse/OAK-9473
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: 
> 0001-OAK-9473-Cold-Standby-configure-decryption-password-.patch
>
>
> We also want to be able to use encrypted keys for cold standby SSL 
> connections. The password would then be stored and fetched from the OSGi 
> config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-9473) [Cold Standby] Add configuration option for SSL key decryption password

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368942#comment-17368942
 ] 

Axel Hanikel commented on OAK-9473:
---

Attaching a patch for adding the option to configure a password in OSGi which 
can decrypt encrypted SSL keys.

Additionally I've renamed the configuration options {sslServerSubjectPattern} 
and {sslClientSubjectPattern} into a single {sslSubjectPattern} because we 
never need both in the same instance. I hope it's still ok to do that change 
given that we haven't had an Oak release since these were introduced.

I've also created a PR at https://github.com/apache/jackrabbit-oak/pull/301

[~adulceanu] Would you like to have a look? Thanks!


> [Cold Standby] Add configuration option for SSL key decryption password
> ---
>
> Key: OAK-9473
> URL: https://issues.apache.org/jira/browse/OAK-9473
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: 
> 0001-OAK-9473-Cold-Standby-configure-decryption-password-.patch
>
>
> We also want to be able to use encrypted keys for cold standby SSL 
> connections. The password would then be stored and fetched from the OSGi 
> config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-9473) [Cold Standby] Add configuration option for SSL key decryption password

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-9473:
--
Attachment: 0001-OAK-9473-Cold-Standby-configure-decryption-password-.patch

> [Cold Standby] Add configuration option for SSL key decryption password
> ---
>
> Key: OAK-9473
> URL: https://issues.apache.org/jira/browse/OAK-9473
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Axel Hanikel
>Priority: Major
> Attachments: 
> 0001-OAK-9473-Cold-Standby-configure-decryption-password-.patch
>
>
> We also want to be able to use encrypted keys for cold standby SSL 
> connections. The password would then be stored and fetched from the OSGi 
> config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (OAK-9473) [Cold Standby] Add configuration option for SSL key decryption password

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-9473:
-

 Summary: [Cold Standby] Add configuration option for SSL key 
decryption password
 Key: OAK-9473
 URL: https://issues.apache.org/jira/browse/OAK-9473
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: segment-tar
Reporter: Axel Hanikel


We also want to be able to use encrypted keys for cold standby SSL connections. 
The password would then be stored and fetched from the OSGi config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-9469) Unsuccessful lease refresh in AzureRepositoryLock can cause two processes using Oak to have write access

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-9469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17365817#comment-17365817
 ] 

Axel Hanikel commented on OAK-9469:
---

[~miroslav] I think we should retry forever (with a delay) and write a log 
entry every minute or so as long as the problem persists to trigger alerts. 
There is no way to remedy the situation, and giving up won't help either, so I 
think the best thing we can do is keep on trying.

> Unsuccessful lease refresh in AzureRepositoryLock can cause two processes 
> using Oak to have write access 
> -
>
> Key: OAK-9469
> URL: https://issues.apache.org/jira/browse/OAK-9469
> Project: Jackrabbit Oak
>  Issue Type: Bug
>Reporter: Miroslav Smiljanic
>Assignee: Miroslav Smiljanic
>Priority: Major
> Attachments: OAK-9469_op_timeout.patch, OAK-9469_test.patch
>
>
> Lease for *repo.lock* is initially 
> [requested|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.40.0/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureRepositoryLock.java#L68]
>  for period of 60 seconds, and later it is being periodically 
> [renewed|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.40.0/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureRepositoryLock.java#L95]
>  in the separate thread. 
> Renewal of the lease can be unsuccessful, when StorageException is being 
> thrown, and shutdown hook being 
> [invoked|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.40.0/oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureRepositoryLock.java#L107].
> Shutdown hood defined in AzurePersistence is basically NOOP that only prints 
> warning log message. Process running Oak will continue to run.
> Later the other process running Oak can try to acquire the same lock and will 
> be able to do it. From that moment two processes might have write access over 
> the remote repo, and cause conflicts to each other.
> Test case that demonstrates the issue in AzureRepositoryLock: 
> [^OAK-9469_test.patch]
> One cause for the StorageException to be thrown, that I have seen is 
> operation timeout.
> {noformat}
> [pool-13-thread-1] 
> org.apache.jackrabbit.oak.segment.azure.AzureRepositoryLock Can't renew the 
> lease
> com.microsoft.azure.storage.StorageException: The client could not finish the 
> operation within specified maximum execution timeout.
>   at 
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:254)
>   at 
> com.microsoft.azure.storage.blob.CloudBlob.renewLease(CloudBlob.java:2866)
>   at 
> com.microsoft.azure.storage.blob.CloudBlob.renewLease(CloudBlob.java:2832)
>   at 
> org.apache.jackrabbit.oak.segment.azure.AzureRepositoryLock.refreshLease(AzureRepositoryLock.java:102)
>  [org.apache.jackrabbit.oak-segment-azure:1.39.0.R1888564]
>   at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>   at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
>   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown 
> Source)
>   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
> Source)
>   at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: The client could not finish 
> the operation within specified maximum execution timeout.
>   at 
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:253)
>   ... 8 common frames omitted
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-9451) Cold Standby SSL certificates should be configurable

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-9451:
--
Attachment: OAK-9451.patch.txt

> Cold Standby SSL certificates should be configurable
> 
>
> Key: OAK-9451
> URL: https://issues.apache.org/jira/browse/OAK-9451
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Andrei Dulceanu
>Priority: Major
> Attachments: OAK-9451.patch.txt
>
>
> The cold standby is able to do SSL connections to the primary, but currently 
> only using on-the-fly generated certificates. This means that data is 
> transferred over an encrypted connection but there is no protection against a 
> man in the middle yet.
> With this issue we want to:
> * make server and client certificates configurable
> * optionally validate the client certificate
> * optionally only allow matching subjects in client and server certificates 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-9451) Cold Standby SSL certificates should be configurable

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357192#comment-17357192
 ] 

Axel Hanikel commented on OAK-9451:
---

[~adulceanu] Sure, I've created it here: 
https://github.com/apache/jackrabbit-oak/pull/297

> Cold Standby SSL certificates should be configurable
> 
>
> Key: OAK-9451
> URL: https://issues.apache.org/jira/browse/OAK-9451
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Andrei Dulceanu
>Priority: Major
>
> The cold standby is able to do SSL connections to the primary, but currently 
> only using on-the-fly generated certificates. This means that data is 
> transferred over an encrypted connection but there is no protection against a 
> man in the middle yet.
> With this issue we want to:
> * make server and client certificates configurable
> * optionally validate the client certificate
> * optionally only allow matching subjects in client and server certificates 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (OAK-9451) Cold Standby SSL certificates should be configurable

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-9451:
-

 Summary: Cold Standby SSL certificates should be configurable
 Key: OAK-9451
 URL: https://issues.apache.org/jira/browse/OAK-9451
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Reporter: Axel Hanikel


The cold standby is able to do SSL connections to the primary, but currently 
only using on-the-fly generated certificates. This means that data is 
transferred over an encrypted connection but there is no protection against a 
man in the middle yet.

With this issue we want to:
* make server and client certificates configurable
* optionally validate the client certificate
* optionally only allow matching subjects in client and server certificates 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322179#comment-17322179
 ] 

Axel Hanikel commented on OAK-7932:
---

h2. Current status
I've replaced 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] with what 
was previously referred to as the "third implementation". It is based on oak 
1.38.0 and produces version 1.38.0-zeromq when built with mvn install. The 
universal threadId has been prepended to every event. This enables several 
threads to write their events intertwined and the NodeStateAggregator is able 
to tell them apart.

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> *The first idea* was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> *A second implementation*, was a simple
> nodestore implementation which was kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy (compared with the segment store with its fine-grained record 
> structure) wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  *A third implementation*, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is based 
> on the second one but uses a more compact serialisation format, which records 
> the differences between a new nodestate and its base state, similar to what 
> {{.compareAgainstBaseState()}} produces in code. It looks like this:
> {noformat}
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 
> 823f2252-db37-b0ca-3f7e-09cd073b530a ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ root 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ checkpoints 
> ----
> cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ blobs 
> 

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

*The first idea* was to use the segment node store, and ZeroMQ for 
communication because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

*A second implementation*, was a simple
nodestore implementation which was kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

Here is what a serialised nodestate looks like:
{noformat}
begin ZeroMQNodeState
begin children
allow   856d1356-7054-3993-894b-a04426956a78
end children
begin properties
jcr:primaryType  = rep:ACL
:childOrder  = [allow]
end properties
end ZeroMQNodeState
{noformat}

This verbose format is good for debugging but expensive to generate and parse, 
so it may be replaced with a binary format at some point. But it shows how 
child nodestates are referenced and how properties are represented. Binary 
properties (not shown here) are represented by a reference to the blob store.

The redundancy (compared with the segment store with its fine-grained record 
structure) wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 *A third implementation*, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is based on 
the second one but uses a more compact serialisation format, which records the 
differences between a new nodestate and its base state, similar to what 
{{.compareAgainstBaseState()}} produces in code. It looks like this:

{noformat}
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 823f2252-db37-b0ca-3f7e-09cd073b530a 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ root 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ checkpoints 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ blobs 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 journal golden 
823f2252-db37-b0ca-3f7e-09cd073b530a
cc97d19b-efa9-4f64-a5e9-a94d9418f271-151 checkpoints golden 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 026cb997-d3f7-2c9f-a5ad-d7396b86c267 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 p+ :clusterId  = 
0cb833ee-2678-4383-93dd-fe993cbfd56a
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 3ceb887c-0875-1171-94d2-df9bc8a57edd 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ :clusterConfig 
026cb997-d3f7-2c9f-a5ad-d7396b86c267
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: ce8ace33-9d80-d6e1-c4ba-f0882f75e9c9 
823f2252-db37-b0ca-3f7e-09cd073b530a
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n^ root 
3ceb887c-0875-1171-94d2-df9bc8a57edd 

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

*The first idea* was to use the segment node store, and ZeroMQ for 
communication because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

*A second implementation*, was a simple
nodestore implementation which was kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

Here is what a serialised nodestate looks like:
{noformat}
begin ZeroMQNodeState
begin children
allow   856d1356-7054-3993-894b-a04426956a78
end children
begin properties
jcr:primaryType  = rep:ACL
:childOrder  = [allow]
end properties
end ZeroMQNodeState
{noformat}

This verbose format is good for debugging but expensive to generate and parse, 
so it may be replaced with a binary format at some point. But it shows how 
child nodestates are referenced and how properties are represented. Binary 
properties (not shown here) are represented by a reference to the blob store.

The redundancy (compared with the segment store with its fine-grained record 
structure) wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 *A third implementation*, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is based on 
the second one but uses a more compact serialisation format, which records the 
differences between a new nodestate and its base state, similar to what 
{{.compareAgainstBaseState()}} produces in code. It looks like this:

{noformat}
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 823f2252-db37-b0ca-3f7e-09cd073b530a 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ root 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ checkpoints 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ blobs 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 journal golden 
823f2252-db37-b0ca-3f7e-09cd073b530a
cc97d19b-efa9-4f64-a5e9-a94d9418f271-151 checkpoints golden 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 026cb997-d3f7-2c9f-a5ad-d7396b86c267 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 p+ :clusterId  = 
0cb833ee-2678-4383-93dd-fe993cbfd56a
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: 3ceb887c-0875-1171-94d2-df9bc8a57edd 
----
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n+ :clusterConfig 
026cb997-d3f7-2c9f-a5ad-d7396b86c267
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n!
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n: ce8ace33-9d80-d6e1-c4ba-f0882f75e9c9 
823f2252-db37-b0ca-3f7e-09cd073b530a
cc97d19b-efa9-4f64-a5e9-a94d9418f271-26 n^ root 
3ceb887c-0875-1171-94d2-df9bc8a57edd 

[jira] [Comment Edited] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300266#comment-17300266
 ] 

Axel Hanikel edited comment on OAK-7932 at 3/12/21, 12:15 PM:
--

h2. Current status
The "third implementation" described above works pretty well although it is 
still a bit slow. The code is at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore-kafka-finegrained-checkpoints-1.38.0].
 The next steps would be to find the bottlenecks and to change the internal 
handling of nodestates. At the moment they are "special" but they should 
actually be handled just like blobs, because that's what they are. It would 
also make the code simpler.


was (Author: ahanikel):
h1. Current status
The "third implementation" described above works pretty well although it is 
still a bit slow. The code is at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore-kafka-finegrained-checkpoints-1.38.0].
 The next steps would be to find the bottlenecks and to change the internal 
handling of nodestates. At the moment they are "special" but they should 
actually be handled just like blobs, because that's what they are. It would 
also make the code simpler.

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> *The first idea* was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> *A second implementation*, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy (compared with the segment store with its fine-grained record 
> structure) wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  *A third implementation*, at 
> 

[jira] [Commented] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300266#comment-17300266
 ] 

Axel Hanikel commented on OAK-7932:
---

h1. Current status
The "third implementation" described above works pretty well although it is 
still a bit slow. The code is at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore-kafka-finegrained-checkpoints-1.38.0].
 The next steps would be to find the bottlenecks and to change the internal 
handling of nodestates. At the moment they are "special" but they should 
actually be handled just like blobs, because that's what they are. It would 
also make the code simpler.

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> *The first idea* was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> *A second implementation*, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy (compared with the segment store with its fine-grained record 
> structure) wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  *A third implementation*, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore-kafka-finegrained-checkpoints-1.38.0]
>  is based on the second one but uses a more compact serialisation format, 
> which records the differences between a new nodestate and its base state, 
> similar to what {{.compareAgainstBaseState()}} produces in code. It looks 
> like this:
> {noformat}
> n: 823f2252-db37-b0ca-3f7e-09cd073b530a ----
> n+ root ----
> n+ checkpoints ----
> n+ blobs 

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

*The first idea* was to use the segment node store, and ZeroMQ for 
communication because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

*A second implementation*, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

Here is what a serialised nodestate looks like:
{noformat}
begin ZeroMQNodeState
begin children
allow   856d1356-7054-3993-894b-a04426956a78
end children
begin properties
jcr:primaryType  = rep:ACL
:childOrder  = [allow]
end properties
end ZeroMQNodeState
{noformat}

This verbose format is good for debugging but expensive to generate and parse, 
so it may be replaced with a binary format at some point. But it shows how 
child nodestates are referenced and how properties are represented. Binary 
properties (not shown here) are represented by a reference to the blob store.

The redundancy (compared with the segment store with its fine-grained record 
structure) wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 *A third implementation*, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore-kafka-finegrained-checkpoints-1.38.0]
 is based on the second one but uses a more compact serialisation format, which 
records the differences between a new nodestate and its base state, similar to 
what {{.compareAgainstBaseState()}} produces in code. It looks like this:

{noformat}
n: 823f2252-db37-b0ca-3f7e-09cd073b530a ----
n+ root ----
n+ checkpoints ----
n+ blobs ----
n!
journal golden 823f2252-db37-b0ca-3f7e-09cd073b530a
checkpoints golden ----
n: 394cdbf7-c75b-1feb-23f9-2ca93a8d97eb ----
p+ :clusterId  = 2441f7f8-5ce8-493f-9b27-c4bca802ea0b
n!
n: 794ccfa2-4372-5b05-c54f-d2b6b6d2df03 ----
n+ :clusterConfig 394cdbf7-c75b-1feb-23f9-2ca93a8d97eb
n!
n: fe9f9008-7734-bd69-17d0-ccecf4003323 823f2252-db37-b0ca-3f7e-09cd073b530a
n^ root 794ccfa2-4372-5b05-c54f-d2b6b6d2df03 
----
n!
journal golden fe9f9008-7734-bd69-17d0-ccecf400332
b64+ 516f4f64-4e5c-432e-b9bf-0267c1e96541
b64d 
ClVVRU5DT0RFKDEpICAgICAgICAgICAgICAgQlNEIEdlbmVyYWwgQ29tbWFuZHMgTWFudWFsICAg
b64d 
ICAgICAgICAgICBVVUVOQ09ERSgxKQoKTghOQQhBTQhNRQhFCiAgICAgdQh1dQh1ZAhkZQhlYwhj
b64d 
bwhvZAhkZQhlLCB1CHV1CHVlCGVuCG5jCGNvCG9kCGRlCGUgLS0gZW5jb2RlL2RlY29kZSBhIGJp
b64d 
bmFyeSBmaWxlCgpTCFNZCFlOCE5PCE9QCFBTCFNJCElTCFMKICAgICB1CHV1CHVlCGVuCG5jCGNv
b64d 

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

The first idea was to use the segment node store, and ZeroMQ for communication 
because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

Here is what a serialised nodestate looks like:
{noformat}
begin ZeroMQNodeState
begin children
allow   856d1356-7054-3993-894b-a04426956a78
end children
begin properties
jcr:primaryType  = rep:ACL
:childOrder  = [allow]
end properties
end ZeroMQNodeState
{noformat}

This verbose format is good for debugging but expensive to generate and parse, 
so it may be replaced with a binary format at some point. But it shows how 
child nodestates are referenced and how properties are represented. Binary 
properties (not shown here) are represented by a reference to the blob store.

The redundancy (compared with the segment store with its fine-grained record 
structure) wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 A third implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore-kafka-finegrained-checkpoints-1.38.0]
 is based on the second one but uses a more compact serialisation format, which 
records the differences between a new nodestate and its base state, similar to 
what {{.compareAgainstBaseState()}} produces in code. It looks like this:

{noformat}
n: 823f2252-db37-b0ca-3f7e-09cd073b530a ----
n+ root ----
n+ checkpoints ----
n+ blobs ----
n!
journal golden 823f2252-db37-b0ca-3f7e-09cd073b530a
checkpoints golden ----
n: 394cdbf7-c75b-1feb-23f9-2ca93a8d97eb ----
p+ :clusterId  = 2441f7f8-5ce8-493f-9b27-c4bca802ea0b
n!
n: 794ccfa2-4372-5b05-c54f-d2b6b6d2df03 ----
n+ :clusterConfig 394cdbf7-c75b-1feb-23f9-2ca93a8d97eb
n!
n: fe9f9008-7734-bd69-17d0-ccecf4003323 823f2252-db37-b0ca-3f7e-09cd073b530a
n^ root 794ccfa2-4372-5b05-c54f-d2b6b6d2df03 
----
n!
journal golden fe9f9008-7734-bd69-17d0-ccecf400332
b64+ 516f4f64-4e5c-432e-b9bf-0267c1e96541
b64d 
ClVVRU5DT0RFKDEpICAgICAgICAgICAgICAgQlNEIEdlbmVyYWwgQ29tbWFuZHMgTWFudWFsICAg
b64d 
ICAgICAgICAgICBVVUVOQ09ERSgxKQoKTghOQQhBTQhNRQhFCiAgICAgdQh1dQh1ZAhkZQhlYwhj
b64d 
bwhvZAhkZQhlLCB1CHV1CHVlCGVuCG5jCGNvCG9kCGRlCGUgLS0gZW5jb2RlL2RlY29kZSBhIGJp
b64d 
bmFyeSBmaWxlCgpTCFNZCFlOCE5PCE9QCFBTCFNJCElTCFMKICAgICB1CHV1CHVlCGVuCG5jCGNv
b64d 

[jira] [Commented] (OAK-9338) RemotePersistentCacheService throws if both redis and disk cache are disabled

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272280#comment-17272280
 ] 

Axel Hanikel commented on OAK-9338:
---

Proposed fix:

{noformat}
diff --git 
a/oak-segment-remote/src/main/java/org/apache/jackrabbit/oak/segment/remote/persistentcache/RemotePersistentCacheService.java
 b/oak-segment-remote/src/main/java/org/apache/jackrabbit/oak/segment/r\
emote/persistentcache/RemotePersistentCacheService.java
index 4fd074a9d3..b8c4aa38f9 100644
--- 
a/oak-segment-remote/src/main/java/org/apache/jackrabbit/oak/segment/remote/persistentcache/RemotePersistentCacheService.java
+++ 
b/oak-segment-remote/src/main/java/org/apache/jackrabbit/oak/segment/remote/persistentcache/RemotePersistentCacheService.java
@@ -60,7 +60,9 @@ public class RemotePersistentCacheService {
 public void activate(ComponentContext context, Configuration config) 
throws IOException {
 osgiWhiteboard = new OsgiWhiteboard(context.getBundleContext());
 persistentCache = createPersistentCache(config, closer);
-registration = 
context.getBundleContext().registerService(PersistentCache.class.getName(), 
persistentCache, new Properties());
+if (persistentCache != null) {
+registration = 
context.getBundleContext().registerService(PersistentCache.class.getName(), 
persistentCache, new Properties());
+}
 }

 @Deactivate
{noformat}

[~adulceanu] Could you have a look please? :-)


> RemotePersistentCacheService throws if both redis and disk cache are disabled
> -
>
> Key: OAK-9338
> URL: https://issues.apache.org/jira/browse/OAK-9338
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: 1.38.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.40.0
>
>
> RemotePersistentCacheService throws if both redis and disk cache are disabled 
> in the configuration:
> {noformat}
> [FelixLogListener] 
> org.apache.jackrabbit.oak.segment.remote.persistentcache.RemotePersistentCacheService
>  bundle org.apache.jackrabbit.oak-segment-remote:1.37.0.R1885266 
> (135)[org.apache.jackrabbit.oak.segment.remote.persistentcache.RemotePersistentCacheService(224)]
>  : The activate method has thrown an exception 
> (java.lang.IllegalArgumentException: Service object cannot be null.)
> java.lang.IllegalArgumentException: Service object cannot be null.
>   at org.apache.felix.framework.Felix.registerService(Felix.java:3748)
>   at 
> org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
>   at 
> org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:302)
>   at 
> org.apache.jackrabbit.oak.segment.remote.persistentcache.RemotePersistentCacheService.activate(RemotePersistentCacheService.java:63)
>  [org.apache.jackrabbit.oak-segment-remote:1.37.0.R1885266]
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:242)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:678)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:524)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:318)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:308)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:342)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:115)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:984)
>  [org.apache.felix.scr:2.1.20]
>   at 
> org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:957)
>  [org.apache.felix.scr:2.1.20]
>   at 
> 

[jira] [Created] (OAK-9338) RemotePersistentCacheService throws if both redis and disk cache are disabled

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-9338:
-

 Summary: RemotePersistentCacheService throws if both redis and 
disk cache are disabled
 Key: OAK-9338
 URL: https://issues.apache.org/jira/browse/OAK-9338
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Affects Versions: 1.38.0
Reporter: Axel Hanikel
 Fix For: 1.40.0


RemotePersistentCacheService throws if both redis and disk cache are disabled 
in the configuration:

{noformat}
[FelixLogListener] 
org.apache.jackrabbit.oak.segment.remote.persistentcache.RemotePersistentCacheService
 bundle org.apache.jackrabbit.oak-segment-remote:1.37.0.R1885266 
(135)[org.apache.jackrabbit.oak.segment.remote.persistentcache.RemotePersistentCacheService(224)]
 : The activate method has thrown an exception 
(java.lang.IllegalArgumentException: Service object cannot be null.)
java.lang.IllegalArgumentException: Service object cannot be null.
at org.apache.felix.framework.Felix.registerService(Felix.java:3748)
at 
org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at 
org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:302)
at 
org.apache.jackrabbit.oak.segment.remote.persistentcache.RemotePersistentCacheService.activate(RemotePersistentCacheService.java:63)
 [org.apache.jackrabbit.oak-segment-remote:1.37.0.R1885266]
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:242)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:678)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:524) 
[org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:318)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:308)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:342)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:115)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:984)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:957)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:766)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:667)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:433)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:666)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:353)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:553) 
[org.apache.felix.scr:2.1.20]
at org.apache.felix.scr.impl.Activator.access$200(Activator.java:69) 
[org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:424) 
[org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.AbstractExtender.createExtension(AbstractExtender.java:196)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:169)
 [org.apache.felix.scr:2.1.20]
at 
org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:49)
 [org.apache.felix.scr:2.1.20]
at 
org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:488)
at 
org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:420)
at 

[jira] [Comment Edited] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199265#comment-17199265
 ] 

Axel Hanikel edited comment on OAK-7932 at 9/21/20, 9:06 AM:
-

h2. Current status
The ZeroMQ NodeStore POC at 
https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore implements the 
idea of a Document-/NodeState-oriented node store with its own simple blob 
store backend instead of using mongo. The quickstart is coming up fine now and 
successfully renders some test pages. However, it is currently 2 - 3 times 
slower than the SegmentStore (or more when the caches are empty).

In order to get better performance we could now try to get the best of both 
worlds and store the NodeStates in segments, but only using the node, map, and 
list record types, and clearing the deduplication cache after writing a segment.

I currently think it is better to write just a single node state per segment 
rather than more than one but there is room for experiments here.


was (Author: ahanikel):
h2. Current status
The ZeroMQ NodeStore POC at 
https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore implements the 
idea of a Document-/NodeState-oriented node store with its own simple blob 
store backend instead of using mongo. However, it is currently 2 - 3 times 
slower than the SegmentStore (or more when the caches are empty).

In order to get better performance we could now try to get the best of both 
worlds and store the NodeStates in segments, but only using the node, map, and 
list record types, and clearing the deduplication cache after writing a segment.

I currently think it is better to write just a single node state per segment 
rather than more than one but there is room for experiments here.

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> The first idea was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> A second implementation, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy 

[jira] [Commented] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199265#comment-17199265
 ] 

Axel Hanikel commented on OAK-7932:
---

h2. Current status
The ZeroMQ NodeStore POC at 
https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore implements the 
idea of a Document-/NodeState-oriented node store with its own simple blob 
store backend instead of using mongo. However, it is currently 2 - 3 times 
slower than the SegmentStore (or more when the caches are empty).

In order to get better performance we could now try to get the best of both 
worlds and store the NodeStates in segments, but only using the node, map, and 
list record types, and clearing the deduplication cache after writing a segment.

I currently think it is better to write just a single node state per segment 
rather than more than one but there is room for experiments here.

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> *   Communication between actors (services) is done exclusively via messages.
> *   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
> *   Nodestates are kept in RAM and are written to external storage lazily 
> only for disaster recovery.
> *   A nodestate is identified by a uuid, which in turn is a hash on its 
> serialised string representation.
> *   As RAM is a very limited resource, different actors own their share of 
> the total uuid space.
> *   An actor might also cache a few nodestates which it does not own but 
> which it uses often (such as
>      the one containing the root node)
> h1. Implementation
> The first idea was to use the segment node store, and ZeroMQ for 
> communication because it seems to be a high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far. This approach is probably not
>  pursued because due to the nature of how things are stored in segments, they 
> are hard to cache (it seems like a large part of the repository
> will eventually end up in the cache).
> A second implementation, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> Here is what a serialised nodestate looks like:
> {noformat}
> begin ZeroMQNodeState
> begin children
> allow 856d1356-7054-3993-894b-a04426956a78
> end children
> begin properties
> jcr:primaryType  = rep:ACL
> :childOrder  = [allow]
> end properties
> end ZeroMQNodeState
> {noformat}
> This verbose format is good for debugging but expensive to generate and 
> parse, so it may be replaced with a binary format at some point. But it shows 
> how child nodestates are referenced and how properties are represented. 
> Binary properties (not shown here) are represented by a reference to the blob 
> store.
> The redundancy (compared with the segment store with its fine-grained record 
> structure) wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-9201) PersistentRedisCache: failure to write segment is not an error

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192075#comment-17192075
 ] 

Axel Hanikel commented on OAK-9201:
---

[~adulceanu] WDYT?

> PersistentRedisCache: failure to write segment is not an error
> --
>
> Key: OAK-9201
> URL: https://issues.apache.org/jira/browse/OAK-9201
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-azure
>Affects Versions: 1.34.0
>Reporter: Axel Hanikel
>Priority: Minor
>  Labels: patch
> Fix For: 1.36.0
>
> Attachments: 
> 0001-OAK-9201-PersistentRedisCache-failure-to-write-segme.patch
>
>
> Failure to write a segment to the redis cache results in an error level log 
> message with a stack trace. However, this is expected behaviour: socket 
> timeouts prevent the cache from effectively slowing down a request. OTOH too 
> many socket timeouts make the cache ineffective, so it's good to have a way 
> to log such errors when debugging. My suggestion is therefore to change the 
> log level to "debug" and avoid the stack trace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-9201) PersistentRedisCache: failure to write segment is not an error

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-9201:
--
Labels: patch  (was: )

> PersistentRedisCache: failure to write segment is not an error
> --
>
> Key: OAK-9201
> URL: https://issues.apache.org/jira/browse/OAK-9201
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-azure
>Affects Versions: 1.34.0
>Reporter: Axel Hanikel
>Priority: Minor
>  Labels: patch
> Fix For: 1.36.0
>
> Attachments: 
> 0001-OAK-9201-PersistentRedisCache-failure-to-write-segme.patch
>
>
> Failure to write a segment to the redis cache results in an error level log 
> message with a stack trace. However, this is expected behaviour: socket 
> timeouts prevent the cache from effectively slowing down a request. OTOH too 
> many socket timeouts make the cache ineffective, so it's good to have a way 
> to log such errors when debugging. My suggestion is therefore to change the 
> log level to "debug" and avoid the stack trace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-9201) PersistentRedisCache: failure to write segment is not an error

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-9201:
--
Attachment: 0001-OAK-9201-PersistentRedisCache-failure-to-write-segme.patch

> PersistentRedisCache: failure to write segment is not an error
> --
>
> Key: OAK-9201
> URL: https://issues.apache.org/jira/browse/OAK-9201
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-azure
>Affects Versions: 1.34.0
>Reporter: Axel Hanikel
>Priority: Minor
> Fix For: 1.36.0
>
> Attachments: 
> 0001-OAK-9201-PersistentRedisCache-failure-to-write-segme.patch
>
>
> Failure to write a segment to the redis cache results in an error level log 
> message with a stack trace. However, this is expected behaviour: socket 
> timeouts prevent the cache from effectively slowing down a request. OTOH too 
> many socket timeouts make the cache ineffective, so it's good to have a way 
> to log such errors when debugging. My suggestion is therefore to change the 
> log level to "debug" and avoid the stack trace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (OAK-9201) PersistentRedisCache: failure to write segment is not an error

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-9201:
-

 Summary: PersistentRedisCache: failure to write segment is not an 
error
 Key: OAK-9201
 URL: https://issues.apache.org/jira/browse/OAK-9201
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-azure
Affects Versions: 1.34.0
Reporter: Axel Hanikel
 Fix For: 1.36.0


Failure to write a segment to the redis cache results in an error level log 
message with a stack trace. However, this is expected behaviour: socket 
timeouts prevent the cache from effectively slowing down a request. OTOH too 
many socket timeouts make the cache ineffective, so it's good to have a way to 
log such errors when debugging. My suggestion is therefore to change the log 
level to "debug" and avoid the stack trace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

The first idea was to use the segment node store, and ZeroMQ for communication 
because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

Here is what a serialised nodestate looks like:
{noformat}
begin ZeroMQNodeState
begin children
allow   856d1356-7054-3993-894b-a04426956a78
end children
begin properties
jcr:primaryType  = rep:ACL
:childOrder  = [allow]
end properties
end ZeroMQNodeState
{noformat}

This verbose format is good for debugging but expensive to generate and parse, 
so it may be replaced with a binary format at some point. But it shows how 
child nodestates are referenced and how properties are represented. Binary 
properties (not shown here) are represented by a reference to the blob store.

The redundancy (compared with the segment store with its fine-grained record 
structure) wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 

  was:
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

The first idea was to use the segment node store, and ZeroMQ for communication 
because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore 

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

*   Communication between actors (services) is done exclusively via messages.
*   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
*   Nodestates are kept in RAM and are written to external storage lazily only 
for disaster recovery.
*   A nodestate is identified by a uuid, which in turn is a hash on its 
serialised string representation.
*   As RAM is a very limited resource, different actors own their share of the 
total uuid space.
*   An actor might also cache a few nodestates which it does not own but which 
it uses often (such as
     the one containing the root node)

h1. Implementation

The first idea was to use the segment node store, and ZeroMQ for communication 
because it seems to be a high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far. This approach is probably not
 pursued because due to the nature of how things are stored in segments, they 
are hard to cache (it seems like a large part of the repository
will eventually end up in the cache).

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

This redundancy wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 

  was:
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

-   Communication between actors (services) is done exclusively via messages.
 -   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
 -   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
 -   As RAM is a very limited resource, different actors own their share of the 
total segment space.
 -   An actor can also cache a few segments which it does not own but which it 
uses often (such as
     the one containing the root node)
 -   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment
     size would improve performance.
 -   We could even use the segment solely as an addressing component and 
operate at the record level.
     That would avoid copying data around when collecting garbage: garbage 
records would just be
     evicted from RAM.

h1. Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far,
 but I don't know about its performance yet.

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

This redundancy wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid 

[jira] [Updated] (OAK-7932) A distributed node store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Summary: A distributed node store for the cloud  (was: A distributed 
segment store for the cloud)

> A distributed node store for the cloud
> --
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> -   Communication between actors (services) is done exclusively via messages.
>  -   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
>  -   Segments are kept in RAM and are written to external storage lazily only 
> for disaster recovery.
>  -   As RAM is a very limited resource, different actors own their share of 
> the total segment space.
>  -   An actor can also cache a few segments which it does not own but which 
> it uses often (such as
>      the one containing the root node)
>  -   The granularity of operating on whole segments may be too coarse, so 
> perhaps reducing the segment
>      size would improve performance.
>  -   We could even use the segment solely as an addressing component and 
> operate at the record level.
>      That would avoid copying data around when collecting garbage: garbage 
> records would just be
>      evicted from RAM.
> h1. Implementation
> The first idea was to use ZeroMQ for communication because it seems to be a 
> high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far,
>  but I don't know about its performance yet.
> A second implementation, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> This redundancy wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147784#comment-17147784
 ] 

Axel Hanikel commented on OAK-7932:
---

h2. Current status

* All the integration tests (oak-it) pass with {{-Dnsfixtures=ZEROMQ_NS}}
* Rebased everything on oak 1.30.0

I've also tested running a classic AEM/CQ quickstart with the ZeroMQ nodestore. 
When run, it happily but slowly installs packages, but it gets slower and 
slower over time, and I haven't found out why yet. In order to better 
understand what happens, I've reimplemented the NodeBuilder (the old 
implementation ZeroMQNodeBuilder was based on the MemoryNodeBuilder, the new 
one ZeroMQBuilder implements the interface directly). Unfortunately, the new 
implementation again leads to failures in the integration tests.

So in order to see {{oak-it}} pass, check out 
https://github.com/ahanikel/jackrabbit-oak/commit/919043394298e755cc0d051389a232d9f98a281b
 (before the Builder reimplementation), mvn clean install everything, and do 
{{mvn clean test -Dnsfixtures=ZEROMQ_NS}} in {{oak-it}}

Note that the version is set to 1.30.0 (not SNAPSHOT), so when you mvn install 
things, {{oak-commons}} (and perhaps others) will be overwritten in your local 
maven repo.



> A distributed segment store for the cloud
> -
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> -   Communication between actors (services) is done exclusively via messages.
>  -   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
>  -   Segments are kept in RAM and are written to external storage lazily only 
> for disaster recovery.
>  -   As RAM is a very limited resource, different actors own their share of 
> the total segment space.
>  -   An actor can also cache a few segments which it does not own but which 
> it uses often (such as
>      the one containing the root node)
>  -   The granularity of operating on whole segments may be too coarse, so 
> perhaps reducing the segment
>      size would improve performance.
>  -   We could even use the segment solely as an addressing component and 
> operate at the record level.
>      That would avoid copying data around when collecting garbage: garbage 
> records would just be
>      evicted from RAM.
> h1. Implementation
> The first idea was to use ZeroMQ for communication because it seems to be a 
> high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far,
>  but I don't know about its performance yet.
> A second implementation, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID (a hash of the serialised
> nodestate) and is saved together with its properties, similar to the document 
> node store.
> This redundancy wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

-   Communication between actors (services) is done exclusively via messages.
 -   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
 -   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
 -   As RAM is a very limited resource, different actors own their share of the 
total segment space.
 -   An actor can also cache a few segments which it does not own but which it 
uses often (such as
     the one containing the root node)
 -   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment
     size would improve performance.
 -   We could even use the segment solely as an addressing component and 
operate at the record level.
     That would avoid copying data around when collecting garbage: garbage 
records would just be
     evicted from RAM.

h1. Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far,
 but I don't know about its performance yet.

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID (a hash of the serialised
nodestate) and is saved together with its properties, similar to the document 
node store.

This redundancy wastes space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 

  was:
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

-   Communication between actors (services) is done exclusively via messages.
 -   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
 -   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
 -   As RAM is a very limited resource, different actors own their share of the 
total segment space.
 -   An actor can also cache a few segments which it does not own but which it 
uses often (such as
     the one containing the root node)
 -   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment
     size would improve performance.
 -   We could even use the segment solely as an addressing component and 
operate at the record level.
     That would avoid copying data around when collecting garbage: garbage 
records would just be
     evicted from RAM.
h1. Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far,
 but I don't know about its performance yet.

There is an attempt to use jeromq in the segment store in a very very very 
early stage at
 [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq] . It is based on the 
memory segment store
 and currently just replaces direct function calls for reading and writing 
segments with messages being
 sent and received.

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The 

[jira] [Commented] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025798#comment-17025798
 ] 

Axel Hanikel commented on OAK-8878:
---

Thank you! We could add some SuppressWarnings annotations:  [^OAK-8878-2.patch] 
but the tests worked fine for me as well. Could you commit the change (I don't 
have the rights to do so)?

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Assignee: Marcel Reutegger
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch, OAK-8878-2.patch, OAK-8878.patch, 
> Screenshot 2020-01-28 at 13.56.47 .png
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-8878:
--
Attachment: OAK-8878-2.patch

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Assignee: Marcel Reutegger
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch, OAK-8878-2.patch, OAK-8878.patch, 
> Screenshot 2020-01-28 at 13.56.47 .png
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025095#comment-17025095
 ] 

Axel Hanikel edited comment on OAK-8878 at 1/28/20 1:43 PM:


Attaching a screenshot which shows the size and time totals after applying the 
patch:  !Screenshot 2020-01-28 at 13.56.47 .png! 
The lines from top to bottom are:
* The first two lines are the download size total for pod 1 and 2.
* The red line is the total time in microseconds for pod 1.
* The next line is the download rate in bytes / sec for pod 2.
* The green line is the total time in microseconds for pod 2.
* The mauve line is the download rate in bytes / sec for pod 1.


was (Author: ahanikel):
Attaching a screenshot which shows the size and time totals after applying the 
patch:  !Screenshot 2020-01-28 at 13.56.47 .png! 
The lines from top to bottom are:
* The first two lines are the download size total for pod 1 and 2.
* The red line is the total time in microseconds for pod 1.
* The next line is the download rate in bytes / sec for pod 2.
* The green line is the total time in microseconds for pod 2.

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch, Screenshot 2020-01-28 at 13.56.47 
> .png
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025095#comment-17025095
 ] 

Axel Hanikel commented on OAK-8878:
---

Attaching a screenshot which shows the size and time totals after applying the 
patch:  !Screenshot 2020-01-28 at 13.56.47 .png! 
The lines from top to bottom are:
* The first two lines are the download size total for pod 1 and 2.
* The red line is the total time in microseconds for pod 1.
* The next line is the download rate in bytes / sec for pod 2.
* The green line is the total time in microseconds for pod 2.

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch, Screenshot 2020-01-28 at 13.56.47 
> .png
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-8878:
--
Attachment: Screenshot 2020-01-28 at 13.56.47 .png

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch, Screenshot 2020-01-28 at 13.56.47 
> .png
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-8878:
--
Attachment: (was: Screenshot 2020-01-28 at 13.52.41 .png)

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-8878:
--
Attachment: Screenshot 2020-01-28 at 13.52.41 .png

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch, Screenshot 2020-01-28 at 13.52.41 
> .png
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-8878:
--
Description: It would be useful to me to have the downloaded and uploaded 
blob bytes and times exposed as a metric and not only as a time series. This 
helps getting meaningful statistics e.g. in the case of Prometheus, which 
scrapes values at some interval:  [^GRANITE-28652-1.patch]   (was: It would be 
useful to me to have the downloaded and uploaded blob bytes and times exposed 
as a metric and not only as a time series. This helps when recording meaningful 
statistics e.g. in the case of Prometheus, which scrapes values at some 
interval:  [^GRANITE-28652-1.patch] )

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps getting 
> meaningful statistics e.g. in the case of Prometheus, which scrapes values at 
> some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025075#comment-17025075
 ] 

Axel Hanikel commented on OAK-8878:
---

[~mreutegg] WDYT?

> Expose downloaded (and uploaded) blob bytes as metrics, not only as time 
> series
> ---
>
> Key: OAK-8878
> URL: https://issues.apache.org/jira/browse/OAK-8878
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-plugins
>Affects Versions: 1.24.0
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.26.0
>
> Attachments: GRANITE-28652-1.patch
>
>
> It would be useful to me to have the downloaded and uploaded blob bytes and 
> times exposed as a metric and not only as a time series. This helps when 
> recording meaningful statistics e.g. in the case of Prometheus, which scrapes 
> values at some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (OAK-8878) Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

[Axel Hanikel (Jira)]

Axel Hanikel created OAK-8878:
-

 Summary: Expose downloaded (and uploaded) blob bytes as metrics, 
not only as time series
 Key: OAK-8878
 URL: https://issues.apache.org/jira/browse/OAK-8878
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: blob-plugins
Affects Versions: 1.24.0
Reporter: Axel Hanikel
 Fix For: 1.26.0
 Attachments: GRANITE-28652-1.patch

It would be useful to me to have the downloaded and uploaded blob bytes and 
times exposed as a metric and not only as a time series. This helps when 
recording meaningful statistics e.g. in the case of Prometheus, which scrapes 
values at some interval:  [^GRANITE-28652-1.patch] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

-   Communication between actors (services) is done exclusively via messages.
 -   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
 -   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
 -   As RAM is a very limited resource, different actors own their share of the 
total segment space.
 -   An actor can also cache a few segments which it does not own but which it 
uses often (such as
     the one containing the root node)
 -   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment
     size would improve performance.
 -   We could even use the segment solely as an addressing component and 
operate at the record level.
     That would avoid copying data around when collecting garbage: garbage 
records would just be
     evicted from RAM.
h1. Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far,
 but I don't know about its performance yet.

There is an attempt to use jeromq in the segment store in a very very very 
early stage at
 [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq] . It is based on the 
memory segment store
 and currently just replaces direct function calls for reading and writing 
segments with messages being
 sent and received.

A second implementation, at 
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
nodestore implementation which is kind of a dual to the segment store in the 
sense that it is on the other end
of the compactness spectrum. The segment store is very dense and avoids 
duplication whereever possible.
The nodestore in this implementation, however, is quite redundant: Every 
nodestate gets its own UUID and is saved together
with its properties, similar to the document node store. This redundancy wastes 
space, but on the other hand garbage
collection (yet unimplemented) is easier because there is no segment that needs 
to be rewritten to get rid of data that is no
longer referenced; unreferenced nodes can just be deleted. This implementation 
still has bugs, but being much simpler
than the segment store, it can eventually be used to experiment with different 
configurations and examine their
performance.

 

  was:
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

-   Communication between actors (services) is done exclusively via messages.
 -   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
 -   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
 -   As RAM is a very limited resource, different actors own their share of the 
total segment space.
 -   An actor can also cache a few segments which it does not own but which it 
uses often (such as
     the one containing the root node)
 -   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment
    size would improve performance.
 -   We could even use the segment solely as an addressing component and 
operate at the record level.
     That would avoid copying data around when collecting garbage: garbage 
records would just be
     evicted from RAM.
h1. Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far,
but I don't know about its performance yet.

There is an attempt to use jeromq in the segment store in a very very very 
early stage at
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq] . It is based on the 
memory segment store
and currently just replaces direct function calls for reading and writing 
segments with messages being
sent and received.


> A distributed segment store for the cloud
> 

[jira] [Commented] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962800#comment-16962800
 ] 

Axel Hanikel commented on OAK-7932:
---

Added reference to second implementation to description.

> A distributed segment store for the cloud
> -
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> -   Communication between actors (services) is done exclusively via messages.
>  -   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
>  -   Segments are kept in RAM and are written to external storage lazily only 
> for disaster recovery.
>  -   As RAM is a very limited resource, different actors own their share of 
> the total segment space.
>  -   An actor can also cache a few segments which it does not own but which 
> it uses often (such as
>      the one containing the root node)
>  -   The granularity of operating on whole segments may be too coarse, so 
> perhaps reducing the segment
>      size would improve performance.
>  -   We could even use the segment solely as an addressing component and 
> operate at the record level.
>      That would avoid copying data around when collecting garbage: garbage 
> records would just be
>      evicted from RAM.
> h1. Implementation
> The first idea was to use ZeroMQ for communication because it seems to be a 
> high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far,
>  but I don't know about its performance yet.
> There is an attempt to use jeromq in the segment store in a very very very 
> early stage at
>  [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq] . It is based on 
> the memory segment store
>  and currently just replaces direct function calls for reading and writing 
> segments with messages being
>  sent and received.
> A second implementation, at 
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq-nodestore] is a simple
> nodestore implementation which is kind of a dual to the segment store in the 
> sense that it is on the other end
> of the compactness spectrum. The segment store is very dense and avoids 
> duplication whereever possible.
> The nodestore in this implementation, however, is quite redundant: Every 
> nodestate gets its own UUID and is saved together
> with its properties, similar to the document node store. This redundancy 
> wastes space, but on the other hand garbage
> collection (yet unimplemented) is easier because there is no segment that 
> needs to be rewritten to get rid of data that is no
> longer referenced; unreferenced nodes can just be deleted. This 
> implementation still has bugs, but being much simpler
> than the segment store, it can eventually be used to experiment with 
> different configurations and examine their
> performance.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-8613) Azure Segment Store clustering PoC

[Axel Hanikel (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926586#comment-16926586
 ] 

Axel Hanikel commented on OAK-8613:
---

Cool! :) Do I understand correctly that changes are written to the private repo 
first (so they succeed very quickly), and then synced with the shared repo, 
where the shared repo requests a rebase if needed? If so, the private 
repositories differ from the shared repo: not by their content, but they have 
different segments, right? (Not that I think that's a problem, I just want to 
make sure I understand the mechanism)

> Azure Segment Store clustering PoC
> --
>
> Key: OAK-8613
> URL: https://issues.apache.org/jira/browse/OAK-8613
> Project: Jackrabbit Oak
>  Issue Type: Story
>  Components: segment-azure
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
>Priority: Major
> Attachments: OAK-8613.patch, remote node store.png
>
>
> Azure Segment Store offers a way to read the same segments, concurrently in 
> many Oak instances. With a way to coordinate writes, it's possible to 
> implement a distributed node store based on the SegmentNS. The solution will 
> consist of following elements:
> * a central server, that'd coordinate the writes to the shared repository,
> * a number of clients, that can read directly from the shared repository. 
> They also have their own, private repositories within the same cloud storage,
> * the shared repository, which represents the current state. It can be read 
> by anyone, but only the central server can write it,
> * the private repositories. Clients can write their own segments there and 
> then pass their references to the server.
> As above, every client uses two repositories: shared (in the read only mode) 
> and private (in read-write mode). When a client wants to read the current 
> root, it asks the server for its revision. Then, it looks in the shared 
> repository, reads the segment and creates a node state.
> If the client wants to modify the repository, it convert the fetched node 
> state into node builder. The applied changes will eventually be serialized in 
> the new segment, in the client's private repository. In order to merge it 
> changes, the client will send two revision ids: of the base root (fetched 
> from shared repo) and of the current root (stored in the private repo). 
> Server will check if the base root wasn't updated in the meantime - in such 
> case it requests a rebase. Otherwise, it'll read the current root from the 
> private repository, apply the commit changes in the shared repository and 
> update the journal.
> gRPC is used for the communication between the server and clients. This is 
> used only for the coordination. All the data are actually exchanged via 
> segment stores.
>  !remote node store.png|width=100%! 
> The attached [^OAK-8613.patch] contains the implementation split into 3 parts:
> * oak-store-remote-commons, which contains the gRPC descriptors of used 
> services and embeds the required libraries,
> * oak-store-remote-server builds an executable JAR, that starts the server,
> * oak-store-remote-client is an OSGi bundle that starts NodeStore connecting 
> to the configured server and Azure Storage.
> There are also some changes in the oak-segment-tar - new SPIs that allows to 
> read segments with their revisions (record ids) and exposes the revision in 
> the node state.
> The gRPC uses Guava 26. I was able to somehow get it running with other Oak 
> bundles, using Guava 15, but if we want to produtize it, we'd need to update 
> Oak's Guava.
> There's a new fixture that tests the implementation. It can be run with:
> {noformat}
> mvn clean install -f oak-it/pom.xml -Dnsfixtures=REMOTE -Dtest=NodeStoreTest 
> -Dtest.opts=-Xmx4g
> {noformat}
> This is prototype. It lacks of the tests and important resilience features:
> * unit tests, especially for the discovery lite implementation,
> * enable online compaction for the shared repository (the private ones can be 
> just thrown away after merge),
> * server resilience and support disconnecting clients in a clean way (eg. 
> unregister node observers),
> * client resilience, with support to re-connect,
> * clean resources on both sides on disconnection (eg. remove private 
> repository).
> Potential improvements:
> * can we have multiple replicas for the server, in the active-passive mode, 
> to increase resilience?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8515) Make the Azure Persistence timeouts configurable

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896972#comment-16896972
 ] 

Axel Hanikel commented on OAK-8515:
---

IMHO giving up after retrying a few times is bad: when the segment store is 
trying to read its journal file it should try to do so no matter how long it 
takes. It is not up to the segment store to decide how long is too much, any 
value is kind of arbitrary. If it takes a long time the segment store should 
issue a warning from time to time, but if it throws it's becoming the segment 
store's fault, not azure's or the network's or whatever.

> Make the Azure Persistence timeouts configurable
> 
>
> Key: OAK-8515
> URL: https://issues.apache.org/jira/browse/OAK-8515
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: segment-azure
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: OAK-8515.patch
>
>
> OAK-8406 introduced timeout for the server-side execution in Azure cloud. 
> This may cause issues like this:
> {noformat}
> Exception in thread "main" java.util.NoSuchElementException: An error 
> occurred while enumerating the result, check the original exception for 
> details.
> at 
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
> at java.util.Iterator.forEachRemaining(Iterator.java:115)
> at 
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at 
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at 
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at 
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at 
> org.apache.jackrabbit.oak.segment.azure.AzureSegmentArchiveReader.(AzureSegmentArchiveReader.java:61)
> at 
> org.apache.jackrabbit.oak.segment.azure.AzureArchiveManager.forceOpen(AzureArchiveManager.java:103)
> at 
> org.apache.jackrabbit.oak.segment.azure.tool.SegmentStoreMigrator.migrateArchives(SegmentStoreMigrator.java:149)
> at 
> org.apache.jackrabbit.oak.segment.azure.tool.SegmentStoreMigrator.migrate(SegmentStoreMigrator.java:87)
> [...]
> Caused by: com.microsoft.azure.storage.StorageException: The client could not 
> finish the operation within specified maximum execution timeout.
> at 
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:233)
> at 
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> ... 14 more
> Caused by: java.util.concurrent.TimeoutException: The client could not finish 
> the operation within specified maximum execution timeout.
> at 
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:232)
> ... 15 more
> {noformat}
> Let's make the timeouts configurable.
> //cc: [~frm], [~ierandra]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (OAK-8094) JMX monitoring to detect commits carrying over from previous GC generation can block other threads from committing

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780632#comment-16780632
 ] 

Axel Hanikel commented on OAK-8094:
---

[~mduerig] perhaps we could make the link between the otherWritersLimit 
parameter in the CommitsTracker, the number of iterations in the for loop, and 
the min value more clear as in 
[https://github.com/ahanikel/jackrabbit-oak/commit/c9b20741d2c12b9360da8352d212b116956ba721]
 - otherwise I'd say: nihil obstat :)

> JMX monitoring to detect commits carrying over from previous GC generation 
> can block other threads from committing
> --
>
> Key: OAK-8094
> URL: https://issues.apache.org/jira/browse/OAK-8094
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Major
> Fix For: 1.12
>
>
> This is a follow up of OAK-8071 to implement JMX monitoring to detect the 
> situation from OAK-8014.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8071) Logging to detect commits carrying over from previous GC generation can block other threads from committing

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779385#comment-16779385
 ] 

Axel Hanikel commented on OAK-8071:
---

[~mduerig] Sorry for the delay, I had to deal with some upgrade-related issues. 
The patch looks good to me.

> Logging to detect commits carrying over from previous GC generation can block 
> other threads from committing
> ---
>
> Key: OAK-8071
> URL: https://issues.apache.org/jira/browse/OAK-8071
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
> Fix For: 1.12, 1.11.0, 1.8.12, 1.10.2
>
>
> Add logging / monitoring to detect the situation from OAK-8014.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8071) Logging to detect commits carrying over from previous GC generation can block other threads from committing

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777917#comment-16777917
 ] 

Axel Hanikel commented on OAK-8071:
---

[~mduerig] I agree with what you say about the gc generation. For the 
CommitTracker, perhaps we can add a comment like 
[https://github.com/ahanikel/jackrabbit-oak/commits/OAK-8071-2] to make clear 
that the CommitTracker's dequeue and execute methods must now be used in 
sequence within the same critical section?

> Logging to detect commits carrying over from previous GC generation can block 
> other threads from committing
> ---
>
> Key: OAK-8071
> URL: https://issues.apache.org/jira/browse/OAK-8071
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
> Fix For: 1.12, 1.11.0, 1.8.12, 1.10.2
>
>
> Add logging / monitoring to detect the situation from OAK-8014.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8071) Logging to detect commits carrying over from previous GC generation can block other threads from committing

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777050#comment-16777050
 ] 

Axel Hanikel commented on OAK-8071:
---

[~mduerig] I've made a few changes in 
[https://github.com/ahanikel/jackrabbit-oak/commits/OAK-8071] :
 * Cosmetic change: Renamed variable "w" to "k" because I couldn't figure out 
what "w" stands for ;)
 * I think {{currentCommit}} should be volatile because it is probably called 
from a different thread via {{#getCurrentWriter}} (although it's probably not a 
big issue).
 * I've tried to add the full GC generation to the {{CommitsTracker}} (but it's 
not reflected in the {{SegmentNodeStoreStats}} yet). Perhaps you can quickly 
check if what I did makes any sense...

Next steps:
 * Add the generation to the {{SegmentNodeStoreStats}}.
 * The {{CommitsTrackerTest}} (both tests) are failing for me. I'm going to 
figure out why.

> Logging to detect commits carrying over from previous GC generation can block 
> other threads from committing
> ---
>
> Key: OAK-8071
> URL: https://issues.apache.org/jira/browse/OAK-8071
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
> Fix For: 1.12, 1.11.0, 1.8.12, 1.10.2
>
>
> Add logging / monitoring to detect the situation from OAK-8014.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8069) Log warning for too many transient modifications of direct child nodes

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775062#comment-16775062
 ] 

Axel Hanikel commented on OAK-8069:
---

[~mduerig] Sorry for the stupid question, I was a bit out of context :) After 
looking at it again (and I hope this is a more helpful remark), could the 
problem stem from the recursive {{writeNode}} call in 
{{ChildNodeCollectorDiff}}? Not sure but it looks to me like this keeps the map 
and all the maps of the children "open" until the deepest nodes have been 
written.

> Log warning for too many transient modifications of direct child nodes
> --
>
> Key: OAK-8069
> URL: https://issues.apache.org/jira/browse/OAK-8069
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
>  Labels: TarMK
> Fix For: 1.12, 1.11.0, 1.10.1, 1.8.12
>
>
> In a first step towards resolving OAK-8066, I want to add some logging 
> regarding the number of transiently modified direct child nodes in 
> {{DefaultSegmentWriter}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-8069) Log warning for too many transient modifications of direct child nodes

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774046#comment-16774046
 ] 

Axel Hanikel edited comment on OAK-8069 at 2/21/19 12:57 PM:
-

[~mduerig] That looks good to me. Perhaps we could use some sort of a linked 
list for the path and defer the string concatenations to the 
{{warnOnManyChildren}} function?

What's the purpose of the childNodes map?


was (Author: ahanikel):
[~mduerig] That looks good to me. Perhaps we could use some sort of a linked 
list for the path and defer the string concatenations to the 
{{warnOnManyChildren}} function?

> Log warning for too many transient modifications of direct child nodes
> --
>
> Key: OAK-8069
> URL: https://issues.apache.org/jira/browse/OAK-8069
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Major
>  Labels: TarMK
>
> In a first step towards resolving OAK-8066, I want to add some logging 
> regarding the number of transiently modified direct child nodes in 
> {{DefaultSegmentWriter}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8069) Log warning for too many transient modifications of direct child nodes

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774046#comment-16774046
 ] 

Axel Hanikel commented on OAK-8069:
---

[~mduerig] That looks good to me. Perhaps we could use some sort of a linked 
list for the path and defer the string concatenations to the 
{{warnOnManyChildren}} function?

> Log warning for too many transient modifications of direct child nodes
> --
>
> Key: OAK-8069
> URL: https://issues.apache.org/jira/browse/OAK-8069
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Major
>  Labels: TarMK
>
> In a first step towards resolving OAK-8066, I want to add some logging 
> regarding the number of transiently modified direct child nodes in 
> {{DefaultSegmentWriter}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8014) Commits carrying over from previous GC generation can block other threads from committing

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771153#comment-16771153
 ] 

Axel Hanikel commented on OAK-8014:
---

[~mduerig] Don't we have to also check the underlying lock in 
LockFixture#assertUnlock? As in 
https://github.com/ahanikel/jackrabbit-oak/commit/bfe93a0d922c8942209da1ab1d25f700477e1deb

> Commits carrying over from previous GC generation can block other threads 
> from committing
> -
>
> Key: OAK-8014
> URL: https://issues.apache.org/jira/browse/OAK-8014
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Affects Versions: 1.10.0, 1.8.11
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
>  Labels: TarMK
> Fix For: 1.12, 1.11.0, 1.8.12
>
> Attachments: OAK-8014.patch
>
>
> A commit that is based on a previous (full) generation can block other 
> commits from progressing for a long time. This happens because such a commit 
> will do a deep copy of its state to avoid linking to old segments (see 
> OAK-3348). Most of the deep copying is usually avoided by the deduplication 
> caches. However, in cases where the cache hit rate is not good enough we have 
> seen deep copy operations up to several minutes. Sometimes this deep copy 
> operation happens inside the commit lock of 
> {{LockBasedScheduler.schedule()}}, which then causes all other commits to 
> become blocked.
> cc [~rma61...@adobe.com], [~edivad]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8014) Commits carrying over from previous GC generation can block other threads from committing

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765848#comment-16765848
 ] 

Axel Hanikel commented on OAK-8014:
---

[~mduerig] I think the comments are very clear and document the reasoning that 
has lead to the way things are implemented. Perhaps we could add the term 
"rebase" to the description in {{writeAhead}}, making it even more obvious why 
a commit might take such a long time: 
https://github.com/ahanikel/jackrabbit-oak/commit/9f6873290433a49f50f50a46e0cd8831a705ade5

> Commits carrying over from previous GC generation can block other threads 
> from committing
> -
>
> Key: OAK-8014
> URL: https://issues.apache.org/jira/browse/OAK-8014
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Affects Versions: 1.10.0, 1.8.11
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
>  Labels: TarMK
> Fix For: 1.12, 1.11.0, 1.8.12
>
> Attachments: OAK-8014.patch
>
>
> A commit that is based on a previous (full) generation can block other 
> commits from progressing for a long time. This happens because such a commit 
> will do a deep copy of its state to avoid linking to old segments (see 
> OAK-3348). Most of the deep copying is usually avoided by the deduplication 
> caches. However, in cases where the cache hit rate is not good enough we have 
> seen deep copy operations up to several minutes. Sometimes this deep copy 
> operation happens inside the commit lock of 
> {{LockBasedScheduler.schedule()}}, which then causes all other commits to 
> become blocked.
> cc [~rma61...@adobe.com], [~edivad]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-8033) Node states sometimes refer to more than a single generation of segments after a full compaction

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763496#comment-16763496
 ] 

Axel Hanikel commented on OAK-8033:
---

[~mduerig] the fix looks good to me. In particular I like the elegant test, and 
I'll happily reuse the idea when trying to come up with a test for OAK-8014!

> Node states sometimes refer to more than a single generation of segments 
> after a full compaction
> 
>
> Key: OAK-8033
> URL: https://issues.apache.org/jira/browse/OAK-8033
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: 1.10.0, 1.8.10, 1.6.16, 1.8.11, 1.10
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Major
>  Labels: TarMK, candidate_oak_1_10, candidate_oak_1_6, 
> candidate_oak_1_8
>
> Due to a regression introduced with OAK-7867 a full compaction can sometimes 
> cause nodes that are written concurrently to reference segments from more 
> than a single gc generation.
> This happens when the {{borrowWriter}} method needs to [create a new 
> writer|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBufferWriterPool.java#L197-L201].
>  In this case the new writer will be of the generation of the current head 
> state instead of the generation associated with the current write operation 
> in progress.
>  
> cc [~frm], [~ahanikel]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711389#comment-16711389
 ] 

Axel Hanikel commented on OAK-7932:
---

[~mduerig] Yes, I think we were talking about this during the September 
Oakathon.

The "root" socket does not publish the root segment, but the RecordId of the 
head - basically what would normally be written to the journal.log. Probably I 
should have named it differently: {{headWriterSocket}} or 
{{journalWriterSocket}}.

> A distributed segment store for the cloud
> -
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> -   Communication between actors (services) is done exclusively via messages.
>  -   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
>  -   Segments are kept in RAM and are written to external storage lazily only 
> for disaster recovery.
>  -   As RAM is a very limited resource, different actors own their share of 
> the total segment space.
>  -   An actor can also cache a few segments which it does not own but which 
> it uses often (such as
>      the one containing the root node)
>  -   The granularity of operating on whole segments may be too coarse, so 
> perhaps reducing the segment
>     size would improve performance.
>  -   We could even use the segment solely as an addressing component and 
> operate at the record level.
>      That would avoid copying data around when collecting garbage: garbage 
> records would just be
>      evicted from RAM.
> h1. Implementation
> The first idea was to use ZeroMQ for communication because it seems to be a 
> high-quality and
>  easy to use implementation. A major drawback is that the library is written 
> in C and the Java
>  library which does the JNI stuff seems hard to set up and did not work for 
> me. There is a native
>  Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
> to work well so far,
> but I don't know about its performance yet.
> There is an attempt to use jeromq in the segment store in a very very very 
> early stage at
> [https://github.com/ahanikel/jackrabbit-oak/tree/zeromq] . It is based on the 
> memory segment store
> and currently just replaces direct function calls for reading and writing 
> segments with messages being
> sent and received.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7932:
--
Description: 
h1. Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
 distributed environment. The main idea is to adopt an actor-like model, 
meaning:

-   Communication between actors (services) is done exclusively via messages.
 -   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
     state with other actors as far as possible.
 -   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
 -   As RAM is a very limited resource, different actors own their share of the 
total segment space.
 -   An actor can also cache a few segments which it does not own but which it 
uses often (such as
     the one containing the root node)
 -   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment
    size would improve performance.
 -   We could even use the segment solely as an addressing component and 
operate at the record level.
     That would avoid copying data around when collecting garbage: garbage 
records would just be
     evicted from RAM.
h1. Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
 easy to use implementation. A major drawback is that the library is written in 
C and the Java
 library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
 Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems 
to work well so far,
but I don't know about its performance yet.

There is an attempt to use jeromq in the segment store in a very very very 
early stage at
[https://github.com/ahanikel/jackrabbit-oak/tree/zeromq] . It is based on the 
memory segment store
and currently just replaces direct function calls for reading and writing 
segments with messages being
sent and received.

  was:
# Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
distributed environment. The main idea is to adopt an actor-like model, meaning:

-   Communication between actors (services) is done exclusively via messages.
-   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
    state with other actors as far as possible.
-   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
-   As RAM is a very limited resource, different actors own their share of the 
total segment space.
-   An actor can also cache a few segments which it does not own but which it 
uses often (such as
    the one containing the root node)
-   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment size would improve performance.
-   We could even use the segment solely as an addressing component and operate 
at the record level.
    That would avoid copying data around when collecting garbage: garbage 
records would just be
    evicted from RAM.

# Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
easy to use implementation. A major drawback is that the library is written in 
C and the Java
library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems to 
work well so far, but I don't know about its performance yet.

There is an attempt to use jeromq in the segment store in a very very very 
early stage at
 . It is based on the 
memory segment store and currently just replaces direct function calls for 
reading and writing segments with messages being sent and received.


> A distributed segment store for the cloud
> -
>
> Key: OAK-7932
> URL: https://issues.apache.org/jira/browse/OAK-7932
> Project: Jackrabbit Oak
>  Issue Type: Wish
>  Components: segment-tar
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
>
> h1. Outline
> This issue documents some proof-of-concept work for adapting the segment tar 
> nodestore to a
>  distributed environment. The main idea is to adopt an actor-like model, 
> meaning:
> -   Communication between actors (services) is done exclusively via messages.
>  -   An actor (which could also be a thread) processes one message at a time, 
> avoiding sharing
>      state with other actors as far as possible.
>  -   Segments are kept in RAM and are written to external storage lazily only 
> for disaster recovery.
>  -   As RAM is a very limited resource, different actors own their share of 
> the total 

[jira] [Created] (OAK-7932) A distributed segment store for the cloud

[Axel Hanikel (JIRA)]

Axel Hanikel created OAK-7932:
-

 Summary: A distributed segment store for the cloud
 Key: OAK-7932
 URL: https://issues.apache.org/jira/browse/OAK-7932
 Project: Jackrabbit Oak
  Issue Type: Wish
  Components: segment-tar
Reporter: Axel Hanikel
Assignee: Axel Hanikel


# Outline

This issue documents some proof-of-concept work for adapting the segment tar 
nodestore to a
distributed environment. The main idea is to adopt an actor-like model, meaning:

-   Communication between actors (services) is done exclusively via messages.
-   An actor (which could also be a thread) processes one message at a time, 
avoiding sharing
    state with other actors as far as possible.
-   Segments are kept in RAM and are written to external storage lazily only 
for disaster recovery.
-   As RAM is a very limited resource, different actors own their share of the 
total segment space.
-   An actor can also cache a few segments which it does not own but which it 
uses often (such as
    the one containing the root node)
-   The granularity of operating on whole segments may be too coarse, so 
perhaps reducing the segment size would improve performance.
-   We could even use the segment solely as an addressing component and operate 
at the record level.
    That would avoid copying data around when collecting garbage: garbage 
records would just be
    evicted from RAM.

# Implementation

The first idea was to use ZeroMQ for communication because it seems to be a 
high-quality and
easy to use implementation. A major drawback is that the library is written in 
C and the Java
library which does the JNI stuff seems hard to set up and did not work for me. 
There is a native
Java implementation of the ZeroMQ protocol, aptly called jeromq, which seems to 
work well so far, but I don't know about its performance yet.

There is an attempt to use jeromq in the segment store in a very very very 
early stage at
 . It is based on the 
memory segment store and currently just replaces direct function calls for 
reading and writing segments with messages being sent and received.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697227#comment-16697227
 ] 

Axel Hanikel commented on OAK-7878:
---

[~frm] I've added some tests in 
[^0001-OAK-7878-Add-more-unit-tests-for-LoggingHook.patch] and I think it 
covers most positive cases. I can't think of any negative test cases except 
passing in null values to all the public methods, but that doesn't seem useful 
to me.

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
> Fix For: 1.10
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch, 
> 0001-OAK-7878-Add-LoggingHookTest.patch, 
> 0001-OAK-7878-Add-more-unit-tests-for-LoggingHook.patch, OAK-7878-02.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7878:
--
Attachment: 0001-OAK-7878-Add-more-unit-tests-for-LoggingHook.patch

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Assignee: Axel Hanikel
>Priority: Minor
> Fix For: 1.10
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch, 
> 0001-OAK-7878-Add-LoggingHookTest.patch, 
> 0001-OAK-7878-Add-more-unit-tests-for-LoggingHook.patch, OAK-7878-02.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679728#comment-16679728
 ] 

Axel Hanikel commented on OAK-7878:
---

[~frm] is that what you meant? Regarding the additional "n!": this is an "up" 
operation on the root node, which is equivalent of saying "commit" or "end of 
transaction". The root node is not logged as {{n^ /}} or similar, which is 
arguably an asymmetry. OTOH it would not add any information, so I thought I 
would save the bytes.

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Fix For: 1.9.11
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch, 
> 0001-OAK-7878-Add-LoggingHookTest.patch, OAK-7878-02.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7878:
--
Attachment: 0001-OAK-7878-Add-LoggingHookTest.patch

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Fix For: 1.9.11
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch, 
> 0001-OAK-7878-Add-LoggingHookTest.patch, OAK-7878-02.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673163#comment-16673163
 ] 

Axel Hanikel commented on OAK-7878:
---

[~frm] could you have a look? Thanks!

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Priority: Minor
> Fix For: 1.9.11
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7878:
--
Priority: Minor  (was: Major)

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Priority: Minor
> Fix For: 1.9.11
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7878:
--
Attachment: 0001-OAK-7878-Add-LoggingHookTest.patch

> Add unit test for LoggingHook
> -
>
> Key: OAK-7878
> URL: https://issues.apache.org/jira/browse/OAK-7878
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: segment-tar
>Affects Versions: 1.9.10
>Reporter: Axel Hanikel
>Priority: Major
> Fix For: 1.9.11
>
> Attachments: 0001-OAK-7878-Add-LoggingHookTest.patch
>
>
> Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OAK-7878) Add unit test for LoggingHook

[Axel Hanikel (JIRA)]

Axel Hanikel created OAK-7878:
-

 Summary: Add unit test for LoggingHook
 Key: OAK-7878
 URL: https://issues.apache.org/jira/browse/OAK-7878
 Project: Jackrabbit Oak
  Issue Type: Test
  Components: segment-tar
Affects Versions: 1.9.10
Reporter: Axel Hanikel
 Fix For: 1.9.11


Add a unit test for LoggingHook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7877) Avoid unnecessary operations when logging read operations

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672940#comment-16672940
 ] 

Axel Hanikel commented on OAK-7877:
---

Looks good to me.

> Avoid unnecessary operations when logging read operations
> -
>
> Key: OAK-7877
> URL: https://issues.apache.org/jira/browse/OAK-7877
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Major
> Fix For: 1.10
>
> Attachments: OAK-7877-01.patch
>
>
> The log statements introduced in {{CachingSegmentReader}} by OAK-7864 perform 
> multiple string concatenations even if that specific log is disabled. The log 
> statements should be executed only if the right log level has been enabled 
> for that logger.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7864) Recording read operations to the segment store

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670072#comment-16670072
 ] 

Axel Hanikel commented on OAK-7864:
---

The latest attachment is much less invasive than the previous ones and just 
logs the {{RecordId}} in the {{CachingSegmentReader}}. I'm going to use the log 
produced by this one and see if it is just as easy to replay.

> Recording read operations to the segment store
> --
>
> Key: OAK-7864
> URL: https://issues.apache.org/jira/browse/OAK-7864
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: segment-tar
>Affects Versions: 1.9.9
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Attachments: 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch
>
>
> This is the companion to OAK-7849. Its aim is to record read operations in a 
> log file, like this:
> {noformat}
> 1540373238486 FelixStartLevel n? /:clusterConfig
> 1540373238497 FelixStartLevel n? /:clusterConfig
> 1540373238540 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238542 FelixStartLevel n? /:clusterConfig
> 1540373238551 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238560 FelixStartLevel n? /:clusterConfig
> 1540373238569 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/fd:xdp
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OsgiConfig
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:HierarchyNode
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:Folder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OrderedFolder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces/rep:nsdata
> 1540373243634 Apache+Sling+Repository+Startup+Thread p? 
> /jcr:system/rep:namespaces sling
> {noformat}
>  
> where n = node, p = property and '?' stands for "read" (so that the output 
> can be combined with the log from OAK-7849, if desired).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7864) Recording read operations to the segment store

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7864:
--
Attachment: 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch

> Recording read operations to the segment store
> --
>
> Key: OAK-7864
> URL: https://issues.apache.org/jira/browse/OAK-7864
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: segment-tar
>Affects Versions: 1.9.9
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Attachments: 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch
>
>
> This is the companion to OAK-7849. Its aim is to record read operations in a 
> log file, like this:
> {noformat}
> 1540373238486 FelixStartLevel n? /:clusterConfig
> 1540373238497 FelixStartLevel n? /:clusterConfig
> 1540373238540 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238542 FelixStartLevel n? /:clusterConfig
> 1540373238551 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238560 FelixStartLevel n? /:clusterConfig
> 1540373238569 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/fd:xdp
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OsgiConfig
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:HierarchyNode
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:Folder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OrderedFolder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces/rep:nsdata
> 1540373243634 Apache+Sling+Repository+Startup+Thread p? 
> /jcr:system/rep:namespaces sling
> {noformat}
>  
> where n = node, p = property and '?' stands for "read" (so that the output 
> can be combined with the log from OAK-7849, if desired).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OAK-7864) Recording read operations to the segment store

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669969#comment-16669969
 ] 

Axel Hanikel edited comment on OAK-7864 at 10/31/18 11:19 AM:
--

Attaching new patch which only logs property reads in order to reduce the 
amount of data produced. Also logging is about twice as fast by collecting log 
entries in a queue and writing them in a single batch every second or so.

However, this approach does not work very well. There were a few unexpected 
problems, perhaps due to the fact that wrapped and non-wrapped NodeStates are 
being mixed (not sure about that, however). But also, this implementation 
traces API calls rather than actual segment tar accesses, so I'll try to do the 
logging at the {{CachingSegmentReader}} level instead.


was (Author: ahanikel):
Attaching new patch which logs only property reads in order to reduce the 
amount of data produced. Also logging is about twice as fast by collecting log 
entries in a queue and writing them in a single batch every second or so.

However, this approach does not work very well. There were a few unexpected 
problems, perhaps due to the fact that wrapped and non-wrapped NodeStates are 
being mixed (not sure about that, however). But also, this implementation 
traces API calls rather than actual segment tar accesses, so I'll try to do the 
logging at the {{CachingSegmentReader}} level instead.

> Recording read operations to the segment store
> --
>
> Key: OAK-7864
> URL: https://issues.apache.org/jira/browse/OAK-7864
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: segment-tar
>Affects Versions: 1.9.9
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Attachments: 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch
>
>
> This is the companion to OAK-7849. Its aim is to record read operations in a 
> log file, like this:
> {noformat}
> 1540373238486 FelixStartLevel n? /:clusterConfig
> 1540373238497 FelixStartLevel n? /:clusterConfig
> 1540373238540 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238542 FelixStartLevel n? /:clusterConfig
> 1540373238551 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238560 FelixStartLevel n? /:clusterConfig
> 1540373238569 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/fd:xdp
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OsgiConfig
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:HierarchyNode
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:Folder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OrderedFolder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces/rep:nsdata
> 1540373243634 Apache+Sling+Repository+Startup+Thread p? 
> /jcr:system/rep:namespaces sling
> {noformat}
>  
> where n = node, p = property and '?' stands for "read" (so that the output 
> can be combined with the log from OAK-7849, if desired).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OAK-7864) Recording read operations to the segment store

[Axel Hanikel (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669969#comment-16669969
 ] 

Axel Hanikel commented on OAK-7864:
---

Attaching new patch which logs only property reads in order to reduce the 
amount of data produced. Also logging is about twice as fast by collecting log 
entries in a queue and writing them in a single batch every second or so.

However, this approach does not work very well. There were a few unexpected 
problems, perhaps due to the fact that wrapped and non-wrapped NodeStates are 
being mixed (not sure about that, however). But also, this implementation 
traces API calls rather than actual segment tar accesses, so I'll try to do the 
logging at the {{CachingSegmentReader}} level instead.

> Recording read operations to the segment store
> --
>
> Key: OAK-7864
> URL: https://issues.apache.org/jira/browse/OAK-7864
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: segment-tar
>Affects Versions: 1.9.9
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Attachments: 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch
>
>
> This is the companion to OAK-7849. Its aim is to record read operations in a 
> log file, like this:
> {noformat}
> 1540373238486 FelixStartLevel n? /:clusterConfig
> 1540373238497 FelixStartLevel n? /:clusterConfig
> 1540373238540 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238542 FelixStartLevel n? /:clusterConfig
> 1540373238551 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238560 FelixStartLevel n? /:clusterConfig
> 1540373238569 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/fd:xdp
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OsgiConfig
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:HierarchyNode
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:Folder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OrderedFolder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces/rep:nsdata
> 1540373243634 Apache+Sling+Repository+Startup+Thread p? 
> /jcr:system/rep:namespaces sling
> {noformat}
>  
> where n = node, p = property and '?' stands for "read" (so that the output 
> can be combined with the log from OAK-7849, if desired).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OAK-7864) Recording read operations to the segment store

[Axel Hanikel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Hanikel updated OAK-7864:
--
Attachment: 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch

> Recording read operations to the segment store
> --
>
> Key: OAK-7864
> URL: https://issues.apache.org/jira/browse/OAK-7864
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: segment-tar
>Affects Versions: 1.9.9
>Reporter: Axel Hanikel
>Assignee: Francesco Mari
>Priority: Minor
> Attachments: 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch, 
> 0001-OAK-7864-Recording-read-operations-to-the-segment-st.patch
>
>
> This is the companion to OAK-7849. Its aim is to record read operations in a 
> log file, like this:
> {noformat}
> 1540373238486 FelixStartLevel n? /:clusterConfig
> 1540373238497 FelixStartLevel n? /:clusterConfig
> 1540373238540 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238542 FelixStartLevel n? /:clusterConfig
> 1540373238551 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373238560 FelixStartLevel n? /:clusterConfig
> 1540373238569 FelixStartLevel p? /:clusterConfig :clusterId
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243447 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/fd:xdp
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243541 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OsgiConfig
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:HierarchyNode
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:Folder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/jcr:nodeTypes/sling:OrderedFolder
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? /jcr:system
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces
> 1540373243619 Apache+Sling+Repository+Startup+Thread n? 
> /jcr:system/rep:namespaces/rep:nsdata
> 1540373243634 Apache+Sling+Repository+Startup+Thread p? 
> /jcr:system/rep:namespaces sling
> {noformat}
>  
> where n = node, p = property and '?' stands for "read" (so that the output 
> can be combined with the log from OAK-7849, if desired).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)