[jira] [Commented] (OAK-6584) Add tooling API
[ https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154890#comment-16154890 ] Francesco Mari commented on OAK-6584: - I'm in favour of the second option. Maybe in the future if some features are implemented again and again on top of this API, like structural equality, we could think about collecting them in shared utility classes. > Add tooling API > --- > > Key: OAK-6584 > URL: https://issues.apache.org/jira/browse/OAK-6584 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: tooling > Fix For: 1.8 > > > h3. Current situation > Current segment store related tools are implemented ad-hoc by potentially > relying on internal implementation details of Oak Segment Tar. This makes > those tools less useful, portable, stable and potentially applicable than > they should be. > h3. Goal > Provide a common and sufficiently stable Oak Tooling API for implementing > segment store related tools. The API should be independent of Oak and not > available for normal production use of Oak. Specifically it should not be > possible to it to implement production features and production features must > not rely on it. It must be possible to implement the Oak Tooling API in Oak > 1.8 and it should be possible for Oak 1.6. > h3. Typical use cases > * Query the number of nodes / properties / values in a given path satisfying > some criteria > * Aggregate a certain value on queries like the above > * Calculate size of the content / size on disk > * Analyse changes. E.g. how many binaries bigger than a certain threshold > were added / removed between two given revisions. What is the sum of their > sizes? > * Analyse locality: measure of locality of node states. Incident plots (See > https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973). > * Analyse level of deduplication (e.g. of checkpoint) > h3. Validation > Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the > tooling API. > h3. API draft > * Whiteboard shot of the [API > entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg] > identified initially. > * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api] > takes place on Github for now. We'll move to the Apache SVN as soon as > considered mature enough and have a consensus of where to best move it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (OAK-6619) Async indexer thread may get stuck in CopyOnWriteDirectory close method
[
https://issues.apache.org/jira/browse/OAK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154854#comment-16154854
]
Chetan Mehrotra edited comment on OAK-6619 at 9/6/17 6:40 AM:
--
Some observations when this happens based on heap dump analysis
* queue has a single entry for STOP but it appears that its not getting
processed
* errorInCopy is empty
* currentTask completed is false
was (Author: chetanm):
Some observations when this happens based on heap dump analysis
* queue has a single entry for STOP but it appears that its not getting
processed
* errorInCopy is empty
> Async indexer thread may get stuck in CopyOnWriteDirectory close method
> ---
>
> Key: OAK-6619
> URL: https://issues.apache.org/jira/browse/OAK-6619
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Fix For: 1.8
>
> Attachments: status-threaddump-Sep-5.txt
>
>
> With copy-on-write mode enabled at times its seen that async index thread
> remain stuck in CopyOnWriteDirectory#close method
> {noformat}
> "async-index-update-async" prio=5 tid=0xb9e63 nid=0x timed_waiting
>java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - waiting to lock <0x2504cd51> (a
> java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateSuggester(DefaultIndexWriter.java:177)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.close(DefaultIndexWriter.java:121)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:136)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:154)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:357)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:60)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:727)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:572)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:431)
> - locked <0x3d542de5> (a
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
> at
> org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:245)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The thread is waiting on a latch and no other thread is going to release the
> latch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6619) Async indexer thread may get stuck in CopyOnWriteDirectory close method
[
https://issues.apache.org/jira/browse/OAK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-6619:
-
Attachment: status-threaddump-Sep-5.txt
> Async indexer thread may get stuck in CopyOnWriteDirectory close method
> ---
>
> Key: OAK-6619
> URL: https://issues.apache.org/jira/browse/OAK-6619
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Fix For: 1.8
>
> Attachments: status-threaddump-Sep-5.txt
>
>
> With copy-on-write mode enabled at times its seen that async index thread
> remain stuck in CopyOnWriteDirectory#close method
> {noformat}
> "async-index-update-async" prio=5 tid=0xb9e63 nid=0x timed_waiting
>java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - waiting to lock <0x2504cd51> (a
> java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateSuggester(DefaultIndexWriter.java:177)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.close(DefaultIndexWriter.java:121)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:136)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:154)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:357)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:60)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:727)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:572)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:431)
> - locked <0x3d542de5> (a
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
> at
> org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:245)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The thread is waiting on a latch and no other thread is going to release the
> latch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6619) Async indexer thread may get stuck in CopyOnWriteDirectory close method
[
https://issues.apache.org/jira/browse/OAK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154854#comment-16154854
]
Chetan Mehrotra commented on OAK-6619:
--
Some observations when this happens based on heap dump analysis
* queue has a single entry for STOP but it appears that its not getting
processed
* errorInCopy is empty
> Async indexer thread may get stuck in CopyOnWriteDirectory close method
> ---
>
> Key: OAK-6619
> URL: https://issues.apache.org/jira/browse/OAK-6619
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Fix For: 1.8
>
>
> With copy-on-write mode enabled at times its seen that async index thread
> remain stuck in CopyOnWriteDirectory#close method
> {noformat}
> "async-index-update-async" prio=5 tid=0xb9e63 nid=0x timed_waiting
>java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - waiting to lock <0x2504cd51> (a
> java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateSuggester(DefaultIndexWriter.java:177)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.close(DefaultIndexWriter.java:121)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:136)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:154)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:357)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:60)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:727)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:572)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:431)
> - locked <0x3d542de5> (a
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
> at
> org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:245)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The thread is waiting on a latch and no other thread is going to release the
> latch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6619) Async indexer thread may get stuck in CopyOnWriteDirectory close method
Chetan Mehrotra created OAK-6619:
Summary: Async indexer thread may get stuck in
CopyOnWriteDirectory close method
Key: OAK-6619
URL: https://issues.apache.org/jira/browse/OAK-6619
Project: Jackrabbit Oak
Issue Type: Bug
Components: lucene
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Priority: Critical
Fix For: 1.8
With copy-on-write mode enabled at times its seen that async index thread
remain stuck in CopyOnWriteDirectory#close method
{noformat}
"async-index-update-async" prio=5 tid=0xb9e63 nid=0x timed_waiting
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- waiting to lock <0x2504cd51> (a
java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
at
org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateSuggester(DefaultIndexWriter.java:177)
at
org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.close(DefaultIndexWriter.java:121)
at
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:136)
at
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:154)
at
org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:357)
at
org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:60)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
at
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:727)
at
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:572)
at
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:431)
- locked <0x3d542de5> (a
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
at
org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:245)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The thread is waiting on a latch and no other thread is going to release the
latch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
[
https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154852#comment-16154852
]
Amit Jain commented on OAK-6611:
BTW the errors do indicate that there are still uploads in progress and hence
the stats should have indicated that (since, there is a lag in the purge
controlled by {{stagingPurgeInterval}} and subsequent update in the stats. The
problem is due to the fact that a meaningful stats provider has to be
initialized to get accurate stats as proposed below.
{code}
Index:
oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===
---
oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
(revision 59dfb84da88157ed59c7c5fb670d9956257b0d18)
+++
oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java
(revision 93fb425bc7f30f2299b6b24e76c1addd192a79ad)
@@ -22,6 +22,7 @@
import java.io.IOException;
import java.util.HashSet;
import java.util.Properties;
+import java.util.concurrent.Executors;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -35,6 +36,8 @@
import com.google.common.io.Closer;
import com.google.common.io.Files;
+import org.apache.jackrabbit.oak.stats.DefaultStatisticsProvider;
+import org.apache.jackrabbit.oak.stats.StatisticsProvider;
public class S3DataStoreFactory implements BlobStoreFactory {
@@ -72,6 +75,13 @@
S3DataStore delegate = new S3DataStore();
delegate.setProperties(props);
delegate.setPath(directory);
+
+// Initialize a default stats provider
+StatisticsProvider statsProvider = new
DefaultStatisticsProvider(Executors.newSingleThreadScheduledExecutor());
+delegate.setStatisticsProvider(statsProvider);
+// Reduce staging purge interval to 60 seconds
+delegate.setStagingPurgeInterval(60);
+
try {
delegate.init(tempHomeDir.getPath());
} catch (RepositoryException e) {
{code}
What I found confusing is that the logs related to the close of the caches are
missing which should have been around the time tar was closed.
{noformat}
05.09.2017 12:04:24.028 INFO o.a.j.o.s.f.FileStore: TarMK closed:
/data/cq/crx-quickstart/repository-segment-tar-20170905-120210/segmentstore
05.09.2017 12:04:24.827 INFO o.a.j.o.p.s.f.FileStore: TarMK closed:
/data/cq/crx-quickstart/repository/segmentstore
05.09.2017 12:04:25.107 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [8632286ebab5d4a9ebda0a2e0abab4c55ca4825c2f0358bb05f5926dda367f99],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/86/32/28/8632286ebab5d4a9ebda0a2e0abab4c55ca4825c2f0358bb05f5926dda367f99]
05.09.2017 12:04:25.107 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [c685419afb510a9fa010f50a808a18b72e2bc0701ce522ea783e8a1e2a292e9d],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/c6/85/41/c685419afb510a9fa010f50a808a18b72e2bc0701ce522ea783e8a1e2a292e9d]
05.09.2017 12:04:25.209 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [8bfa063c184e18178393452f9e4760a53a482ce3f7501f0cb85fcd47e3cab0c1],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/8b/fa/06/8bfa063c184e18178393452f9e4760a53a482ce3f7501f0cb85fcd47e3cab0c1]
05.09.2017 12:04:25.209 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [45f573b27fea6b0b54be49391ea32e62c1ef1d31f66257772a75473d30692e77],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/45/f5/73/45f573b27fea6b0b54be49391ea32e62c1ef1d31f66257772a75473d30692e77]
05.09.2017 12:04:25.213 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [e8df0bbc0286e2e4673c5ae964d03ba4fcf5869eec61b1f2b510aa0c83ed0cb0],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/e8/df/0b/e8df0bbc0286e2e4673c5ae964d03ba4fcf5869eec61b1f2b510aa0c83ed0cb0]
05.09.2017 12:04:25.311 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [febf1aaa839a16b43d93ba4424dc88c93c1c54755cf1c5b4368207f936d29275],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/fe/bf/1a/febf1aaa839a16b43d93ba4424dc88c93c1c54755cf1c5b4368207f936d29275]
05.09.2017 12:04:25.311 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [06498358dde411634fe5ef9f31c66bce03fe6adef3f3aded2513419b2a76a74f],
[/data/cq/crx-quickstart/repository/repository/datastore/upload/06/49/83/06498358dde411634fe5ef9f31c66bce03fe6adef3f3aded2513419b2a76a74f]
05.09.2017 12:04:25.316 INFO o.a.j.o.p.b.UploadStagingCache: Successfully
added [769f241c95dee87eee65ff846c17b14bbadc8febac293a3953fd1141d8af3385],
[/data/cq/crx-quickstart/repository/repository/datastore/uploa
[jira] [Commented] (OAK-6618) Query with offset 29 with any limit more than 21 gives node size -1 , when number of nodes in repository is 50
[ https://issues.apache.org/jira/browse/OAK-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154847#comment-16154847 ] Chetan Mehrotra commented on OAK-6618: -- Can you try with 1.6.3 build as it looks like duplicate of OAK-6391 > Query with offset 29 with any limit more than 21 gives node size -1 , when > number of nodes in repository is 50 > -- > > Key: OAK-6618 > URL: https://issues.apache.org/jira/browse/OAK-6618 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.6.1 >Reporter: Mouli >Priority: Critical > > There is something wrong with offset and limit feature in JCR, i have 50 > nodes in repository and i started to test pagination so i used offset and > limit . It works fine for all offset and limit combination except one case > that is offset 29 and limit =21, only this case node size returning as -1 in > all other cases it working as expected. i tried with other combinations of > node size viz 52 limit=22&offset=31 > not working for this combination too but it works if i increase the offset to > 32. > listing out combinations it doesn't work > size=50, offset=29 and limit=21(any number > 21 ) will give u node size -1. > size=52, offset=31 and limit=22(any number > 22 ) will give u node size -1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6618) Query with offset 29 with any limit more than 21 gives node size -1 , when number of nodes in repository is 50
Mouli created OAK-6618: --- Summary: Query with offset 29 with any limit more than 21 gives node size -1 , when number of nodes in repository is 50 Key: OAK-6618 URL: https://issues.apache.org/jira/browse/OAK-6618 Project: Jackrabbit Oak Issue Type: Bug Components: core Affects Versions: 1.6.1 Reporter: Mouli Priority: Critical There is something wrong with offset and limit feature in JCR, i have 50 nodes in repository and i started to test pagination so i used offset and limit . It works fine for all offset and limit combination except one case that is offset 29 and limit =21, only this case node size returning as -1 in all other cases it working as expected. i tried with other combinations of node size viz 52 limit=22&offset=31 not working for this combination too but it works if i increase the offset to 32. listing out combinations it doesn't work size=50, offset=29 and limit=21(any number > 21 ) will give u node size -1. size=52, offset=31 and limit=22(any number > 22 ) will give u node size -1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6353) Use Document order traversal for reindexing performed on DocumentNodeStore setups
[ https://issues.apache.org/jira/browse/OAK-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154791#comment-16154791 ] Chetan Mehrotra commented on OAK-6353: -- Applied the patch with r1807438 > Use Document order traversal for reindexing performed on DocumentNodeStore > setups > - > > Key: OAK-6353 > URL: https://issues.apache.org/jira/browse/OAK-6353 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8 > > Attachments: OAK-6353-v1.patch, OAK-6353-v2.patch > > > [~tmueller] suggested > [here|https://issues.apache.org/jira/browse/OAK-6246?focusedCommentId=16034442&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16034442] > that document order traversal can be faster compared to current mode of path > based traversal. Initial test indicate that such a traversal can be order of > magnitude faster. > So this task is meant to implement such an approach and see if it can be a > viable indexing mode used for DocumentNodeStore based setups -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
[
https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Jain reassigned OAK-6611:
--
Assignee: Amit Jain (was: Tomek Rękawek)
> [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with
> oak-upgrade
> ---
>
> Key: OAK-6611
> URL: https://issues.apache.org/jira/browse/OAK-6611
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-cloud, upgrade
>Affects Versions: 1.8, 1.7.7
>Reporter: Arek Kita
>Assignee: Amit Jain
>Priority: Critical
> Fix For: 1.8, 1.7.7
>
> Attachments: oak-upgrade-oak-blob-cloud-20170905.log.gz,
> oak-upgrade-with-oak-blob-cloud.fragment.log
>
>
> [~tomek.rekawek], [~amitjain] Due to async nature of S3 datastore
> format/upload process the migration ends up way quicker than S3 datastore is
> being migrated. This leads to a huge number of exceptions shown due to *non
> synchronised* nature of *oak-upgrade* migration process vs async S3 datastore
> background processes.
> I see a few possible solutions for that:
> * disable migration/uploading of S3 cache for the time of migration (bad idea
> IMHO)
> * wait for it (it might be desired or a bad idea as it might take longer than
> migration for a few cases)
> * pause it when migration is completed in a clean way (so some binaries
> aren't uploaded and moved to a new datastore format) -- not sure if such
> mixed state is ok at all
> WDYT?
> Please also note that this happens only when {{\-\-src-s3config
> \-\-src-s3datastore}} options are specified during migration which in many
> cases is true (this would be the same for the destination DataStore options).
> Referencing a source datastore is needed (even if {{\-\-copy-binaries}} is
> not included) in example to copy checkpoints properly.
> The example exception is like the below:
> {code}
> 01.09.2017 11:39:41.088 ERROR o.a.j.o.p.b.UploadStagingCache: Error adding
> file to backend
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
> at
> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3996)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1161)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1136)
> at
> org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.write(S3Backend.java:201)
> at
> org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore$2.write(AbstractSharedCachingDataStore.java:170)
> at
> org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.java:341)
> at
> org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.j
[jira] [Resolved] (OAK-6581) Ensure mounts are consistent with the namespace registry
[ https://issues.apache.org/jira/browse/OAK-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Munteanu resolved OAK-6581. -- Resolution: Fixed Fixed in [r1807403|https://svn.apache.org/r1807403] > Ensure mounts are consistent with the namespace registry > > > Key: OAK-6581 > URL: https://issues.apache.org/jira/browse/OAK-6581 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: composite >Reporter: Robert Munteanu >Assignee: Robert Munteanu > Fix For: 1.8, 1.7.7 > > > When a mount is added, we should make sure that the nodes and properties are > consistent with the global namespace declarations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OAK-6580) Ensure mounts are consistent with the node type registry
[ https://issues.apache.org/jira/browse/OAK-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Munteanu reassigned OAK-6580: Assignee: Robert Munteanu > Ensure mounts are consistent with the node type registry > > > Key: OAK-6580 > URL: https://issues.apache.org/jira/browse/OAK-6580 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: composite >Reporter: Robert Munteanu >Assignee: Robert Munteanu > Fix For: 1.8, 1.7.7 > > > When a mount is added, we should make sure that the nodes are: > * defined in the NodeTypeRegistry > * consistent with the node type definitions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-5855) RDBDocumentStore: improve query support for VersionGC
[
https://issues.apache.org/jira/browse/OAK-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154066#comment-16154066
]
Julian Reschke commented on OAK-5855:
-
trunk: [r1788476|http://svn.apache.org/r1788476]
1.6: [r1807378|http://svn.apache.org/r1807378]
> RDBDocumentStore: improve query support for VersionGC
> -
>
> Key: OAK-5855
> URL: https://issues.apache.org/jira/browse/OAK-5855
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: rdbmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-5855-10.diff, OAK-5855-11.diff, OAK-5855-12.diff,
> OAK-5855-13.diff, OAK-5855-14.diff, OAK-5855-15.diff, OAK-5855-16.diff
>
>
> ...matching {{MongoVersionSupport}}, such as: no batched query needed; just
> return an iterable over delete candidates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (OAK-5855) RDBDocumentStore: improve query support for VersionGC
[
https://issues.apache.org/jira/browse/OAK-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-5855:
Comment: was deleted
(was: trunk: [r1788476|http://svn.apache.org/r1788476]
)
> RDBDocumentStore: improve query support for VersionGC
> -
>
> Key: OAK-5855
> URL: https://issues.apache.org/jira/browse/OAK-5855
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: rdbmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-5855-10.diff, OAK-5855-11.diff, OAK-5855-12.diff,
> OAK-5855-13.diff, OAK-5855-14.diff, OAK-5855-15.diff, OAK-5855-16.diff
>
>
> ...matching {{MongoVersionSupport}}, such as: no batched query needed; just
> return an iterable over delete candidates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-5855) RDBDocumentStore: improve query support for VersionGC
[
https://issues.apache.org/jira/browse/OAK-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-5855:
Labels: candidate_oak_1_4 (was: )
> RDBDocumentStore: improve query support for VersionGC
> -
>
> Key: OAK-5855
> URL: https://issues.apache.org/jira/browse/OAK-5855
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: rdbmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-5855-10.diff, OAK-5855-11.diff, OAK-5855-12.diff,
> OAK-5855-13.diff, OAK-5855-14.diff, OAK-5855-15.diff, OAK-5855-16.diff
>
>
> ...matching {{MongoVersionSupport}}, such as: no batched query needed; just
> return an iterable over delete candidates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-5855) RDBDocumentStore: improve query support for VersionGC
[
https://issues.apache.org/jira/browse/OAK-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-5855:
Fix Version/s: 1.6.6
> RDBDocumentStore: improve query support for VersionGC
> -
>
> Key: OAK-5855
> URL: https://issues.apache.org/jira/browse/OAK-5855
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: rdbmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-5855-10.diff, OAK-5855-11.diff, OAK-5855-12.diff,
> OAK-5855-13.diff, OAK-5855-14.diff, OAK-5855-15.diff, OAK-5855-16.diff
>
>
> ...matching {{MongoVersionSupport}}, such as: no batched query needed; just
> return an iterable over delete candidates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-5985) add CloseableIterator similar to CloseableIterable
[ https://issues.apache.org/jira/browse/OAK-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153957#comment-16153957 ] Julian Reschke commented on OAK-5985: - trunk: [r1788463|http://svn.apache.org/r1788463] 1.6: [r1807369|http://svn.apache.org/r1807369] > add CloseableIterator similar to CloseableIterable > -- > > Key: OAK-5985 > URL: https://issues.apache.org/jira/browse/OAK-5985 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.7.0, 1.8, 1.6.6 > > > Needed by OAK-5855 but tracked separately so it can be independently > back-ported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-5985) add CloseableIterator similar to CloseableIterable
[ https://issues.apache.org/jira/browse/OAK-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-5985: Fix Version/s: 1.6.6 > add CloseableIterator similar to CloseableIterable > -- > > Key: OAK-5985 > URL: https://issues.apache.org/jira/browse/OAK-5985 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.7.0, 1.8, 1.6.6 > > > Needed by OAK-5855 but tracked separately so it can be independently > back-ported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-5985) add CloseableIterator similar to CloseableIterable
[ https://issues.apache.org/jira/browse/OAK-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-5985: Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 (was: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 candidate_oak_1_6) > add CloseableIterator similar to CloseableIterable > -- > > Key: OAK-5985 > URL: https://issues.apache.org/jira/browse/OAK-5985 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.7.0, 1.8, 1.6.6 > > > Needed by OAK-5855 but tracked separately so it can be independently > back-ported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (OAK-5985) add CloseableIterator similar to CloseableIterable
[ https://issues.apache.org/jira/browse/OAK-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-5985: Comment: was deleted (was: trunk: [r1788463|http://svn.apache.org/r1788463] ) > add CloseableIterator similar to CloseableIterable > -- > > Key: OAK-5985 > URL: https://issues.apache.org/jira/browse/OAK-5985 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.7.0, 1.8, 1.6.6 > > > Needed by OAK-5855 but tracked separately so it can be independently > back-ported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-5985) add CloseableIterator similar to CloseableIterable
[ https://issues.apache.org/jira/browse/OAK-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-5985: Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 candidate_oak_1_6 (was: ) > add CloseableIterator similar to CloseableIterable > -- > > Key: OAK-5985 > URL: https://issues.apache.org/jira/browse/OAK-5985 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Julian Reschke >Assignee: Julian Reschke > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, > candidate_oak_1_6 > Fix For: 1.7.0, 1.8 > > > Needed by OAK-5855 but tracked separately so it can be independently > back-ported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OAK-6581) Ensure mounts are consistent with the namespace registry
[ https://issues.apache.org/jira/browse/OAK-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Munteanu reassigned OAK-6581: Assignee: Robert Munteanu > Ensure mounts are consistent with the namespace registry > > > Key: OAK-6581 > URL: https://issues.apache.org/jira/browse/OAK-6581 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: composite >Reporter: Robert Munteanu >Assignee: Robert Munteanu > Fix For: 1.8, 1.7.7 > > > When a mount is added, we should make sure that the nodes and properties are > consistent with the global namespace declarations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6506) Ensure unique property indexes are consistent when mounting NodeStores
[
https://issues.apache.org/jira/browse/OAK-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153790#comment-16153790
]
Robert Munteanu commented on OAK-6506:
--
Note that my initial comment was incorrect - we neither needed to get all the
strategies using the {{Multiplexers}} class, and the checks are reflexive.
> Ensure unique property indexes are consistent when mounting NodeStores
> --
>
> Key: OAK-6506
> URL: https://issues.apache.org/jira/browse/OAK-6506
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: composite
>Reporter: Robert Munteanu
>Assignee: Robert Munteanu
> Fix For: 1.8, 1.7.7
>
>
> When mounting a NodeStore in a composite setup, we should validate that the
> unique property indexes do not contain duplicate entries across all the node
> stores.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6171) Refactor MongoBlobReferenceIterator
[
https://issues.apache.org/jira/browse/OAK-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153788#comment-16153788
]
Julian Reschke commented on OAK-6171:
-
trunk: [r1794683|http://svn.apache.org/r1794683]
1.6: [r1807360|http://svn.apache.org/r1807360]
> Refactor MongoBlobReferenceIterator
> ---
>
> Key: OAK-6171
> URL: https://issues.apache.org/jira/browse/OAK-6171
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk, mongomk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6171.diff
>
>
> ...to extend from {{BlobReferenceIterator}} once OAK-6162 is ready.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-6506) Ensure unique property indexes are consistent when mounting NodeStores
[ https://issues.apache.org/jira/browse/OAK-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Munteanu resolved OAK-6506. -- Resolution: Fixed Fixed in [r1807361|https://svn.apache.org/r1807361] > Ensure unique property indexes are consistent when mounting NodeStores > -- > > Key: OAK-6506 > URL: https://issues.apache.org/jira/browse/OAK-6506 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: composite >Reporter: Robert Munteanu >Assignee: Robert Munteanu > Fix For: 1.8, 1.7.7 > > > When mounting a NodeStore in a composite setup, we should validate that the > unique property indexes do not contain duplicate entries across all the node > stores. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6171) Refactor MongoBlobReferenceIterator
[
https://issues.apache.org/jira/browse/OAK-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-6171:
Labels: candidate_oak_1_4 (was: )
> Refactor MongoBlobReferenceIterator
> ---
>
> Key: OAK-6171
> URL: https://issues.apache.org/jira/browse/OAK-6171
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk, mongomk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6171.diff
>
>
> ...to extend from {{BlobReferenceIterator}} once OAK-6162 is ready.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (OAK-6171) Refactor MongoBlobReferenceIterator
[
https://issues.apache.org/jira/browse/OAK-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-6171:
Comment: was deleted
(was: trunk: [r1794683|http://svn.apache.org/r1794683]
)
> Refactor MongoBlobReferenceIterator
> ---
>
> Key: OAK-6171
> URL: https://issues.apache.org/jira/browse/OAK-6171
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk, mongomk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6171.diff
>
>
> ...to extend from {{BlobReferenceIterator}} once OAK-6162 is ready.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6171) Refactor MongoBlobReferenceIterator
[
https://issues.apache.org/jira/browse/OAK-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-6171:
Fix Version/s: 1.6.6
> Refactor MongoBlobReferenceIterator
> ---
>
> Key: OAK-6171
> URL: https://issues.apache.org/jira/browse/OAK-6171
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk, mongomk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6171.diff
>
>
> ...to extend from {{BlobReferenceIterator}} once OAK-6162 is ready.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6162) BlobReferenceIterator refactoring
[
https://issues.apache.org/jira/browse/OAK-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153699#comment-16153699
]
Julian Reschke commented on OAK-6162:
-
trunk: [r1794393|http://svn.apache.org/r1794393]
1.6: [r1807352|http://svn.apache.org/r1807352]
> BlobReferenceIterator refactoring
> -
>
> Key: OAK-6162
> URL: https://issues.apache.org/jira/browse/OAK-6162
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6162.diff
>
>
> - generic iterator should use {{Utils.getSelectedDocuments()}} to obtain
> iterator
> - {{MongoBlobReferenceIterator}} should just extend
> {{BlobReferenceIterator}}, using its own iterator (EDIT: will move this into
> a separate ticket)
> - {{Utils.getSelectedDocuments()}} should allow specifying the batch size
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (OAK-6162) BlobReferenceIterator refactoring
[
https://issues.apache.org/jira/browse/OAK-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-6162:
Comment: was deleted
(was: trunk: [r1794393|http://svn.apache.org/r1794393]
)
> BlobReferenceIterator refactoring
> -
>
> Key: OAK-6162
> URL: https://issues.apache.org/jira/browse/OAK-6162
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6162.diff
>
>
> - generic iterator should use {{Utils.getSelectedDocuments()}} to obtain
> iterator
> - {{MongoBlobReferenceIterator}} should just extend
> {{BlobReferenceIterator}}, using its own iterator (EDIT: will move this into
> a separate ticket)
> - {{Utils.getSelectedDocuments()}} should allow specifying the batch size
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6162) BlobReferenceIterator refactoring
[
https://issues.apache.org/jira/browse/OAK-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-6162:
Fix Version/s: 1.6.6
> BlobReferenceIterator refactoring
> -
>
> Key: OAK-6162
> URL: https://issues.apache.org/jira/browse/OAK-6162
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6162.diff
>
>
> - generic iterator should use {{Utils.getSelectedDocuments()}} to obtain
> iterator
> - {{MongoBlobReferenceIterator}} should just extend
> {{BlobReferenceIterator}}, using its own iterator (EDIT: will move this into
> a separate ticket)
> - {{Utils.getSelectedDocuments()}} should allow specifying the batch size
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6162) BlobReferenceIterator refactoring
[
https://issues.apache.org/jira/browse/OAK-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-6162:
Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 (was:
candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 candidate_oak_1_6)
> BlobReferenceIterator refactoring
> -
>
> Key: OAK-6162
> URL: https://issues.apache.org/jira/browse/OAK-6162
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.7.0, 1.8, 1.6.6
>
> Attachments: OAK-6162.diff
>
>
> - generic iterator should use {{Utils.getSelectedDocuments()}} to obtain
> iterator
> - {{MongoBlobReferenceIterator}} should just extend
> {{BlobReferenceIterator}}, using its own iterator (EDIT: will move this into
> a separate ticket)
> - {{Utils.getSelectedDocuments()}} should allow specifying the batch size
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6479) misleading log entry when there's no revision to write to the journal
[
https://issues.apache.org/jira/browse/OAK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153665#comment-16153665
]
Julian Reschke commented on OAK-6479:
-
[~elemer] - I guess we currently do not understand the root cause.
> misleading log entry when there's no revision to write to the journal
> -
>
> Key: OAK-6479
> URL: https://issues.apache.org/jira/browse/OAK-6479
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.0.38, 1.2.26, 1.4.17
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Attachments: OAK-6479.diff
>
>
> When there's no new revision to write to the journal, we see:
> {noformat}
> 21.06.2017 05:04:03.666 *ERROR* [DocumentNodeStore background update thread
> (1)] org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore Failed to
> write to journal, accumulating changes for future write (~112 bytes).
> {noformat}
> This was fixed in 1.6 and newer as part of OAK-3976.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-3150) Update Lucene to 6.x series
[
https://issues.apache.org/jira/browse/OAK-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153630#comment-16153630
]
Vikas Saurabh commented on OAK-3150:
[~teofili], it seems lucene provides an {{IndexUpgrader}} tool \[0]. I think
one of the concerns here is that reindexing to new format might be expensive
for the customers. I think/hope lucene's tool might be faster that reindexing
from oak side.
PS: I looked at lucene6 migrate too \[1], but it doesn't specify anything in
particular, so maybe the version5 {{IndexUpgrader}} is all we need.
PS1: I think this is the only issue we are tracking for moving to lucene 6.
Please paste this info elsewhere if that's more relevant.
\[0]: https://lucene.apache.org/core/5_0_0/MIGRATE.html
\[1]: https://lucene.apache.org/core/6_0_0/MIGRATE.html
> Update Lucene to 6.x series
> ---
>
> Key: OAK-3150
> URL: https://issues.apache.org/jira/browse/OAK-3150
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
> Labels: technical_debt
> Fix For: 1.8
>
>
> We should look into updating the Lucene version to 6.x. Java 8 is the minimum
> Java version required
> Note this is to be done for trunk only
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6610) Document Node Store Merge Conflit for concurrent requests
[ https://issues.apache.org/jira/browse/OAK-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153602#comment-16153602 ] Marcel Reutegger commented on OAK-6610: --- bq. How do we maintain the eventual consistency in the clustered environment. This is done by Oak and not something the application needs to do. bq. Is there optimisation can we at node discovery I don't think discovery is the problem here. At least based on the information you provided. The cluster nodes are active, but propagation of changes is slow. This may mean either the Oak cluster nodes or MongoDB is slow. One of the things you should therefore check is how MongoDB performs when you run your tests. Check for number of operations, slow queries, etc. > Document Node Store Merge Conflit for concurrent requests > - > > Key: OAK-6610 > URL: https://issues.apache.org/jira/browse/OAK-6610 > Project: Jackrabbit Oak > Issue Type: Bug > Components: documentmk >Affects Versions: 1.6.4 >Reporter: Satish kumar chitikena >Priority: Blocker > > When we are running the performance test Document MK throwing the exception > saying Merge conflict . > We have 3 JCR instances talking to only one MongoDB primary . How to handle > concurrent requests > /content/Draft/Content/Intuit/Tax/All/Article/FAQ , This node not > versionable Node but why it is looking for base revision and modified > revision etc. > NTP time sync is enabled . > 31.08.2017 06:15:21.545 *ERROR* [127.0.0.1 [1504185184379] POST > /content/Draft/Content/Intuit/Tax/All/Article/FAQ/LV3xQqZdJ/US/en_US > HTTP/1.1] org.apache.sling.servlets.post.impl.operations.ModifyOperation > Exception during response processing. > org.apache.sling.api.resource.PersistenceException: Unable to commit changes > to session. > at > org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit(JcrResourceProvider.java:490) > at > org.apache.sling.resourceresolver.impl.providers.stateful.AuthenticatedResourceProvider.commit(AuthenticatedResourceProvider.java:215) > at > org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.commit(ResourceResolverControl.java:421) > at > org.apache.sling.resourceresolver.impl.ResourceResolverImpl.commit(ResourceResolverImpl.java:1177) > at > org.apache.sling.servlets.post.impl.operations.AbstractPostOperation.run(AbstractPostOperation.java:164) > at > org.apache.sling.servlets.post.impl.SlingPostServlet.doPost(SlingPostServlet.java:228) > at > org.apache.sling.api.servlets.SlingAllMethodsServlet.mayService(SlingAllMethodsServlet.java:149) > at > org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:346) > at > org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:378) > at > org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:552) > at > org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:44) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:77) > at > org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:282) > at > org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:77) > at > com.intuit.cas.extension.CSRFilterServlet.doFilter(CSRFilterServlet.java:57) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68) > at > org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter.doFilter(RequestProgressTrackerLogFilter.java:107) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68) > at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:138) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68) > at > org.apache.sling.engine.impl.SlingRequestProcessorImpl.doProcessRequest(SlingRequestProcessorImpl.java:151) > at > org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:234) > at > org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:85) > at > org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:79) > at > org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:89) > at > org.apache.felix.http.base.internal.handler.FilterHandler.han
[jira] [Resolved] (OAK-6617) Mounts.DefaultMount.getName() should not be empty
[ https://issues.apache.org/jira/browse/OAK-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Munteanu resolved OAK-6617. -- Resolution: Fixed Fixed in [r1807345|https://svn.apache.org/r1807345] > Mounts.DefaultMount.getName() should not be empty > - > > Key: OAK-6617 > URL: https://issues.apache.org/jira/browse/OAK-6617 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: composite >Reporter: Robert Munteanu >Assignee: Robert Munteanu >Priority: Minor > Fix For: 1.8, 1.7.7 > > > For logging and debugging purposes having the mount name be empty is > confusing. I propose using __ as the name. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
[
https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153575#comment-16153575
]
Arek Kita commented on OAK-6611:
Thanks Tomek!
I've validated and I see the code and it used when I'm testing the migration on
my side. Maybe the problem is in fact because the binaries are added serially
so the stats are low and condition is never true?
{code:title=No queued tasks}
05.09.2017 12:04:26.213 ERROR o.a.j.o.p.b.UploadStagingCache: Error adding
file to backend
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@3d831c6e rejected from
java.util.concurrent.ThreadPoolExecutor@3968ebc1[Shutting down, pool size = 19,
active threads = 1, queued tasks = 0, completed tasks = 538]
{code}
> [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with
> oak-upgrade
> ---
>
> Key: OAK-6611
> URL: https://issues.apache.org/jira/browse/OAK-6611
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-cloud, upgrade
>Affects Versions: 1.8, 1.7.7
>Reporter: Arek Kita
>Assignee: Tomek Rękawek
>Priority: Critical
> Fix For: 1.8, 1.7.7
>
> Attachments: oak-upgrade-oak-blob-cloud-20170905.log.gz,
> oak-upgrade-with-oak-blob-cloud.fragment.log
>
>
> [~tomek.rekawek], [~amitjain] Due to async nature of S3 datastore
> format/upload process the migration ends up way quicker than S3 datastore is
> being migrated. This leads to a huge number of exceptions shown due to *non
> synchronised* nature of *oak-upgrade* migration process vs async S3 datastore
> background processes.
> I see a few possible solutions for that:
> * disable migration/uploading of S3 cache for the time of migration (bad idea
> IMHO)
> * wait for it (it might be desired or a bad idea as it might take longer than
> migration for a few cases)
> * pause it when migration is completed in a clean way (so some binaries
> aren't uploaded and moved to a new datastore format) -- not sure if such
> mixed state is ok at all
> WDYT?
> Please also note that this happens only when {{\-\-src-s3config
> \-\-src-s3datastore}} options are specified during migration which in many
> cases is true (this would be the same for the destination DataStore options).
> Referencing a source datastore is needed (even if {{\-\-copy-binaries}} is
> not included) in example to copy checkpoints properly.
> The example exception is like the below:
> {code}
> 01.09.2017 11:39:41.088 ERROR o.a.j.o.p.b.UploadStagingCache: Error adding
> file to backend
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
> at
> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3996)
> at
> com.amazonaws.services
[jira] [Created] (OAK-6617) Mounts.DefaultMount.getName() should not be empty
Robert Munteanu created OAK-6617: Summary: Mounts.DefaultMount.getName() should not be empty Key: OAK-6617 URL: https://issues.apache.org/jira/browse/OAK-6617 Project: Jackrabbit Oak Issue Type: Improvement Components: composite Reporter: Robert Munteanu Assignee: Robert Munteanu Priority: Minor Fix For: 1.8, 1.7.7 For logging and debugging purposes having the mount name be empty is confusing. I propose using __ as the name. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
[
https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153563#comment-16153563
]
Tomek Rękawek commented on OAK-6611:
Closer is created here:
https://github.com/apache/jackrabbit-oak/blob/a7a2bba8879746da47ccaeebafdca170493a3953/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/OakUpgrade.java#L71
The S3DataStoreFactory registers its close() method:
https://github.com/apache/jackrabbit-oak/blob/a7a2bba8879746da47ccaeebafdca170493a3953/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java#L80
Which contains the active waiting:
https://github.com/apache/jackrabbit-oak/blob/a7a2bba8879746da47ccaeebafdca170493a3953/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/blob/S3DataStoreFactory.java#L80
It's closed here:
https://github.com/apache/jackrabbit-oak/blob/a7a2bba8879746da47ccaeebafdca170493a3953/oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/OakUpgrade.java#L83
[~amitjain] - is there something missing in the active waiting method linked
above?
> [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with
> oak-upgrade
> ---
>
> Key: OAK-6611
> URL: https://issues.apache.org/jira/browse/OAK-6611
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-cloud, upgrade
>Affects Versions: 1.8, 1.7.7
>Reporter: Arek Kita
>Assignee: Tomek Rękawek
>Priority: Critical
> Fix For: 1.8, 1.7.7
>
> Attachments: oak-upgrade-oak-blob-cloud-20170905.log.gz,
> oak-upgrade-with-oak-blob-cloud.fragment.log
>
>
> [~tomek.rekawek], [~amitjain] Due to async nature of S3 datastore
> format/upload process the migration ends up way quicker than S3 datastore is
> being migrated. This leads to a huge number of exceptions shown due to *non
> synchronised* nature of *oak-upgrade* migration process vs async S3 datastore
> background processes.
> I see a few possible solutions for that:
> * disable migration/uploading of S3 cache for the time of migration (bad idea
> IMHO)
> * wait for it (it might be desired or a bad idea as it might take longer than
> migration for a few cases)
> * pause it when migration is completed in a clean way (so some binaries
> aren't uploaded and moved to a new datastore format) -- not sure if such
> mixed state is ok at all
> WDYT?
> Please also note that this happens only when {{\-\-src-s3config
> \-\-src-s3datastore}} options are specified during migration which in many
> cases is true (this would be the same for the destination DataStore options).
> Referencing a source datastore is needed (even if {{\-\-copy-binaries}} is
> not included) in example to copy checkpoints properly.
> The example exception is like the below:
> {code}
> 01.09.2017 11:39:41.088 ERROR o.a.j.o.p.b.UploadStagingCache: Error adding
> file to backend
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
>
[jira] [Commented] (OAK-5173) Path in uniqueness constraint violation exception is always the root
[
https://issues.apache.org/jira/browse/OAK-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153554#comment-16153554
]
Robert Munteanu commented on OAK-5173:
--
OAK-6578 might make this easier, as pointed out by Chetan on that issue.
> Path in uniqueness constraint violation exception is always the root
>
>
> Key: OAK-5173
> URL: https://issues.apache.org/jira/browse/OAK-5173
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, property-index
>Affects Versions: 1.4.10, 1.5.14
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.8
>
>
> OAK-1997 added a (single) path to the uniqueness constraint exception message
> in the PropertyIndexEditor to point out where the duplicate came from, but it
> is always the root:
> {noformat}
> OakConstraint0030: Uniqueness constraint violated at path [/] for one of the
> property in [rep:externalId] having value xyz1234
> {noformat}
> That is because it [uses
> getPath()|https://github.com/apache/jackrabbit-oak/blob/3a88b23d51beae4798e9b29c4deaba81c81fb427/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/property/PropertyIndexEditor.java#L315-L318]
> of the index editor itself and the uniqueness check always [happens at the
> root
> level|https://github.com/apache/jackrabbit-oak/blob/3a88b23d51beae4798e9b29c4deaba81c81fb427/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/property/PropertyIndexEditor.java#L303]
> at the end.
> It probably has to read from the index to find out the 2 or more paths with
> the same property value (just printing one path is not enough information for
> duplicates).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-6578) Enhance the UniqueEntryStoreStrategy to return list of matching values and paths
[
https://issues.apache.org/jira/browse/OAK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Munteanu resolved OAK-6578.
--
Resolution: Fixed
Fixed in [r1807343|https://svn.apache.org/r1807343]
> Enhance the UniqueEntryStoreStrategy to return list of matching values and
> paths
>
>
> Key: OAK-6578
> URL: https://issues.apache.org/jira/browse/OAK-6578
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing
>Reporter: Robert Munteanu
>Assignee: Robert Munteanu
> Fix For: 1.8, 1.7.7
>
> Attachments: OAK-6578.patch
>
>
> For OAK-6506 I would need to extract from each index a list of all property
> values and paths. The property values would be used to check for duplicates,
> while the paths will be used for reporting.
> The changes will be localized to the {{UniqueEntryStoreStrategy}}, as the
> other strategies don't make use of it for now.
> -As the IndexStoreStrategy interface and its implementations do not support
> such access, we need to provide it.-
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6578) Enhance the UniqueEntryStoreStrategy to return list of matching values and paths
[ https://issues.apache.org/jira/browse/OAK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Munteanu updated OAK-6578: - Summary: Enhance the UniqueEntryStoreStrategy to return list of matching values and paths (was: Enhance the IndexStoreStrategy to return list of matching values and paths) > Enhance the UniqueEntryStoreStrategy to return list of matching values and > paths > > > Key: OAK-6578 > URL: https://issues.apache.org/jira/browse/OAK-6578 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: indexing >Reporter: Robert Munteanu >Assignee: Robert Munteanu > Fix For: 1.8, 1.7.7 > > Attachments: OAK-6578.patch > > > For OAK-6506 I would need to extract from each index a list of all property > values and paths. The property values would be used to check for duplicates, > while the paths will be used for reporting. > As the IndexStoreStrategy interface and its implementations do not support > such access, we need to provide it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
[
https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arek Kita reopened OAK-6611:
Unfortunately it doesn't help. I'm using latest Oak 1.8-SNAPSHOT where those
changes should be already included but binaries are moved across DataStore and
uploaded in asynchronous way and the migration takes place as well (finishing
earlier than DS uploads, hence exceptions in the log).
Please have a look: [^oak-upgrade-oak-blob-cloud-20170905.log.gz]
BTW: Where to {{oak-upgrade}} active wait has been added?
/cc [~amitjain]
> [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with
> oak-upgrade
> ---
>
> Key: OAK-6611
> URL: https://issues.apache.org/jira/browse/OAK-6611
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-cloud, upgrade
>Affects Versions: 1.8, 1.7.7
>Reporter: Arek Kita
>Assignee: Tomek Rękawek
>Priority: Critical
> Fix For: 1.8, 1.7.7
>
> Attachments: oak-upgrade-oak-blob-cloud-20170905.log.gz,
> oak-upgrade-with-oak-blob-cloud.fragment.log
>
>
> [~tomek.rekawek], [~amitjain] Due to async nature of S3 datastore
> format/upload process the migration ends up way quicker than S3 datastore is
> being migrated. This leads to a huge number of exceptions shown due to *non
> synchronised* nature of *oak-upgrade* migration process vs async S3 datastore
> background processes.
> I see a few possible solutions for that:
> * disable migration/uploading of S3 cache for the time of migration (bad idea
> IMHO)
> * wait for it (it might be desired or a bad idea as it might take longer than
> migration for a few cases)
> * pause it when migration is completed in a clean way (so some binaries
> aren't uploaded and moved to a new datastore format) -- not sure if such
> mixed state is ok at all
> WDYT?
> Please also note that this happens only when {{\-\-src-s3config
> \-\-src-s3datastore}} options are specified during migration which in many
> cases is true (this would be the same for the destination DataStore options).
> Referencing a source datastore is needed (even if {{\-\-copy-binaries}} is
> not included) in example to copy checkpoints properly.
> The example exception is like the below:
> {code}
> 01.09.2017 11:39:41.088 ERROR o.a.j.o.p.b.UploadStagingCache: Error adding
> file to backend
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
> at
> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3996)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1161)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1136)
> at
> org.apache.jackrabbit.oak.blob.cloud.s3.S3Bac
[jira] [Updated] (OAK-6578) Enhance the UniqueEntryStoreStrategy to return list of matching values and paths
[
https://issues.apache.org/jira/browse/OAK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Munteanu updated OAK-6578:
-
Description:
For OAK-6506 I would need to extract from each index a list of all property
values and paths. The property values would be used to check for duplicates,
while the paths will be used for reporting.
The changes will be localized to the {{UniqueEntryStoreStrategy}}, as the other
strategies don't make use of it for now.
-As the IndexStoreStrategy interface and its implementations do not support
such access, we need to provide it.-
was:
For OAK-6506 I would need to extract from each index a list of all property
values and paths. The property values would be used to check for duplicates,
while the paths will be used for reporting.
As the IndexStoreStrategy interface and its implementations do not support such
access, we need to provide it.
> Enhance the UniqueEntryStoreStrategy to return list of matching values and
> paths
>
>
> Key: OAK-6578
> URL: https://issues.apache.org/jira/browse/OAK-6578
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing
>Reporter: Robert Munteanu
>Assignee: Robert Munteanu
> Fix For: 1.8, 1.7.7
>
> Attachments: OAK-6578.patch
>
>
> For OAK-6506 I would need to extract from each index a list of all property
> values and paths. The property values would be used to check for duplicates,
> while the paths will be used for reporting.
> The changes will be localized to the {{UniqueEntryStoreStrategy}}, as the
> other strategies don't make use of it for now.
> -As the IndexStoreStrategy interface and its implementations do not support
> such access, we need to provide it.-
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6616) Update Oak trunk to Jackrabbit 2.15.6
Julian Reschke created OAK-6616: --- Summary: Update Oak trunk to Jackrabbit 2.15.6 Key: OAK-6616 URL: https://issues.apache.org/jira/browse/OAK-6616 Project: Jackrabbit Oak Issue Type: Task Reporter: Julian Reschke Assignee: Julian Reschke Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6611) [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
[
https://issues.apache.org/jira/browse/OAK-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arek Kita updated OAK-6611:
---
Attachment: oak-upgrade-oak-blob-cloud-20170905.log.gz
> [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with
> oak-upgrade
> ---
>
> Key: OAK-6611
> URL: https://issues.apache.org/jira/browse/OAK-6611
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-cloud, upgrade
>Affects Versions: 1.8, 1.7.7
>Reporter: Arek Kita
>Assignee: Tomek Rękawek
>Priority: Critical
> Fix For: 1.8, 1.7.7
>
> Attachments: oak-upgrade-oak-blob-cloud-20170905.log.gz,
> oak-upgrade-with-oak-blob-cloud.fragment.log
>
>
> [~tomek.rekawek], [~amitjain] Due to async nature of S3 datastore
> format/upload process the migration ends up way quicker than S3 datastore is
> being migrated. This leads to a huge number of exceptions shown due to *non
> synchronised* nature of *oak-upgrade* migration process vs async S3 datastore
> background processes.
> I see a few possible solutions for that:
> * disable migration/uploading of S3 cache for the time of migration (bad idea
> IMHO)
> * wait for it (it might be desired or a bad idea as it might take longer than
> migration for a few cases)
> * pause it when migration is completed in a clean way (so some binaries
> aren't uploaded and moved to a new datastore format) -- not sure if such
> mixed state is ok at all
> WDYT?
> Please also note that this happens only when {{\-\-src-s3config
> \-\-src-s3datastore}} options are specified during migration which in many
> cases is true (this would be the same for the destination DataStore options).
> Referencing a source datastore is needed (even if {{\-\-copy-binaries}} is
> not included) in example to copy checkpoints properly.
> The example exception is like the below:
> {code}
> 01.09.2017 11:39:41.088 ERROR o.a.j.o.p.b.UploadStagingCache: Error adding
> file to backend
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
> at
> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3996)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1161)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1136)
> at
> org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.write(S3Backend.java:201)
> at
> org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore$2.write(AbstractSharedCachingDataStore.java:170)
> at
> org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.java:341)
> at
> org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.j
[jira] [Commented] (OAK-6575) Provide a secure external URL to a DataStore binary.
[
https://issues.apache.org/jira/browse/OAK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153541#comment-16153541
]
Ian Boston commented on OAK-6575:
-
After discussion on oak-dev, 2 more alternative patches.
https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-1?expand=1
This drops the OSGi AdapterManager/AdapterFactory in favour of a non OSGi
static pattern. Implementations of the AdapterFactory self register rather than
rely on OSGi doing the wiring. This is probably an IoC anti pattern, but does
avoid exposing the AdapterFactory/AdapterManager outside Oak.
https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-2?expand=1
This drops the AdapterManager concept completely and attempts to get from Value
to URI using mix in interfaces and instanceof. I cant be certain it manages to
do this as there is a disconnect between Blob, Blobstore and DataStore
implementations with no guarantee that a BlobStore as seen by the Blob
implementation actually implements DataStore, or the Blob that is exposed in
the JCR Value (implemented by OakValue) actually connects to the correct
DataStore of it it connects to a FileDatastore cache on local disk. I could
only wire this as far as I did with API changes. I may have broken some of the
new multi node store and multi datastore code used for 0DT in the process. An
Oak committer with global knowledge will probably be able to do better.
> Provide a secure external URL to a DataStore binary.
>
>
> Key: OAK-6575
> URL: https://issues.apache.org/jira/browse/OAK-6575
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: blob, core, jcr
>Reporter: Ian Boston
> Fix For: 1.8
>
>
> Where the DataStore is a DataStore that may be accessed over an independent
> API it would be advantageous for Oak to provide a secure URL to allow direct,
> read only access to the current immutable instance of that binary. The term
> "secure" needs to be defined, but typically it would a URL that is valid for
> a appropriately short length of time to ensure that the risk of the URL being
> used by a user that it was not intended for, is minimised. It should also
> ensure that anyone in possession of the URL could not use the information in
> the url to create a valid URL or a valid URL to a different binary.
> One example of such a URL might be a AWS Signed URL as used by AWS CloudFront
> to access private content. The signed url being signed by a private key known
> only to the Oak instance and the the CloudFront or S3 instance. The signed
> url having a significantly low ttl so that a redirect by the same client
> would work.
> Oak should only emit these URLs to sessions that could otherwise read the
> binary directly from Oak, and Oak should be in complete control of the nature
> of the url and the security mechanisms applied to the URL.
> The viability of the approach has been investigated showing that given a JCR
> Binary it is possible to get the Oak Blob Content Identifier using
> ValueImpl.getBlob((Value)jcrBinary).getContentIentifier() and form there,
> knowing the way in which the DataStore implementation transforms that into a
> pointer into the datastore implementation form a URL to be made secure.
> To achieve the above, internal implementation details specific to the Oak
> DataStore implementation are required, hence this request to implement as a
> part of Oak rather than to reverse engineer in some external project.
> Since API changes are often significant using the Sling AdapaterFactory
> approach would allow a ServletFilter to selectively use the URL in a
> redirect, avoiding any new API methods to existing Oak APIs. A new interface
> might be required, in the example below that interface is SignedBinaryURL.
> {code}
> public void doFilter(ServletRequest servletRequest, ServletResponse
> servletResponse, FilterChain filterChain) throws IOException,
> ServletException {
> if ( servletRequest instanceof SlingHttpServletRequest &&
> servletResponse instanceof SlingHttpServletResponse) {
> if ("GET".equals(((SlingHttpServletRequest)
> servletRequest).getMethod())){
> Resource resource = ((SlingHttpServletRequest)
> servletRequest).getResource();
> SignedBinaryURL url = resource.adaptTo(SignedBinaryURL.class);
> if (url != null) {
> ((SlingHttpServletResponse)
> servletResponse).sendRedirect(url.getURL());
> return;
> }
> }
> }
> filterChain.doFilter(servletRequest, servletResponse);
> }
> {code}
> If the AdapterFactory to go from Binary to SingedBinaryURL is not present
> then url will always be null, and no-op. If it is pr
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153540#comment-16153540
]
Michael Dürig commented on OAK-6584:
What is our position regarding equality? I'm thinking mainly of {{Node}} and
{{Property}} here:
# Should we mandate implementers to provide an implementation for {{equals()}}?
# Alternatively, we could leave this to the API consumers having to come up
with their own implementation of structural equality.
# Or we provide a utility class with methods for determining structural
equality.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6615) Add --cold-standby [n] option for Oak-Segment-Tar* fixtures in benchmark mode
Andrei Dulceanu created OAK-6615:
Summary: Add --cold-standby [n] option for Oak-Segment-Tar*
fixtures in benchmark mode
Key: OAK-6615
URL: https://issues.apache.org/jira/browse/OAK-6615
Project: Jackrabbit Oak
Issue Type: Improvement
Components: benchmarks, segment-tar
Reporter: Andrei Dulceanu
Assignee: Andrei Dulceanu
Priority: Minor
Fix For: 1.8, 1.7.7
It would be nice to have a new option for {{benchmark}} mode in
{{oak-benchmarks}}, {{--cold-standby [n]}}. This will start a cold standby
instance, syncing with the primary every {{n}} seconds. All the benchmarks
specified via {{[testcases]}} argument will be run on primary instance, and all
statistics and reports will be linked to primary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153509#comment-16153509
]
Michael Dürig commented on OAK-6584:
re. 9. (consumer instantiation of {{RecordId}}) and 10. (usefulness of
{{Store.cast()}}
These two are dual to each other as they allow transitioning between the node
store entities to the tooling entities and vice versa. So we should remove/keep
them together. I agree there is a tension here between encapsulation and
utility. Since the intended consumers of the API is for tooling I leaned toward
the latter. The way I look at this is that the cast method enables users to do
the ~10% of things they cannot do through the tool API itself, allowing us to
keep the API simple, easy to implement and easy to maintain and at the same
time enabling consumers to "hack their way" through if need be. From a
historical point of view this is already a great improvement as in my [existing
tooling|https://github.com/mduerig/script-oak] I have to hack my way through
all the time in much worse ways.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153442#comment-16153442
]
Francesco Mari commented on OAK-6584:
-
bq. The closest you could get is ... which is awkward to me.
That's right. It looks pretty awkward. Let's keep {{NULL_NODE}} and
{{NULL_PROPERTY}}.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153413#comment-16153413
]
Michael Dürig commented on OAK-6584:
re. 1., 7. and 8. ({{Optional}} instead of {{NULL_NODE}}).
I tried using {{Optional}} but it prevents using the API in fluent style
like {{Property x = root.node("a").node("aa").property("x")}}. The closest you
could get is
{code}
Optional x = root
.flatMap(n -> n.node("a"))
.flatMap(n -> n. node("aa"))
.map(n -> n.property("x"));
{code}
which is awkward to me.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[ https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153405#comment-16153405 ] Francesco Mari commented on OAK-6584: - bq. I'd prefer to leave it as is for now as I think most consumers of this would be humans where we'd end up converting the dump to a string anyway. Also I somewhat dislike relying on side effects if we'd make this return void. Sounds legit. > Add tooling API > --- > > Key: OAK-6584 > URL: https://issues.apache.org/jira/browse/OAK-6584 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: tooling > Fix For: 1.8 > > > h3. Current situation > Current segment store related tools are implemented ad-hoc by potentially > relying on internal implementation details of Oak Segment Tar. This makes > those tools less useful, portable, stable and potentially applicable than > they should be. > h3. Goal > Provide a common and sufficiently stable Oak Tooling API for implementing > segment store related tools. The API should be independent of Oak and not > available for normal production use of Oak. Specifically it should not be > possible to it to implement production features and production features must > not rely on it. It must be possible to implement the Oak Tooling API in Oak > 1.8 and it should be possible for Oak 1.6. > h3. Typical use cases > * Query the number of nodes / properties / values in a given path satisfying > some criteria > * Aggregate a certain value on queries like the above > * Calculate size of the content / size on disk > * Analyse changes. E.g. how many binaries bigger than a certain threshold > were added / removed between two given revisions. What is the sum of their > sizes? > * Analyse locality: measure of locality of node states. Incident plots (See > https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973). > * Analyse level of deduplication (e.g. of checkpoint) > h3. Validation > Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the > tooling API. > h3. API draft > * Whiteboard shot of the [API > entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg] > identified initially. > * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api] > takes place on Github for now. We'll move to the Apache SVN as soon as > considered mature enough and have a consensus of where to best move it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[ https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153400#comment-16153400 ] Michael Dürig commented on OAK-6584: bq. 4. I would like Segment#hexDump more if it would accept an OutputStream and return void instead, I'd prefer to leave it as is for now as I think most consumers of this would be humans where we'd end up converting the dump to a string anyway. Also I somewhat dislike relying on side effects if we'd make this return void. > Add tooling API > --- > > Key: OAK-6584 > URL: https://issues.apache.org/jira/browse/OAK-6584 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: tooling > Fix For: 1.8 > > > h3. Current situation > Current segment store related tools are implemented ad-hoc by potentially > relying on internal implementation details of Oak Segment Tar. This makes > those tools less useful, portable, stable and potentially applicable than > they should be. > h3. Goal > Provide a common and sufficiently stable Oak Tooling API for implementing > segment store related tools. The API should be independent of Oak and not > available for normal production use of Oak. Specifically it should not be > possible to it to implement production features and production features must > not rely on it. It must be possible to implement the Oak Tooling API in Oak > 1.8 and it should be possible for Oak 1.6. > h3. Typical use cases > * Query the number of nodes / properties / values in a given path satisfying > some criteria > * Aggregate a certain value on queries like the above > * Calculate size of the content / size on disk > * Analyse changes. E.g. how many binaries bigger than a certain threshold > were added / removed between two given revisions. What is the sum of their > sizes? > * Analyse locality: measure of locality of node states. Incident plots (See > https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973). > * Analyse level of deduplication (e.g. of checkpoint) > h3. Validation > Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the > tooling API. > h3. API draft > * Whiteboard shot of the [API > entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg] > identified initially. > * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api] > takes place on Github for now. We'll move to the Apache SVN as soon as > considered mature enough and have a consensus of where to best move it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153392#comment-16153392
]
Michael Dürig commented on OAK-6584:
At
https://github.com/mduerig/oak-tooling-api/commit/1f9b4454ecdaa6b8a753eabc97bd18c5169a27ce
I remove the {{read()}} methods on {{Segment}} and {{Record}}. I agree with
your concern regarding having the consumer to specify the count. Let's rethink
this once we tackle record inspection or once we have a case for it.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[ https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153370#comment-16153370 ] Michael Dürig commented on OAK-6584: bq. 3. Segment#header looks underspecified. Segment headers are quite easy to model as a proper interface. I think we should go this way. I was considering this as well but wanted to keep it (too?) simple for now. Especially regarding evolution of the segment format, which would introduce API compatibility concerns. I added a TODO to that respect at https://github.com/mduerig/oak-tooling-api/commit/8a6bf3fc44d20a1d47f89bd60daa19b90da1bc3d > Add tooling API > --- > > Key: OAK-6584 > URL: https://issues.apache.org/jira/browse/OAK-6584 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: tooling > Fix For: 1.8 > > > h3. Current situation > Current segment store related tools are implemented ad-hoc by potentially > relying on internal implementation details of Oak Segment Tar. This makes > those tools less useful, portable, stable and potentially applicable than > they should be. > h3. Goal > Provide a common and sufficiently stable Oak Tooling API for implementing > segment store related tools. The API should be independent of Oak and not > available for normal production use of Oak. Specifically it should not be > possible to it to implement production features and production features must > not rely on it. It must be possible to implement the Oak Tooling API in Oak > 1.8 and it should be possible for Oak 1.6. > h3. Typical use cases > * Query the number of nodes / properties / values in a given path satisfying > some criteria > * Aggregate a certain value on queries like the above > * Calculate size of the content / size on disk > * Analyse changes. E.g. how many binaries bigger than a certain threshold > were added / removed between two given revisions. What is the sum of their > sizes? > * Analyse locality: measure of locality of node states. Incident plots (See > https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973). > * Analyse level of deduplication (e.g. of checkpoint) > h3. Validation > Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the > tooling API. > h3. API draft > * Whiteboard shot of the [API > entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg] > identified initially. > * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api] > takes place on Github for now. We'll move to the Apache SVN as soon as > considered mature enough and have a consensus of where to best move it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153354#comment-16153354
]
Francesco Mari commented on OAK-6584:
-
I agree on removing {{Segment#read}} and {{Record#read}}. I am skeptic about
adding {{Record#hexDump}}, because the API doesn't currently provide enough
context to safely estimate the number of bytes to dump. Before we add
{{Record#hexDump}}, we should implement proper low-level record inspection in
the API. It would be more practical if callers wouldn't be forced to pass a
{{count}} parameter {{Record#hexDump}}.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153348#comment-16153348
]
Michael Dürig edited comment on OAK-6584 at 9/5/17 9:21 AM:
bq. 2. think that Segment#read is too powerful but not as useful. This method
allows tooling to create their own segment parsers, which will never be as
complete as the one provided by the Segment Store.
My thinking was that in certain involved debugging cases it would be useful to
have direct access to the segment data. However the hexdump is probably more
useful here. Should we thus remove that method? If so we should probably also
remove {{Record#read()}}. However this would leave us with no insight into
records. Could we add a {{Record#hexDump(count)}} instead?
was (Author: mduerig):
bq. I think that Segment#read is too powerful but not as useful. This method
allows tooling to create their own segment parsers, which will never be as
complete as the one provided by the Segment Store.
My thinking was that in certain involved debugging cases it would be useful to
have direct access to the segment data. However the hexdump is probably more
useful here. Should we thus remove that method? If so we should probably also
remove {{Record#read()}}. However this would leave us with no insight into
records. Could we add a {{Record#hexDump(count)}} instead?
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153346#comment-16153346
]
Francesco Mari commented on OAK-6584:
-
I haven't considered the fact that {{Node}} instances don't have a name. I
guess we can safely ignore #6.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153348#comment-16153348
]
Michael Dürig commented on OAK-6584:
bq. I think that Segment#read is too powerful but not as useful. This method
allows tooling to create their own segment parsers, which will never be as
complete as the one provided by the Segment Store.
My thinking was that in certain involved debugging cases it would be useful to
have direct access to the segment data. However the hexdump is probably more
useful here. Should we thus remove that method? If so we should probably also
remove {{Record#read()}}. However this would leave us with no insight into
records. Could we add a {{Record#hexDump(count)}} instead?
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153338#comment-16153338
]
Michael Dürig commented on OAK-6584:
bq. 6. Node is not symmetric with regard to nodes and properties. Why is there
{{Node#getNodeNames}} but not a {{Node#getPropertyNames}}?
Because the property names can easily be derived by mapping
{{Property::getName}} over {{properties()}}. This is not possible in the case
of {{Node}} as those (and {{NodeState}} instances do not have a name). If there
is a preference we can add a {{Node#getPropertyNames}} method as convenience
for the former.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[ https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153329#comment-16153329 ] Michael Dürig commented on OAK-6584: bq. 5. I would rename Node#getNodeNames and Node#getNodes to Node#childNames and Node#children respectively Done at https://github.com/mduerig/oak-tooling-api/commit/01d5b9a94d4592ba9c6be8ae764c8a5813d835f9 > Add tooling API > --- > > Key: OAK-6584 > URL: https://issues.apache.org/jira/browse/OAK-6584 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: tooling > Fix For: 1.8 > > > h3. Current situation > Current segment store related tools are implemented ad-hoc by potentially > relying on internal implementation details of Oak Segment Tar. This makes > those tools less useful, portable, stable and potentially applicable than > they should be. > h3. Goal > Provide a common and sufficiently stable Oak Tooling API for implementing > segment store related tools. The API should be independent of Oak and not > available for normal production use of Oak. Specifically it should not be > possible to it to implement production features and production features must > not rely on it. It must be possible to implement the Oak Tooling API in Oak > 1.8 and it should be possible for Oak 1.6. > h3. Typical use cases > * Query the number of nodes / properties / values in a given path satisfying > some criteria > * Aggregate a certain value on queries like the above > * Calculate size of the content / size on disk > * Analyse changes. E.g. how many binaries bigger than a certain threshold > were added / removed between two given revisions. What is the sum of their > sizes? > * Analyse locality: measure of locality of node states. Incident plots (See > https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973). > * Analyse level of deduplication (e.g. of checkpoint) > h3. Validation > Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the > tooling API. > h3. API draft > * Whiteboard shot of the [API > entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg] > identified initially. > * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api] > takes place on Github for now. We'll move to the Apache SVN as soon as > considered mature enough and have a consensus of where to best move it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153266#comment-16153266
]
Francesco Mari edited comment on OAK-6584 at 9/5/17 9:02 AM:
-
I had a first look at the API. Here are some comments.
# Why {{Store#node}} does not return an {{Optional}}?
# I think that {{Segment#read}} is too powerful but not as useful. This method
allows tooling to create their own segment parsers, which will never be as
complete as the one provided by the Segment Store.
# {{Segment#header}} looks underspecified. Segment headers are quite easy to
model as a proper interface. I think we should go this way.
# I would like {{Segment#hexDump}} more if it would accept an OutputStream and
return void instead, more like {{o.a.j.o.segment.data.SegmentData#hexDump}}. It
might be a matter of personal taste, though.
# I would rename {{Node#getNodeNames}} and {{Node#getNodes}} to
{{Node#childNames}} and {{Node#children}} respectively.
# {{Node}} is not symmetric with regard to nodes and properties. Why is there
{{Node#getNodeNames}} but not a {{Node#getPropertyNames}}?
# Should {{Node#node}} and {{Node#property}} return {{Optional}} instancesd
instead?
# {{Node#NULL_NODE}} and {{Property#NULL_PROPERTY}} could disappear if we
returned {{Optional}} instances consistently.
# I'm not sure if we should allow users of the API to instantiate new
{{RecordId}} instances. Users should be able to navigate the contents of the
Segment Store without crafting {{RecordId}} instances of their own.
# I don't see the usefulness of {{Store#cast}}. For a user to gain something
from this method, he should have a dependency on both the API and a concrete
implementation. This defeats the abstraction layer. In such a situation why not
using the concrete implementation directly? I would remove this method from the
API. If the API has shortcomings that are currently addressable only via
{{Store#cast}}, we should take care of them. {{Store#cast}} is too easy to
misuse.
was (Author: frm):
I had a first look at the API. Here are some comments.
# Why {{Store#node}} does not return an {{Optional}}?
# I think that {{Segment#read}} is too powerful but not as useful. This method
allows tooling to create their own segment parsers, which will never be as
complete as the one provided by the Segment Store.
# {{Segment#header}} looks underspecified. Segment headers are quite easy to
model as a proper interface. I think we should go this way.
# I would like {{Segment#hexDump}} more if it would accept an OutputStream and
return void instead, more like {{o.a.j.o.segment.data.SegmentData#hexDump}}. It
might be a matter of personal taste, though.
# I would rename {{Node#getNodeNames}} and {{Node#getNodes}} to
{{Node#childNames}} and {{Node#children}} respectively.
# Node is not symmetric with regard to nodes and properties. Why is there
{{Node#getNodeNames}} but not a {{Node#getPropertyNames}}?
# Should {{Node#node}} and {{Node#property}} return {{Optional}} instancesd
instead?
# {{Node#NULL_NODE}} and {{Property#NULL_PROPERTY}} could disappear if we
returned {{Optional}} instances consistently.
# I'm not sure if we should allow users of the API to instantiate new
{{RecordId}} instances. Users should be able to navigate the contents of the
Segment Store without crafting {{RecordId}} instances of their own.
# I don't see the usefulness of {{Store#cast}}. For a user to gain something
from this method, he should have a dependency on both the API and a concrete
implementation. This defeats the abstraction layer. In such a situation why not
using the concrete implementation directly? I would remove this method from the
API. If the API has shortcomings that are currently addressable only via
{{Store#cast}}, we should take care of them. {{Store#cast}} is too easy to
misuse.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling
[jira] [Resolved] (OAK-6614) Add ability to add 'excludeFromAggregation' setting while building index definition
[
https://issues.apache.org/jira/browse/OAK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-6614.
Resolution: Fixed
Fix Version/s: 1.7.7
Thanks [~satyadeep] for the patch. I've applied it on trunk at
[r1807323|https://svn.apache.org/r1807323] with a minor modification of method
name to reflect the actual property name being set on index def
({{excludeFromAggregation}}).
> Add ability to add 'excludeFromAggregation' setting while building index
> definition
> ---
>
> Key: OAK-6614
> URL: https://issues.apache.org/jira/browse/OAK-6614
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.7.6
>Reporter: Satya Deep Maheshwari
>Assignee: Vikas Saurabh
> Fix For: 1.8, 1.7.7
>
> Attachments: OAK-6614.patch
>
>
> Currently there's isn't a method available in {{IndexDefinitionBuilder}} to
> add 'excludeFromAggregation' in index definition. This is needed when we need
> to enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6614) Add ability to add 'excludeFromAggregation' setting while building index definition
[
https://issues.apache.org/jira/browse/OAK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-6614:
---
Component/s: (was: indexing)
lucene
> Add ability to add 'excludeFromAggregation' setting while building index
> definition
> ---
>
> Key: OAK-6614
> URL: https://issues.apache.org/jira/browse/OAK-6614
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.7.6
>Reporter: Satya Deep Maheshwari
>Assignee: Vikas Saurabh
> Fix For: 1.8
>
> Attachments: OAK-6614.patch
>
>
> Currently there's isn't a method available in {{IndexDefinitionBuilder}} to
> add 'excludeFromAggregation' in index definition. This is needed when we need
> to enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6614) Add ability to add 'excludeFromAggregation' setting while building index definition
[
https://issues.apache.org/jira/browse/OAK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-6614:
---
Fix Version/s: 1.8
> Add ability to add 'excludeFromAggregation' setting while building index
> definition
> ---
>
> Key: OAK-6614
> URL: https://issues.apache.org/jira/browse/OAK-6614
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.7.6
>Reporter: Satya Deep Maheshwari
>Assignee: Vikas Saurabh
> Fix For: 1.8
>
> Attachments: OAK-6614.patch
>
>
> Currently there's isn't a method available in {{IndexDefinitionBuilder}} to
> add 'excludeFromAggregation' in index definition. This is needed when we need
> to enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OAK-6614) Add ability to add 'excludeFromAggregation' setting while building index definition
[
https://issues.apache.org/jira/browse/OAK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh reassigned OAK-6614:
--
Assignee: Vikas Saurabh
> Add ability to add 'excludeFromAggregation' setting while building index
> definition
> ---
>
> Key: OAK-6614
> URL: https://issues.apache.org/jira/browse/OAK-6614
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing
>Affects Versions: 1.7.6
>Reporter: Satya Deep Maheshwari
>Assignee: Vikas Saurabh
> Attachments: OAK-6614.patch
>
>
> Currently there's isn't a method available in {{IndexDefinitionBuilder}} to
> add 'excludeFromAggregation' in index definition. This is needed when we need
> to enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6614) Add ability to add 'excludeFromAggregation' setting while building index definition
[
https://issues.apache.org/jira/browse/OAK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satya Deep Maheshwari updated OAK-6614:
---
Attachment: OAK-6614.patch
Attaching a proposed patch [^OAK-6614.patch] which addresses this.
[~chetanm], [~catholicon], could you please help resolve this?
> Add ability to add 'excludeFromAggregation' setting while building index
> definition
> ---
>
> Key: OAK-6614
> URL: https://issues.apache.org/jira/browse/OAK-6614
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing
>Affects Versions: 1.7.6
>Reporter: Satya Deep Maheshwari
> Attachments: OAK-6614.patch
>
>
> Currently there's isn't a method available in {{IndexDefinitionBuilder}} to
> add 'excludeFromAggregation' in index definition. This is needed when we need
> to enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6614) Add ability to add 'excludeFromAggregation' setting while building index definition
[
https://issues.apache.org/jira/browse/OAK-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satya Deep Maheshwari updated OAK-6614:
---
Summary: Add ability to add 'excludeFromAggregation' setting while building
index definition (was: Add ability to add 'excludeFromAggregation' property
while building index definition)
> Add ability to add 'excludeFromAggregation' setting while building index
> definition
> ---
>
> Key: OAK-6614
> URL: https://issues.apache.org/jira/browse/OAK-6614
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing
>Affects Versions: 1.7.6
>Reporter: Satya Deep Maheshwari
>
> Currently there's isn't a method available in {{IndexDefinitionBuilder}} to
> add 'excludeFromAggregation' in index definition. This is needed when we need
> to enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6614) Add ability to add 'excludeFromAggregation' property while building index definition
Satya Deep Maheshwari created OAK-6614:
--
Summary: Add ability to add 'excludeFromAggregation' property
while building index definition
Key: OAK-6614
URL: https://issues.apache.org/jira/browse/OAK-6614
Project: Jackrabbit Oak
Issue Type: Improvement
Components: indexing
Affects Versions: 1.7.6
Reporter: Satya Deep Maheshwari
Currently there's isn't a method available in {{IndexDefinitionBuilder}} to add
'excludeFromAggregation' in index definition. This is needed when we need to
enable this setting while specifying the index definition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6584) Add tooling API
[
https://issues.apache.org/jira/browse/OAK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153266#comment-16153266
]
Francesco Mari commented on OAK-6584:
-
I had a first look at the API. Here are some comments.
# Why {{Store#node}} does not return an {{Optional}}?
# I think that {{Segment#read}} is too powerful but not as useful. This method
allows tooling to create their own segment parsers, which will never be as
complete as the one provided by the Segment Store.
# {{Segment#header}} looks underspecified. Segment headers are quite easy to
model as a proper interface. I think we should go this way.
# I would like {{Segment#hexDump}} more if it would accept an OutputStream and
return void instead, more like {{o.a.j.o.segment.data.SegmentData#hexDump}}. It
might be a matter of personal taste, though.
# I would rename {{Node#getNodeNames}} and {{Node#getNodes}} to
{{Node#childNames}} and {{Node#children}} respectively.
# Node is not symmetric with regard to nodes and properties. Why is there
{{Node#getNodeNames}} but not a {{Node#getPropertyNames}}?
# Should {{Node#node}} and {{Node#property}} return {{Optional}} instancesd
instead?
# {{Node#NULL_NODE}} and {{Property#NULL_PROPERTY}} could disappear if we
returned {{Optional}} instances consistently.
# I'm not sure if we should allow users of the API to instantiate new
{{RecordId}} instances. Users should be able to navigate the contents of the
Segment Store without crafting {{RecordId}} instances of their own.
# I don't see the usefulness of {{Store#cast}}. For a user to gain something
from this method, he should have a dependency on both the API and a concrete
implementation. This defeats the abstraction layer. In such a situation why not
using the concrete implementation directly? I would remove this method from the
API. If the API has shortcomings that are currently addressable only via
{{Store#cast}}, we should take care of them. {{Store#cast}} is too easy to
misuse.
> Add tooling API
> ---
>
> Key: OAK-6584
> URL: https://issues.apache.org/jira/browse/OAK-6584
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: tooling
> Fix For: 1.8
>
>
> h3. Current situation
> Current segment store related tools are implemented ad-hoc by potentially
> relying on internal implementation details of Oak Segment Tar. This makes
> those tools less useful, portable, stable and potentially applicable than
> they should be.
> h3. Goal
> Provide a common and sufficiently stable Oak Tooling API for implementing
> segment store related tools. The API should be independent of Oak and not
> available for normal production use of Oak. Specifically it should not be
> possible to it to implement production features and production features must
> not rely on it. It must be possible to implement the Oak Tooling API in Oak
> 1.8 and it should be possible for Oak 1.6.
> h3. Typical use cases
> * Query the number of nodes / properties / values in a given path satisfying
> some criteria
> * Aggregate a certain value on queries like the above
> * Calculate size of the content / size on disk
> * Analyse changes. E.g. how many binaries bigger than a certain threshold
> were added / removed between two given revisions. What is the sum of their
> sizes?
> * Analyse locality: measure of locality of node states. Incident plots (See
> https://issues.apache.org/jira/browse/OAK-5655?focusedCommentId=15865973&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15865973).
> * Analyse level of deduplication (e.g. of checkpoint)
> h3. Validation
> Reimplement [Script Oak|https://github.com/mduerig/script-oak] on top of the
> tooling API.
> h3. API draft
> * Whiteboard shot of the [API
> entities|https://wiki.apache.org/jackrabbit/Oakathon%20August%202017?action=AttachFile&do=view&target=IMG_20170822_163256.jpg]
> identified initially.
> * Further [drafting of the API|https://github.com/mduerig/oak-tooling-api]
> takes place on Github for now. We'll move to the Apache SVN as soon as
> considered mature enough and have a consensus of where to best move it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-2710) Utils.unshareString should not generate String instances unless needed
[ https://issues.apache.org/jira/browse/OAK-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153233#comment-16153233 ] Thomas Mueller commented on OAK-2710: - [~rombert] sure, this method can be removed. > Utils.unshareString should not generate String instances unless needed > -- > > Key: OAK-2710 > URL: https://issues.apache.org/jira/browse/OAK-2710 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Affects Versions: 1.1.7 >Reporter: Robert Munteanu >Priority: Minor > Attachments: OAK-2710.patch > > > org.apache.jackrabbit.oak.plugins.document.util.Utils.unshareString creates > new String instances unconditionally. Howewer, we can avoid this String > creation if the JRE version is recent enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
