[jira] [Commented] (OAK-7303) Build Jackrabbit Oak #1282 failed
[ https://issues.apache.org/jira/browse/OAK-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382326#comment-16382326 ] Hudson commented on OAK-7303: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #1283|https://builds.apache.org/job/Jackrabbit%20Oak/1283/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1283/console] > Build Jackrabbit Oak #1282 failed > - > > Key: OAK-7303 > URL: https://issues.apache.org/jira/browse/OAK-7303 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1282 has failed. > First failed run: [Jackrabbit Oak > #1282|https://builds.apache.org/job/Jackrabbit%20Oak/1282/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1282/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382209#comment-16382209 ] Julian Reschke commented on OAK-7182: - [~mduerig] - is the question: "how do we make sure we *stay* compatible with 20.0?" (or whatever we plan to support)? I guess we'd need an build profile for that, and a build server testing that consistently... > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: OAK-7182-guava-21.diff, guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7303) Build Jackrabbit Oak #1282 failed
Hudson created OAK-7303: --- Summary: Build Jackrabbit Oak #1282 failed Key: OAK-7303 URL: https://issues.apache.org/jira/browse/OAK-7303 Project: Jackrabbit Oak Issue Type: Bug Components: continuous integration Reporter: Hudson No description is provided The build Jackrabbit Oak #1282 has failed. First failed run: [Jackrabbit Oak #1282|https://builds.apache.org/job/Jackrabbit%20Oak/1282/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1282/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382170#comment-16382170 ] Julian Reschke commented on OAK-7282: - trunk: [r1825655|http://svn.apache.org/r1825655] > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.9.0, 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke resolved OAK-7282. - Resolution: Fixed Fix Version/s: 1.9.0 > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.9.0, 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7299) RDB*Store: update postgresql JDBC driver reference to 42.2.1
[ https://issues.apache.org/jira/browse/OAK-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382164#comment-16382164 ] Julian Reschke commented on OAK-7299: - trunk: [r1825654|http://svn.apache.org/r1825654] > RDB*Store: update postgresql JDBC driver reference to 42.2.1 > > > Key: OAK-7299 > URL: https://issues.apache.org/jira/browse/OAK-7299 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Labels: candidate_oak_1_8 > Fix For: 1.9.0, 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382161#comment-16382161 ] Michael Dürig commented on OAK-7182: [~reschke], how do we ensure we don't re-introduce references to entities that were removed in some of the Guava versions we plan to support? > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: OAK-7182-guava-21.diff, guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7299) RDB*Store: update postgresql JDBC driver reference to 42.2.1
[ https://issues.apache.org/jira/browse/OAK-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-7299: Labels: candidate_oak_1_8 (was: ) > RDB*Store: update postgresql JDBC driver reference to 42.2.1 > > > Key: OAK-7299 > URL: https://issues.apache.org/jira/browse/OAK-7299 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Labels: candidate_oak_1_8 > Fix For: 1.9.0, 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7299) RDB*Store: update postgresql JDBC driver reference to 42.2.1
[ https://issues.apache.org/jira/browse/OAK-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke resolved OAK-7299. - Resolution: Fixed Fix Version/s: 1.9.0 > RDB*Store: update postgresql JDBC driver reference to 42.2.1 > > > Key: OAK-7299 > URL: https://issues.apache.org/jira/browse/OAK-7299 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.9.0, 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7300) Lucene Index: per-column selectivity to improve cost estimation
[
https://issues.apache.org/jira/browse/OAK-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382150#comment-16382150
]
Vikas Saurabh commented on OAK-7300:
[~tmueller],
bq. It would be good to be able to set the selectivity per property, for
example by specifying the number of distinct values,.
I think OAK-6735 utilizes PropertyDefinition::weight property with this
semantics. {{IndexPlanner#getMaxPossibleNumDocs}} scales down
{{numDocsForField}} by {{weight}} - so, if field has 100 documents and weight
is 20, then cost for the field would come out as 5.
{{IndexPlannerTest#weightedPropDefs}} (and a few below it) test that scenario.
That said, I think we _might_ have a bug with this estimation with {{IS NOT
NULL}} type case i.e. I'm not sure if things are ok with {{notNullCheckEnabed}}
config and our estimates are what we expect.
bq. ... or (better yet) the average number of entries for a given key (1 for
unique values, 2 meaning for each distinct values there are two documents on
average).
Yeah, that would have been better - but do you think a LARGE weight would
essentially estimate similar to almost unique values.
bq. if Apache Lucene has a built-in cardinality estimator?
sorry, I don't know :(.
> Lucene Index: per-column selectivity to improve cost estimation
> ---
>
> Key: OAK-7300
> URL: https://issues.apache.org/jira/browse/OAK-7300
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
>Priority: Major
> Fix For: 1.10
>
>
> In OAK-6735 we have improved cost estimation for Lucene indexes, however the
> following case is still not working as expected: a very common property is
> indexes (many nodes have that property), and each value of that property is
> more or less unique. In this case, currently the cost estimation is the total
> number of documents that contain that property. Assuming the condition
> "property is not null" this is correct, however for the common case "property
> = x" the estimated cost is far too high.
> A known workaround is to set the "costPerEntry" for the given index to a low
> value, for example 0.2. However this isn't a good solution, as it affects all
> properties and queries.
> It would be good to be able to set the selectivity per property, for example
> by specifying the number of distinct values, or (better yet) the average
> number of entries for a given key (1 for unique values, 2 meaning for each
> distinct values there are two documents on average).
> That value can be set manually (cost override), and it can be set
> automatically, e.g. when building the index, or updated from time to time
> during the index update, using a cardinality
> estimation algorithm. That doesn't have to be accurate; we could use an rough
> approximation such as hyperbitbit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382137#comment-16382137 ] Julian Reschke commented on OAK-7182: - FWIW, the latest changes for OAK-7188 would already allow us to switch to Guava 20.0. > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: OAK-7182-guava-21.diff, guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382121#comment-16382121 ] Marcel Reutegger commented on OAK-7282: --- LGTM > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7302) Document the overall design of the TarMK
Michael Dürig created OAK-7302: -- Summary: Document the overall design of the TarMK Key: OAK-7302 URL: https://issues.apache.org/jira/browse/OAK-7302 Project: Jackrabbit Oak Issue Type: Improvement Components: doc, segment-tar Reporter: Michael Dürig Many parts of the TarMK need better documentation from the design POV: * Format of records * Assumptions and requirements re. consistency of the underlying persistence * Interaction between various parts * Garbage collection, compaction and its modes * -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-5922) Utils.abortingIterable should implement Closeable
[
https://issues.apache.org/jira/browse/OAK-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger resolved OAK-5922.
---
Resolution: Fixed
Fix Version/s: 1.10
1.9.0
Done in trunk: http://svn.apache.org/r1825653
> Utils.abortingIterable should implement Closeable
> -
>
> Key: OAK-5922
> URL: https://issues.apache.org/jira/browse/OAK-5922
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.9.0, 1.10
>
>
> Wrapping a {{CloseableIterator}} with {{abortingIterable()}} causes it to
> lose it's closeability, which might lead to hard to debug problems later on.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7301) Javadoc: document hidden assumptions and invariants
Michael Dürig created OAK-7301: -- Summary: Javadoc: document hidden assumptions and invariants Key: OAK-7301 URL: https://issues.apache.org/jira/browse/OAK-7301 Project: Jackrabbit Oak Issue Type: Improvement Components: segment-tar Reporter: Michael Dürig We should improve API documentation wrt. hidden assumption and invariants: where the result of calling a method of a class depends on some state of the system this should be clearly stated and explained in the Javadoc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7300) Lucene Index: per-column selectivity to improve cost estimation
[ https://issues.apache.org/jira/browse/OAK-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382076#comment-16382076 ] Thomas Mueller commented on OAK-7300: - Linked related issue OAK-6735. /cc [~catholicon]. [~teofili] do you know if Apache Lucene has a built-in cardinality estimator? It doesn't look like, although I did find [this|https://lucidworks.com/2015/05/26/hyperloglog-field-value-cardinality-stats/] for Solr. > Lucene Index: per-column selectivity to improve cost estimation > --- > > Key: OAK-7300 > URL: https://issues.apache.org/jira/browse/OAK-7300 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, query >Reporter: Thomas Mueller >Assignee: Thomas Mueller >Priority: Major > Fix For: 1.10 > > > In OAK-6735 we have improved cost estimation for Lucene indexes, however the > following case is still not working as expected: a very common property is > indexes (many nodes have that property), and each value of that property is > more or less unique. In this case, currently the cost estimation is the total > number of documents that contain that property. Assuming the condition > "property is not null" this is correct, however for the common case "property > = x" the estimated cost is far too high. > A known workaround is to set the "costPerEntry" for the given index to a low > value, for example 0.2. However this isn't a good solution, as it affects all > properties and queries. > It would be good to be able to set the selectivity per property, for example > by specifying the number of distinct values, or (better yet) the average > number of entries for a given key (1 for unique values, 2 meaning for each > distinct values there are two documents on average). > That value can be set manually (cost override), and it can be set > automatically, e.g. when building the index, or updated from time to time > during the index update, using a cardinality > estimation algorithm. That doesn't have to be accurate; we could use an rough > approximation such as hyperbitbit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6031) Add TarFiles to the architecture diagram
[
https://issues.apache.org/jira/browse/OAK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382074#comment-16382074
]
Francesco Mari commented on OAK-6031:
-
License header backported to 1.8 at r1825652.
> Add TarFiles to the architecture diagram
>
>
> Key: OAK-6031
> URL: https://issues.apache.org/jira/browse/OAK-6031
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: doc, segment-tar
>Affects Versions: 1.8.0
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Major
> Labels: documentation
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> The newly created {{TarFiles}} should be added to the architecture diagram
> for oak-segment-tar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-6031) Add TarFiles to the architecture diagram
[
https://issues.apache.org/jira/browse/OAK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Mari resolved OAK-6031.
-
Resolution: Fixed
> Add TarFiles to the architecture diagram
>
>
> Key: OAK-6031
> URL: https://issues.apache.org/jira/browse/OAK-6031
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: doc, segment-tar
>Affects Versions: 1.8.0
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Major
> Labels: documentation
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> The newly created {{TarFiles}} should be added to the architecture diagram
> for oak-segment-tar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-6031) Add TarFiles to the architecture diagram
[
https://issues.apache.org/jira/browse/OAK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382068#comment-16382068
]
Francesco Mari commented on OAK-6031:
-
License header added on trunk at r1825651.
> Add TarFiles to the architecture diagram
>
>
> Key: OAK-6031
> URL: https://issues.apache.org/jira/browse/OAK-6031
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: doc, segment-tar
>Affects Versions: 1.8.0
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Major
> Labels: documentation
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> The newly created {{TarFiles}} should be added to the architecture diagram
> for oak-segment-tar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-7057) Segment.toString: Record table should include an index into the hexdump
[
https://issues.apache.org/jira/browse/OAK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-7057.
Resolution: Fixed
Fixed at [http://svn.apache.org/viewvc?rev=1825650=rev] by listing both,
record offset and address as suggested by [~frm]:
{noformat}
Segment a4f3443f-c83f-4084-a616-7feb5c62465d (304 bytes)
Info: {"wid":"sys.","sno":2,"t":1519914351666}, Generation:
GCGeneration{generation=0,fullGeneration=0,isCompacted=false}
--
VALUE record : 0003ffd0 @ 0100
VALUE record 0001: 0003ffcc @ 00fc
VALUE record 0002: 0003ffc8 @ 00f8
VALUE record 0003: 0003ffc4 @ 00f4
BUCKET record 0004: 0003ffb0 @ 00e0
TEMPLATE record 0005: 0003ffa0 @ 00d0
VALUE record 0006: 0003ff9c @ 00cc
VALUE record 0007: 0003ff88 @ 00b8
VALUE record 0008: 0003ff84 @ 00b4
BUCKET record 0009: 0003ff70 @ 00a0
NODE record 000a: 0003ff5c @ 008c
--
30 61 4B 0D 00 00 00 00 00 00 00 00 00 00 00 00 0aK.
0010 00 00 00 00 00 0B 00 00 00 00 00 00 00 00 00 00
0020 00 00 00 00 04 00 03 FF D0 00 00 00 01 04 00 03
0030 FF CC 00 00 00 02 04 00 03 FF C8 00 00 00 03 04
0040 00 03 FF C4 00 00 00 04 02 00 03 FF B0 00 00 00
0050 05 06 00 03 FF A0 00 00 00 06 04 00 03 FF 9C 00
0060 00 00 07 04 00 03 FF 88 00 00 00 08 04 00 03 FF
0070 84 00 00 00 09 02 00 03 FF 70 00 00 00 0A 07 00 .p..
0080 03 FF 5C 00 00 00 00 00 00 00 00 00 00 00 00 00 ..\.
0090 00 0A 00 00 00 00 00 05 00 00 00 00 00 09 00 00
00A0 00 00 00 00 00 06 00 00 00 00 00 07 00 00 00 00
00B0 00 08 00 00 03 61 62 63 11 33 2E 31 34 31 35 39 .abc.3.14159
00C0 32 36 35 33 35 38 39 37 39 33 00 00 03 31 32 33 2653589793...123
00D0 20 00 00 03 00 00 00 00 00 04 03 04 01 00 00 00 ...
00E0 00 00 00 00 00 01 00 00 00 00 00 02 00 00 00 00
00F0 00 03 00 00 03 66 6F 6F 03 62 61 7A 03 62 61 72 .foo.baz.bar
0100 2C 7B 22 77 69 64 22 3A 22 73 79 73 2E 30 30 30 ,{"wid":"sys.000
0110 30 22 2C 22 73 6E 6F 22 3A 32 2C 22 74 22 3A 31 0","sno":2,"t":1
0120 35 31 39 39 31 34 33 35 31 36 36 36 7D 00 00 00 519914351666}...
--
{noformat}
> Segment.toString: Record table should include an index into the hexdump
> ---
>
> Key: OAK-7057
> URL: https://issues.apache.org/jira/browse/OAK-7057
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Minor
> Labels: tooling
> Fix For: 1.9.0, 1.10
>
>
> Currently the Segment dump created in {{Segment.toString}} includes a list of
> records with their offsets. However these offsets do no match the ones in the
> subsequent raw byte dump of the segment. We should add a raw offsets to the
> list of records so finding the actual data that belongs to a record doesn't
> involve manually fiddling with logical / physical offset translation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger closed OAK-7296.
-
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7300) Lucene Index: per-column selectivity to improve cost estimation
Thomas Mueller created OAK-7300: --- Summary: Lucene Index: per-column selectivity to improve cost estimation Key: OAK-7300 URL: https://issues.apache.org/jira/browse/OAK-7300 Project: Jackrabbit Oak Issue Type: Improvement Components: lucene, query Reporter: Thomas Mueller Assignee: Thomas Mueller Fix For: 1.10 In OAK-6735 we have improved cost estimation for Lucene indexes, however the following case is still not working as expected: a very common property is indexes (many nodes have that property), and each value of that property is more or less unique. In this case, currently the cost estimation is the total number of documents that contain that property. Assuming the condition "property is not null" this is correct, however for the common case "property = x" the estimated cost is far too high. A known workaround is to set the "costPerEntry" for the given index to a low value, for example 0.2. However this isn't a good solution, as it affects all properties and queries. It would be good to be able to set the selectivity per property, for example by specifying the number of distinct values, or (better yet) the average number of entries for a given key (1 for unique values, 2 meaning for each distinct values there are two documents on average). That value can be set manually (cost override), and it can be set automatically, e.g. when building the index, or updated from time to time during the index update, using a cardinality estimation algorithm. That doesn't have to be accurate; we could use an rough approximation such as hyperbitbit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger resolved OAK-7296.
---
Resolution: Fixed
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382058#comment-16382058
]
Marcel Reutegger commented on OAK-7296:
---
{noformat}
[WARNING] Files with unapproved licenses:
oak-doc/src/site/markdown/nodestore/segment/classes.svg
{noformat}
I reopened OAK-6031 and will resolve this issue because the recent build was
otherwise OK.
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (OAK-6031) Add TarFiles to the architecture diagram
[
https://issues.apache.org/jira/browse/OAK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger reopened OAK-6031:
---
Those changes removed the license header from classes.svg.
> Add TarFiles to the architecture diagram
>
>
> Key: OAK-6031
> URL: https://issues.apache.org/jira/browse/OAK-6031
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: doc, segment-tar
>Affects Versions: 1.8.0
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Major
> Labels: documentation
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> The newly created {{TarFiles}} should be added to the architecture diagram
> for oak-segment-tar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382047#comment-16382047
]
Hudson commented on OAK-7296:
-
Build is still failing.
Failed run: [Jackrabbit Oak
#1281|https://builds.apache.org/job/Jackrabbit%20Oak/1281/] [console
log|https://builds.apache.org/job/Jackrabbit%20Oak/1281/console]
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7057) Segment.toString: Record table should include an index into the hexdump
[
https://issues.apache.org/jira/browse/OAK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382046#comment-16382046
]
Francesco Mari commented on OAK-7057:
-
Why not printing both the original and the normalized offset in the dump? This
should given enough debugging information (the original offset) and a more
understandable pointer to the record (the normalized offset).
> Segment.toString: Record table should include an index into the hexdump
> ---
>
> Key: OAK-7057
> URL: https://issues.apache.org/jira/browse/OAK-7057
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Minor
> Labels: tooling
> Fix For: 1.9.0, 1.10
>
>
> Currently the Segment dump created in {{Segment.toString}} includes a list of
> records with their offsets. However these offsets do no match the ones in the
> subsequent raw byte dump of the segment. We should add a raw offsets to the
> list of records so finding the actual data that belongs to a record doesn't
> involve manually fiddling with logical / physical offset translation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7299) RDB*Store: update postgresql JDBC driver reference to 42.2.1
Julian Reschke created OAK-7299: --- Summary: RDB*Store: update postgresql JDBC driver reference to 42.2.1 Key: OAK-7299 URL: https://issues.apache.org/jira/browse/OAK-7299 Project: Jackrabbit Oak Issue Type: Technical task Components: rdbmk Reporter: Julian Reschke Assignee: Julian Reschke Fix For: 1.10 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382039#comment-16382039 ] Julian Reschke commented on OAK-7282: - Ok, adjusted patch: https://issues.apache.org/jira/secure/attachment/12912612/OAK-7282.diff > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-7282: Attachment: (was: OAK-7282.diff) > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-7282: Attachment: OAK-7282.diff > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-6891.
Resolution: Fixed
Switched flush thread, filer reaper and disk space check to
Scheduler.scheduleWithFixedDelay at
http://svn.apache.org/viewvc?rev=1825647=rev
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Michael Dürig
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7282) RDB: enable default continuous revision GC
[ https://issues.apache.org/jira/browse/OAK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382025#comment-16382025 ] Marcel Reutegger commented on OAK-7282: --- My preference is to simplify it, now that both MONGO an RDB have continuous RGC enabled by default. > RDB: enable default continuous revision GC > -- > > Key: OAK-7282 > URL: https://issues.apache.org/jira/browse/OAK-7282 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk, rdbmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Major > Fix For: 1.10 > > Attachments: OAK-7282.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-5922) Utils.abortingIterable should implement Closeable
[
https://issues.apache.org/jira/browse/OAK-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-5922:
Description: Wrapping a {{CloseableIterator}} with {{abortingIterable()}}
causes it to lose it's closeability, which might lead to hard to debug problems
later on. (was: Wrapping a {{CloseableIterator}} with {{abortingIterable()}}
causes it to lose it's closeability, which might lead tohard to debug problems
later on.)
> Utils.abortingIterable should implement Closeable
> -
>
> Key: OAK-5922
> URL: https://issues.apache.org/jira/browse/OAK-5922
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Marcel Reutegger
>Priority: Minor
>
> Wrapping a {{CloseableIterator}} with {{abortingIterable()}} causes it to
> lose it's closeability, which might lead to hard to debug problems later on.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381988#comment-16381988
]
Michael Dürig commented on OAK-6891:
As a first step I added {{Scheduler.scheduleWithFixedDelay()}} at
http://svn.apache.org/viewvc?rev=1825644=rev
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Michael Dürig
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7057) Segment.toString: Record table should include an index into the hexdump
[
https://issues.apache.org/jira/browse/OAK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381972#comment-16381972
]
Michael Dürig commented on OAK-7057:
Proposed patch:
{noformat}
===
--- oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
+++ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/Segment.java
@@ -562,7 +562,8 @@
writer.format("reference %02x: %s%n", i++, segmentId);
}
for (Entry entry : recordNumbers) {
-writer.format("%10s record %08x: %08x%n", entry.getType(),
entry.getRecordNumber(), entry.getOffset());
+int offset = data.size() - (MAX_SEGMENT_SIZE -
entry.getOffset());
+writer.format("%10s record %08x: %08x%n", entry.getType(),
entry.getRecordNumber(), offset);
}
}
writer.println("--");
{noformat}
This changes the segment dump from
{noformat}
Segment c8013169-2c7c-42cf-a9ff-eac7e5eb16d0 (304 bytes)
Info: {"wid":"sys.","sno":2,"t":1519909675884}, Generation:
GCGeneration{generation=0,fullGeneration=0,isCompacted=false}
--
VALUE record : 0003ffd0
VALUE record 0001: 0003ffcc
VALUE record 0002: 0003ffc8
VALUE record 0003: 0003ffc4
BUCKET record 0004: 0003ffb0
TEMPLATE record 0005: 0003ffa0
VALUE record 0006: 0003ff9c
VALUE record 0007: 0003ff88
VALUE record 0008: 0003ff84
BUCKET record 0009: 0003ff70
NODE record 000a: 0003ff5c
--
30 61 4B 0D 00 00 00 00 00 00 00 00 00 00 00 00 0aK.
0010 00 00 00 00 00 0B 00 00 00 00 00 00 00 00 00 00
0020 00 00 00 00 04 00 03 FF D0 00 00 00 01 04 00 03
0030 FF CC 00 00 00 02 04 00 03 FF C8 00 00 00 03 04
0040 00 03 FF C4 00 00 00 04 02 00 03 FF B0 00 00 00
0050 05 06 00 03 FF A0 00 00 00 06 04 00 03 FF 9C 00
0060 00 00 07 04 00 03 FF 88 00 00 00 08 04 00 03 FF
0070 84 00 00 00 09 02 00 03 FF 70 00 00 00 0A 07 00 .p..
0080 03 FF 5C 00 00 00 00 00 00 00 00 00 00 00 00 00 ..\.
0090 00 0A 00 00 00 00 00 05 00 00 00 00 00 09 00 00
00A0 00 00 00 00 00 06 00 00 00 00 00 07 00 00 00 00
00B0 00 08 00 00 03 61 62 63 11 33 2E 31 34 31 35 39 .abc.3.14159
00C0 32 36 35 33 35 38 39 37 39 33 00 00 03 31 32 33 2653589793...123
00D0 20 00 00 03 00 00 00 00 00 04 03 04 01 00 00 00 ...
00E0 00 00 00 00 00 01 00 00 00 00 00 02 00 00 00 00
00F0 00 03 00 00 03 66 6F 6F 03 62 61 7A 03 62 61 72 .foo.baz.bar
0100 2C 7B 22 77 69 64 22 3A 22 73 79 73 2E 30 30 30 ,{"wid":"sys.000
0110 30 22 2C 22 73 6E 6F 22 3A 32 2C 22 74 22 3A 31 0","sno":2,"t":1
0120 35 31 39 39 30 39 36 37 35 38 38 34 7D 00 00 00 519909675884}...
--
{noformat}
to
{noformat}
Segment 04900e70-9f6e-46d2-a3cf-e1877760d2d3 (304 bytes)
Info: {"wid":"sys.","sno":2,"t":1519909591781}, Generation:
GCGeneration{generation=0,fullGeneration=0,isCompacted=false}
--
VALUE record : 0100
VALUE record 0001: 00fc
VALUE record 0002: 00f8
VALUE record 0003: 00f4
BUCKET record 0004: 00e0
TEMPLATE record 0005: 00d0
VALUE record 0006: 00cc
VALUE record 0007: 00b8
VALUE record 0008: 00b4
BUCKET record 0009: 00a0
NODE record 000a: 008c
--
30 61 4B 0D 00 00 00 00 00 00 00 00 00 00 00 00 0aK.
0010 00 00 00 00 00 0B 00 00 00 00 00 00 00 00 00 00
0020 00 00 00 00 04 00 03 FF D0 00 00 00 01 04 00 03
0030 FF CC 00 00 00 02 04 00 03 FF C8 00 00 00 03 04
0040 00 03 FF C4 00 00 00 04 02 00 03 FF B0 00 00 00
0050 05 06 00 03 FF A0 00 00 00 06 04 00 03 FF 9C 00
0060 00 00 07 04 00 03 FF 88 00 00 00 08 04 00 03 FF
0070 84 00 00 00 09 02 00 03 FF 70 00 00 00 0A 07 00 .p..
0080 03 FF 5C 00 00 00 00 00 00 00 00 00 00 00 00 00 ..\.
0090 00 0A 00 00 00 00 00 05 00 00 00 00 00 09 00 00
00A0 00 00 00 00 00 06 00 00 00 00 00 07 00 00 00 00
[jira] [Assigned] (OAK-5922) Utils.abortingIterable should implement Closeable
[
https://issues.apache.org/jira/browse/OAK-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger reassigned OAK-5922:
-
Assignee: Marcel Reutegger
> Utils.abortingIterable should implement Closeable
> -
>
> Key: OAK-5922
> URL: https://issues.apache.org/jira/browse/OAK-5922
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Marcel Reutegger
>Priority: Minor
>
> Wrapping a {{CloseableIterator}} with {{abortingIterable()}} causes it to
> lose it's closeability, which might lead tohard to debug problems later on.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381934#comment-16381934
]
Hudson commented on OAK-7296:
-
Build is still failing.
Failed run: [Jackrabbit Oak
#1280|https://builds.apache.org/job/Jackrabbit%20Oak/1280/] [console
log|https://builds.apache.org/job/Jackrabbit%20Oak/1280/console]
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381920#comment-16381920
]
Marcel Reutegger commented on OAK-7296:
---
Build timed out again after 90 minutes. I Changed the config to 100 minutes...
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381920#comment-16381920
]
Marcel Reutegger edited comment on OAK-7296 at 3/1/18 12:30 PM:
Build timed out again after 90 minutes. I changed the config to 100 minutes...
was (Author: mreutegg):
Build timed out again after 90 minutes. I Changed the config to 100 minutes...
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381843#comment-16381843
]
Hudson commented on OAK-7296:
-
Build is still failing.
Failed run: [Jackrabbit Oak
#1279|https://builds.apache.org/job/Jackrabbit%20Oak/1279/] [console
log|https://builds.apache.org/job/Jackrabbit%20Oak/1279/console]
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7286) DocumentNodeStoreBranch handling of non-recoverable DocumentStoreExceptions
[
https://issues.apache.org/jira/browse/OAK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381783#comment-16381783
]
Marcel Reutegger commented on OAK-7286:
---
Thanks. I took it a bit further and changed the ConflictException to not extend
from DocumentStoreException anymore. This required some more changes. See
https://github.com/apache/jackrabbit-oak/commit/5211986a0e931014d4aa4e7e53fc3b040b94b58c
> DocumentNodeStoreBranch handling of non-recoverable DocumentStoreExceptions
> ---
>
> Key: OAK-7286
> URL: https://issues.apache.org/jira/browse/OAK-7286
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Marcel Reutegger
>Priority: Major
> Attachments: OAK-7286.diff, OAK-7286.diff
>
>
> In {{DocumentNodeStoreBranch.merge()}}, any {{DocumentStoreException}} is
> mapped to a {{DocumentStoreException}} to a {{CommitFailedException}} of type
> "MERGE", which leads to the operation being retried, and a non-helpful
> exception being generated.
> The effect can be observed by enabling a test in {{ValidNamesTest}}:
> {noformat}
> --- oak-jcr/src/test/java/org/apache/jackrabbit/oak/jcr/ValidNamesTest.java
> (Revision 1825371)
> +++ oak-jcr/src/test/java/org/apache/jackrabbit/oak/jcr/ValidNamesTest.java
> (Arbeitskopie)
> @@ -300,7 +300,6 @@
> public void testUnpairedHighSurrogateEnd() {
> // see OAK-5506
>
> org.junit.Assume.assumeFalse(super.fixture.toString().toLowerCase().contains("segment"));
> -
> org.junit.Assume.assumeFalse(super.fixture.toString().toLowerCase().contains("rdb"));
> nameTest("foo" + SURROGATE_PAIR[0]);
> }
> @@ -336,6 +335,7 @@
> assertEquals("paths should be equal", p.getPath(), n.getPath());
> return p;
> } catch (RepositoryException ex) {
> +ex.printStackTrace();
> fail(ex.getMessage());
> return null;
> }
> {noformat}
> The underlying issue is that {{RDBDocumentStore}} is throwing a
> {{DocumentStoreException}} due to the invalid ID, and repeating the call will
> not help.
> We probably should have a way to dinstinguish between different types of
> problems.
> I hacked {{DocumentNodeStoreBranch}} like that:
> {noformat}
> ---
> oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreBranch.java
> (Revision 1825371)
> +++
> oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreBranch.java
> (Arbeitskopie)
> @@ -520,8 +520,12 @@
> } catch (ConflictException e) {
> throw e.asCommitFailedException();
> } catch(DocumentStoreException e) {
> -throw new CommitFailedException(MERGE, 1,
> -"Failed to merge changes to the underlying
> store", e);
> +if (e.getMessage().contains("Invalid ID")) {
> +throw new CommitFailedException(OAK, 123,
> +"Failed to store changes in the underlying
> store: " + e.getMessage(), e);
> +} else {
> +throw new CommitFailedException(MERGE, 1, "Failed to
> merge changes to the underlying store", e);
> +}
> } catch (Exception e) {
> throw new CommitFailedException(OAK, 1,
> "Failed to merge changes to the underlying
> store", e);
> {noformat}
> ...which causes the exception to surface immediately (see
> https://issues.apache.org/jira/secure/attachment/12912117/OAK-7286.diff).
> (cc [~mreutegg])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381778#comment-16381778
]
Michael Dürig commented on OAK-6891:
Will take care of this then, thanks!
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Michael Dürig
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig reassigned OAK-6891:
--
Assignee: Michael Dürig
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Michael Dürig
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381776#comment-16381776
]
Francesco Mari commented on OAK-6891:
-
[~mduerig], {{scheduleWithFixedDelay}} seems a better way to implement the
maintenance tasks.
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-7298) Remove debug logging to the console during tests
[ https://issues.apache.org/jira/browse/OAK-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-7298. Resolution: Fixed Fixed at http://svn.apache.org/viewvc?rev=1825636=rev > Remove debug logging to the console during tests > > > Key: OAK-7298 > URL: https://issues.apache.org/jira/browse/OAK-7298 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Michael Dürig >Assignee: Michael Dürig >Priority: Major > Labels: technical_debt > Fix For: 1.9.0, 1.10 > > > OAK-4707 introduced logging at debug logging to the system console for > sorting out test failures on Jenkins. Since we haveen't seen these failures > for a while and that issue is fixed I would like to remove the extra logging > again to avoid cluttering the console unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7298) Remove debug logging to the console during tests
Michael Dürig created OAK-7298: -- Summary: Remove debug logging to the console during tests Key: OAK-7298 URL: https://issues.apache.org/jira/browse/OAK-7298 Project: Jackrabbit Oak Issue Type: Improvement Components: segment-tar Reporter: Michael Dürig Assignee: Michael Dürig Fix For: 1.9.0, 1.10 OAK-4707 introduced logging at debug logging to the system console for sorting out test failures on Jenkins. Since we haveen't seen these failures for a while and that issue is fixed I would like to remove the extra logging again to avoid cluttering the console unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger updated OAK-7296:
--
Summary: Build failure: Rule violated for oak-core (was: Build Jackrabbit
Oak #1273 failed)
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OAK-7296) Build failure: Rule violated for oak-core
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger reassigned OAK-7296:
-
Assignee: Marcel Reutegger
> Build failure: Rule violated for oak-core
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Assignee: Marcel Reutegger
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build Jackrabbit Oak #1273 failed
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381756#comment-16381756
]
Marcel Reutegger commented on OAK-7296:
---
I updated the Jenkins build again to use the {{pedantic}} profile instead of
{{rat}}.
> Build Jackrabbit Oak #1273 failed
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381753#comment-16381753
]
Michael Dürig commented on OAK-6891:
Maybe the simplest approach to fixing this is to switch from
{{ScheduledExecutorService.scheduleAtFixedRate()}} to using
{{ScheduledExecutorService#scheduleWithFixedDelay()}} instead:
{noformat}
* Creates and executes a periodic action that becomes enabled first
* after the given initial delay, and subsequently with the
* given delay between the termination of one execution and the
* commencement of the next. If any execution of the task
* encounters an exception, subsequent executions are suppressed.
* Otherwise, the task will only terminate via cancellation or
* termination of the executor.
{noformat}
[~frm], WDYT?
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OAK-7216) Remove support for binaries and documents in persistent cache
[ https://issues.apache.org/jira/browse/OAK-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380674#comment-16380674 ] Marcel Reutegger edited comment on OAK-7216 at 3/1/18 9:40 AM: --- Done in trunk: http://svn.apache.org/r1825587 and http://svn.apache.org/r1825634 was (Author: mreutegg): Done in trunk: http://svn.apache.org/r1825587 > Remove support for binaries and documents in persistent cache > - > > Key: OAK-7216 > URL: https://issues.apache.org/jira/browse/OAK-7216 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: technical_debt > Fix For: 1.9.0, 1.10 > > Attachments: OAK-7216.patch > > > The persistent cache currently stores binaries up to one MB by default. > However most of the BlobStore implementations already provide some form of > caching. E.g. for S3 a cache on the local filesystem is maintained and when > using a Jackrabbit FileDataStore the persistent cache is actually unnecessary. > Support for documents in the persistent cache should also be removed. In > contrast to other cache entries, documents are mutable and may cause > consistency issues when enabled with the persistent cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OAK-5272) Expose BlobStore API to provide information whether blob id is content hashed
[ https://issues.apache.org/jira/browse/OAK-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller reassigned OAK-5272: --- Assignee: Thomas Mueller > Expose BlobStore API to provide information whether blob id is content hashed > - > > Key: OAK-5272 > URL: https://issues.apache.org/jira/browse/OAK-5272 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Reporter: Amit Jain >Assignee: Thomas Mueller >Priority: Major > Fix For: 1.10 > > > As per discussion in OAK-5253 it's better to have some information from the > BlobStore(s) whether the blob id can be solely relied upon for comparison. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5272) Expose BlobStore API to provide information whether blob id is content hashed
[
https://issues.apache.org/jira/browse/OAK-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381740#comment-16381740
]
Thomas Mueller commented on OAK-5272:
-
To be able to more easily migrate to other hashing algorithms, and also be able
to use identifiers that are not content hash, I think it makes sense to further
extend the API (maybe just the internal API), for example as follows:
{noformat}
/**
* For this binary, returns the map of all known content hashes,
* and CRC codes, together with the hash algorithm used.
* This can save an application from having to call getStream()
* and calculate the CRC / content hash itself.
* The returned map can be empty if the implementation would have to calculate
the values.
* If not empty, then the map contains one entry for each CRC / content hash
already calculated.
* The value is always hex-encoded (lowercase) without spaces.
* For example, it could return a "CRC32" (if known), "SHA-1" (if known),
"SHA-256", and so on.
*/
Map getKnownContentHashes();
{noformat}
For example Amazon S3 seems to calculate the MD5 and provide that as the ETag.
While MD5 isn't secure, it can be used in the same way as the CRC32, to say
whether two binaries are different for sure, or possibly the same.
> Expose BlobStore API to provide information whether blob id is content hashed
> -
>
> Key: OAK-5272
> URL: https://issues.apache.org/jira/browse/OAK-5272
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: blob
>Reporter: Amit Jain
>Priority: Major
> Fix For: 1.10
>
>
> As per discussion in OAK-5253 it's better to have some information from the
> BlobStore(s) whether the blob id can be solely relied upon for comparison.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7259) Improve SegmentNodeStoreStats to include number of commits per thread and threads currently waiting on the semaphore
[
https://issues.apache.org/jira/browse/OAK-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381732#comment-16381732
]
Michael Dürig commented on OAK-7259:
[~dulceanu], I like the new patch! It addresses all my concerns. IMO it's fit
for trunk.
Re. {{OpenDataException}}: An empty table is nothing exceptional, that's why I
think it is wrong to throw an exception in this case. If there is no way to
return an empty table, can we return an explicitly empty table (i.e. just a
single row containing dashes or so)? Or would this just make matters worse?
Somewhat related, it might be better to return the raw data from
{{CommitsTracker}} and do the conversion to {{TabularData}} to the MBean
implementation. This would also simplify testability of {{CommitsTracker}}.
Also we should probably make {{SegmentNodeStoreStats.setCollectStackTraces}} a
property as this makes it easier to use in most JMX clients.
Finally in the case when collecting stack traces is turned off, AFIU the map of
queued writers will contain the thread's name both as key and value. Wouldn't
it be clearer to just put "NA" as values?
> Improve SegmentNodeStoreStats to include number of commits per thread and
> threads currently waiting on the semaphore
>
>
> Key: OAK-7259
> URL: https://issues.apache.org/jira/browse/OAK-7259
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
>Priority: Major
> Labels: tooling
> Fix For: 1.9.0, 1.10
>
>
> When investigating the performance of {{segment-tar}}, the source of the
> writes (commits) is a very useful indicator of the cause.
> To better understand which threads are currently writing in the repository
> and which are blocked on the semaphore, we need to improve
> {{SegmentNodeStoreStats}} to:
> * expose the number of commits executed per thread
> * expose threads currently waiting on the semaphore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-5272) Expose BlobStore API to provide information whether blob id is content hashed
[ https://issues.apache.org/jira/browse/OAK-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-5272: Fix Version/s: 1.10 > Expose BlobStore API to provide information whether blob id is content hashed > - > > Key: OAK-5272 > URL: https://issues.apache.org/jira/browse/OAK-5272 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Reporter: Amit Jain >Priority: Major > Fix For: 1.10 > > > As per discussion in OAK-5253 it's better to have some information from the > BlobStore(s) whether the blob id can be solely relied upon for comparison. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Attachment: (was: OAK-6922.patch)
> Azure support for the segment-tar
> -
>
> Key: OAK-6922
> URL: https://issues.apache.org/jira/browse/OAK-6922
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
>Priority: Major
> Fix For: 1.9.0, 1.10
>
> Attachments: OAK-6922.patch
>
>
> An Azure Blob Storage implementation of the segment storage, based on the
> OAK-6921 work.
> h3. Segment files layout
> Thew new implementation doesn't use tar files. They are replaced with
> directories, storing segments, named after their UUIDs. This approach has
> following advantages:
> * no need to call seek(), which may be expensive on a remote file system.
> Rather than that we can read the whole file (=segment) at once.
> * it's possible to send multiple segments at once, asynchronously, which
> reduces the performance overhead (see below).
> The file structure is as follows:
> {noformat}
> [~]$ az storage blob list -c oak --output table
> Name Blob Type
> Blob TierLengthContent Type Last Modified
> ---
> --- -
> oak/data0a.tar/.ca1326d1-edf4-4d53-aef0-0f14a6d05b63 BlockBlob
> 192 application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0001.c6e03426-db9d-4315-a20a-12559e6aee54 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0002.b3784e27-6d16-4f80-afc1-6f3703f6bdb9 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0003.5d2f9588-0c92-4547-abf7-0263ee7c37bb BlockBlob
> 259216application/octet-stream 2018-01-31T10:59:14+00:00
> ...
> oak/data0a.tar/006e.7b8cf63d-849a-4120-aa7c-47c3dde25e48 BlockBlob
> 4368 application/octet-stream 2018-01-31T12:01:09+00:00
> oak/data0a.tar/006f.93799ae9-288e-4b32-afc2-bbc676fad7e5 BlockBlob
> 3792 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0070.8b2d5ff2-6a74-4ac3-a3cc-cc439367c2aa BlockBlob
> 3680 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0071.2a1c49f0-ce33-4777-a042-8aa8a704d202 BlockBlob
> 7760 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/journal.log.001 AppendBlob
> 1010 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/manifest BlockBlob
> 46application/octet-stream 2018-01-31T10:59:14+00:00
> oak/repo.lock BlockBlob
> application/octet-stream 2018-01-31T10:59:14+00:00
> {noformat}
> For the segment files, each name is prefixed with the index number. This
> allows to maintain an order, as in the tar archive. This order is normally
> stored in the index files as well, but if it's missing, the recovery process
> needs it.
> Each file contains the raw segment data, with no padding/headers. Apart from
> the segment files, there are 3 special files: binary references (.brf),
> segment graph (.gph) and segment index (.idx).
> h3. Asynchronous writes
> Normally, all the TarWriter writes are synchronous, appending the segments to
> the tar file. In case of Azure Blob Storage each write involves a network
> latency. That's why the SegmentWriteQueue was introduced. The segments are
> added to the blocking dequeue, which is served by a number of the consumer
> threads, writing the segments to the cloud. There's also a map UUID->Segment,
> which allows to return the segments in case they are requested by the
> readSegment() method before they are actually persisted. Segments are removed
> from the map only after a successful write operation.
> The flush() method blocks accepting the new segments and returns after all
> waiting segments are written. The close() method waits until the current
> operations are finished and stops all threads.
> The asynchronous mode can be disabled by setting the number of threads to 0.
> h5. Queue recovery mode
> If the Azure Blob Storage write() operation fails, the segment will be
> re-added and the queue is switched to an "recovery mode". In this mode, all
> the threads are suspended and new segments are not accepted (active
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Attachment: OAK-6922.patch
> Azure support for the segment-tar
> -
>
> Key: OAK-6922
> URL: https://issues.apache.org/jira/browse/OAK-6922
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
>Priority: Major
> Fix For: 1.9.0, 1.10
>
> Attachments: OAK-6922.patch
>
>
> An Azure Blob Storage implementation of the segment storage, based on the
> OAK-6921 work.
> h3. Segment files layout
> Thew new implementation doesn't use tar files. They are replaced with
> directories, storing segments, named after their UUIDs. This approach has
> following advantages:
> * no need to call seek(), which may be expensive on a remote file system.
> Rather than that we can read the whole file (=segment) at once.
> * it's possible to send multiple segments at once, asynchronously, which
> reduces the performance overhead (see below).
> The file structure is as follows:
> {noformat}
> [~]$ az storage blob list -c oak --output table
> Name Blob Type
> Blob TierLengthContent Type Last Modified
> ---
> --- -
> oak/data0a.tar/.ca1326d1-edf4-4d53-aef0-0f14a6d05b63 BlockBlob
> 192 application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0001.c6e03426-db9d-4315-a20a-12559e6aee54 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0002.b3784e27-6d16-4f80-afc1-6f3703f6bdb9 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0003.5d2f9588-0c92-4547-abf7-0263ee7c37bb BlockBlob
> 259216application/octet-stream 2018-01-31T10:59:14+00:00
> ...
> oak/data0a.tar/006e.7b8cf63d-849a-4120-aa7c-47c3dde25e48 BlockBlob
> 4368 application/octet-stream 2018-01-31T12:01:09+00:00
> oak/data0a.tar/006f.93799ae9-288e-4b32-afc2-bbc676fad7e5 BlockBlob
> 3792 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0070.8b2d5ff2-6a74-4ac3-a3cc-cc439367c2aa BlockBlob
> 3680 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0071.2a1c49f0-ce33-4777-a042-8aa8a704d202 BlockBlob
> 7760 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/journal.log.001 AppendBlob
> 1010 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/manifest BlockBlob
> 46application/octet-stream 2018-01-31T10:59:14+00:00
> oak/repo.lock BlockBlob
> application/octet-stream 2018-01-31T10:59:14+00:00
> {noformat}
> For the segment files, each name is prefixed with the index number. This
> allows to maintain an order, as in the tar archive. This order is normally
> stored in the index files as well, but if it's missing, the recovery process
> needs it.
> Each file contains the raw segment data, with no padding/headers. Apart from
> the segment files, there are 3 special files: binary references (.brf),
> segment graph (.gph) and segment index (.idx).
> h3. Asynchronous writes
> Normally, all the TarWriter writes are synchronous, appending the segments to
> the tar file. In case of Azure Blob Storage each write involves a network
> latency. That's why the SegmentWriteQueue was introduced. The segments are
> added to the blocking dequeue, which is served by a number of the consumer
> threads, writing the segments to the cloud. There's also a map UUID->Segment,
> which allows to return the segments in case they are requested by the
> readSegment() method before they are actually persisted. Segments are removed
> from the map only after a successful write operation.
> The flush() method blocks accepting the new segments and returns after all
> waiting segments are written. The close() method waits until the current
> operations are finished and stops all threads.
> The asynchronous mode can be disabled by setting the number of threads to 0.
> h5. Queue recovery mode
> If the Azure Blob Storage write() operation fails, the segment will be
> re-added and the queue is switched to an "recovery mode". In this mode, all
> the threads are suspended and new segments are not accepted (active waiting).
>
[jira] [Commented] (OAK-7297) New fixture for the Azure Segment Store
[
https://issues.apache.org/jira/browse/OAK-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381720#comment-16381720
]
Tomek Rękawek commented on OAK-7297:
Branch:
https://github.com/trekawek/jackrabbit-oak/tree/OAK-7297
> New fixture for the Azure Segment Store
> ---
>
> Key: OAK-7297
> URL: https://issues.apache.org/jira/browse/OAK-7297
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: benchmarks, it
>Reporter: Tomek Rękawek
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> We should add a new {{SEGMENT_AZURE}} fixture, so it's possible to run the
> integrationt tests for the [Azure Segment Store|OAK-6922]. By default it
> should connect to a local Azure Storage emulator:
> * https://docs.microsoft.com/en-us/azure/storage/common/storage-use-emulator
> * https://github.com/arafato/azurite
> If the {{oak.segment.azure.connection}} system property is defined, then it
> should try to connect to the specified [connection
> string|https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7297) New fixture for the Azure Segment Store
Tomek Rękawek created OAK-7297:
--
Summary: New fixture for the Azure Segment Store
Key: OAK-7297
URL: https://issues.apache.org/jira/browse/OAK-7297
Project: Jackrabbit Oak
Issue Type: Task
Components: benchmarks, it
Reporter: Tomek Rękawek
Fix For: 1.9.0, 1.10
We should add a new {{SEGMENT_AZURE}} fixture, so it's possible to run the
integrationt tests for the [Azure Segment Store|OAK-6922]. By default it should
connect to a local Azure Storage emulator:
* https://docs.microsoft.com/en-us/azure/storage/common/storage-use-emulator
* https://github.com/arafato/azurite
If the {{oak.segment.azure.connection}} system property is defined, then it
should try to connect to the specified [connection
string|https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build Jackrabbit Oak #1273 failed
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381705#comment-16381705
]
Hudson commented on OAK-7296:
-
Build is still failing.
Failed run: [Jackrabbit Oak
#1278|https://builds.apache.org/job/Jackrabbit%20Oak/1278/] [console
log|https://builds.apache.org/job/Jackrabbit%20Oak/1278/console]
> Build Jackrabbit Oak #1273 failed
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Description:
An Azure Blob Storage implementation of the segment storage, based on the
OAK-6921 work.
h3. Segment files layout
Thew new implementation doesn't use tar files. They are replaced with
directories, storing segments, named after their UUIDs. This approach has
following advantages:
* no need to call seek(), which may be expensive on a remote file system.
Rather than that we can read the whole file (=segment) at once.
* it's possible to send multiple segments at once, asynchronously, which
reduces the performance overhead (see below).
The file structure is as follows:
{noformat}
[~]$ az storage blob list -c oak --output table
Name Blob TypeBlob
TierLengthContent Type Last Modified
---
--- -
oak/data0a.tar/.ca1326d1-edf4-4d53-aef0-0f14a6d05b63 BlockBlob
192 application/octet-stream 2018-01-31T10:59:14+00:00
oak/data0a.tar/0001.c6e03426-db9d-4315-a20a-12559e6aee54 BlockBlob
262144application/octet-stream 2018-01-31T10:59:14+00:00
oak/data0a.tar/0002.b3784e27-6d16-4f80-afc1-6f3703f6bdb9 BlockBlob
262144application/octet-stream 2018-01-31T10:59:14+00:00
oak/data0a.tar/0003.5d2f9588-0c92-4547-abf7-0263ee7c37bb BlockBlob
259216application/octet-stream 2018-01-31T10:59:14+00:00
...
oak/data0a.tar/006e.7b8cf63d-849a-4120-aa7c-47c3dde25e48 BlockBlob
4368 application/octet-stream 2018-01-31T12:01:09+00:00
oak/data0a.tar/006f.93799ae9-288e-4b32-afc2-bbc676fad7e5 BlockBlob
3792 application/octet-stream 2018-01-31T12:01:14+00:00
oak/data0a.tar/0070.8b2d5ff2-6a74-4ac3-a3cc-cc439367c2aa BlockBlob
3680 application/octet-stream 2018-01-31T12:01:14+00:00
oak/data0a.tar/0071.2a1c49f0-ce33-4777-a042-8aa8a704d202 BlockBlob
7760 application/octet-stream 2018-01-31T12:10:54+00:00
oak/journal.log.001 AppendBlob
1010 application/octet-stream 2018-01-31T12:10:54+00:00
oak/manifest BlockBlob
46application/octet-stream 2018-01-31T10:59:14+00:00
oak/repo.lock BlockBlob
application/octet-stream 2018-01-31T10:59:14+00:00
{noformat}
For the segment files, each name is prefixed with the index number. This allows
to maintain an order, as in the tar archive. This order is normally stored in
the index files as well, but if it's missing, the recovery process needs it.
Each file contains the raw segment data, with no padding/headers. Apart from
the segment files, there are 3 special files: binary references (.brf), segment
graph (.gph) and segment index (.idx).
h3. Asynchronous writes
Normally, all the TarWriter writes are synchronous, appending the segments to
the tar file. In case of Azure Blob Storage each write involves a network
latency. That's why the SegmentWriteQueue was introduced. The segments are
added to the blocking dequeue, which is served by a number of the consumer
threads, writing the segments to the cloud. There's also a map UUID->Segment,
which allows to return the segments in case they are requested by the
readSegment() method before they are actually persisted. Segments are removed
from the map only after a successful write operation.
The flush() method blocks accepting the new segments and returns after all
waiting segments are written. The close() method waits until the current
operations are finished and stops all threads.
The asynchronous mode can be disabled by setting the number of threads to 0.
h5. Queue recovery mode
If the Azure Blob Storage write() operation fails, the segment will be re-added
and the queue is switched to an "recovery mode". In this mode, all the threads
are suspended and new segments are not accepted (active waiting). There's a
single thread which retries adding the segment with some delay. If the segment
is successfully written, the queue will back to the normal operation.
This way the unavailable remote service is not flooded by the requests and
we're not accepting the segments when we can't persist them.
The close() method finishes the recovery mode - in this case, some of the
awaiting segments won't be persisted.
h5. Consistency
The asynchronous mode isn't as reliable as the standard, synchronous case.
Following cases are possible:
* TarWriter#writeEntry() returns successfully, but the segments
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Attachment: OAK-6922.patch
> Azure support for the segment-tar
> -
>
> Key: OAK-6922
> URL: https://issues.apache.org/jira/browse/OAK-6922
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
>Priority: Major
> Fix For: 1.9.0, 1.10
>
> Attachments: OAK-6922.patch
>
>
> An Azure Blob Storage implementation of the segment storage, based on the
> OAK-6921 work.
> h3. Segment files layout
> Thew new implementation doesn't use tar files. They are replaced with
> directories, storing segments, named after their UUIDs. This approach has
> following advantages:
> * no need to call seek(), which may be expensive on a remote file system.
> Rather than that we can read the whole file (=segment) at once.
> * it's possible to send multiple segments at once, asynchronously, which
> reduces the performance overhead (see below).
> The file structure is as follows:
> {noformat}
> [~]$ az storage blob list -c oak --output table
> Name Blob Type
> Blob TierLengthContent Type Last Modified
> ---
> --- -
> oak/data0a.tar/.ca1326d1-edf4-4d53-aef0-0f14a6d05b63 BlockBlob
> 192 application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0001.c6e03426-db9d-4315-a20a-12559e6aee54 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0002.b3784e27-6d16-4f80-afc1-6f3703f6bdb9 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0003.5d2f9588-0c92-4547-abf7-0263ee7c37bb BlockBlob
> 259216application/octet-stream 2018-01-31T10:59:14+00:00
> ...
> oak/data0a.tar/006e.7b8cf63d-849a-4120-aa7c-47c3dde25e48 BlockBlob
> 4368 application/octet-stream 2018-01-31T12:01:09+00:00
> oak/data0a.tar/006f.93799ae9-288e-4b32-afc2-bbc676fad7e5 BlockBlob
> 3792 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0070.8b2d5ff2-6a74-4ac3-a3cc-cc439367c2aa BlockBlob
> 3680 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0071.2a1c49f0-ce33-4777-a042-8aa8a704d202 BlockBlob
> 7760 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/journal.log.001 AppendBlob
> 1010 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/manifest BlockBlob
> 46application/octet-stream 2018-01-31T10:59:14+00:00
> oak/repo.lock BlockBlob
> application/octet-stream 2018-01-31T10:59:14+00:00
> {noformat}
> For the segment files, each name is prefixed with the index number. This
> allows to maintain an order, as in the tar archive. This order is normally
> stored in the index files as well, but if it's missing, the recovery process
> needs it.
> Each file contains the raw segment data, with no padding/headers. Apart from
> the segment files, there are 3 special files: binary references (.brf),
> segment graph (.gph) and segment index (.idx).
> h3. Asynchronous writes
> Normally, all the TarWriter writes are synchronous, appending the segments to
> the tar file. In case of Azure Blob Storage each write involves a network
> latency. That's why the SegmentWriteQueue was introduced. The segments are
> added to the blocking dequeue, which is served by a number of the consumer
> threads, writing the segments to the cloud. There's also a map UUID->Segment,
> which allows to return the segments in case they are requested by the
> readSegment() method before they are actually persisted. Segments are removed
> from the map only after a successful write operation.
> The flush() method blocks accepting the new segments and returns after all
> waiting segments are written. The close() method waits until the current
> operations are finished and stops all threads.
> The asynchronous mode can be disabled by setting the number of threads to 0.
> h5. Queue recovery mode
> If the Azure Blob Storage write() operation fails, the segment will be
> re-added and the queue is switched to an "recovery mode". In this mode, all
> the threads are suspended and new segments are not accepted (active waiting).
>
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[
https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6922:
---
Attachment: (was: OAK-6922.patch)
> Azure support for the segment-tar
> -
>
> Key: OAK-6922
> URL: https://issues.apache.org/jira/browse/OAK-6922
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> An Azure Blob Storage implementation of the segment storage, based on the
> OAK-6921 work.
> h3. Segment files layout
> Thew new implementation doesn't use tar files. They are replaced with
> directories, storing segments, named after their UUIDs. This approach has
> following advantages:
> * no need to call seek(), which may be expensive on a remote file system.
> Rather than that we can read the whole file (=segment) at once.
> * it's possible to send multiple segments at once, asynchronously, which
> reduces the performance overhead (see below).
> The file structure is as follows:
> {noformat}
> [~]$ az storage blob list -c oak --output table
> Name Blob Type
> Blob TierLengthContent Type Last Modified
> ---
> --- -
> oak/data0a.tar/.ca1326d1-edf4-4d53-aef0-0f14a6d05b63 BlockBlob
> 192 application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0001.c6e03426-db9d-4315-a20a-12559e6aee54 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0002.b3784e27-6d16-4f80-afc1-6f3703f6bdb9 BlockBlob
> 262144application/octet-stream 2018-01-31T10:59:14+00:00
> oak/data0a.tar/0003.5d2f9588-0c92-4547-abf7-0263ee7c37bb BlockBlob
> 259216application/octet-stream 2018-01-31T10:59:14+00:00
> ...
> oak/data0a.tar/006e.7b8cf63d-849a-4120-aa7c-47c3dde25e48 BlockBlob
> 4368 application/octet-stream 2018-01-31T12:01:09+00:00
> oak/data0a.tar/006f.93799ae9-288e-4b32-afc2-bbc676fad7e5 BlockBlob
> 3792 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0070.8b2d5ff2-6a74-4ac3-a3cc-cc439367c2aa BlockBlob
> 3680 application/octet-stream 2018-01-31T12:01:14+00:00
> oak/data0a.tar/0071.2a1c49f0-ce33-4777-a042-8aa8a704d202 BlockBlob
> 7760 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/journal.log.001 AppendBlob
> 1010 application/octet-stream 2018-01-31T12:10:54+00:00
> oak/manifest BlockBlob
> 46application/octet-stream 2018-01-31T10:59:14+00:00
> oak/repo.lock BlockBlob
> application/octet-stream 2018-01-31T10:59:14+00:00
> {noformat}
> For the segment files, each name is prefixed with the index number. This
> allows to maintain an order, as in the tar archive. This order is normally
> stored in the index files as well, but if it's missing, the recovery process
> needs it.
> Each file contains the raw segment data, with no padding/headers. Apart from
> the segment files, there are 3 special files: binary references (.brf),
> segment graph (.gph) and segment index (.idx).
> h3. Asynchronous writes
> Normally, all the TarWriter writes are synchronous, appending the segments to
> the tar file. In case of Azure Blob Storage each write involves a network
> latency. That's why the SegmentWriteQueue was introduced. The segments are
> added to the blocking dequeue, which is served by a number of the consumer
> threads, writing the segments to the cloud. There's also a map UUID->Segment,
> which allows to return the segments in case they are requested by the
> readSegment() method before they are actually persisted. Segments are removed
> from the map only after a successful write operation.
> The flush() method blocks accepting the new segments and returns after all
> waiting segments are written. The close() method waits until the current
> operations are finished and stops all threads.
> The asynchronous mode can be disabled by setting the number of threads to 0.
> h5. Queue recovery mode
> If the Azure Blob Storage write() operation fails, the segment will be
> re-added and the queue is switched to an "recovery mode". In this mode, all
> the threads are suspended and new segments are not accepted (active waiting).
> There's a single thread which
[jira] [Updated] (OAK-6891) Executions of background threads might pile up
[
https://issues.apache.org/jira/browse/OAK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-6891:
---
Fix Version/s: 1.9.0
> Executions of background threads might pile up
> --
>
> Key: OAK-6891
> URL: https://issues.apache.org/jira/browse/OAK-6891
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Francesco Mari
>Priority: Major
> Labels: production
> Fix For: 1.9.0, 1.10
>
> Attachments: example.txt
>
>
> The background threads used in {{FileStore}} are implemented by wrapping
> {{Runnable}} instances in {{SafeRunnable}}, and by handing the
> {{SafeRunnable}} instances over to a {{ScheduledExecutorService}}.
> The documentation of {{ScheduledExecutorService#scheduleAtFixedRate}} states
> that "if any execution of a task takes longer than its period, then
> subsequent executions may start late, but will not concurrently execute".
> This means that if an execution is delayed, the piled up executions might
> fire in rapid succession.
> This way of running the periodic background threads might not be ideal. For
> example, it doesn't make much sense to flush the File Store five times in a
> row. On the other hand, if the background tasks are coded with this caveat in
> mind, this issue might not be a problem at all. For example, flushing the
> File Store five times in a row might not be a problem if many of those
> executions don't do much and return quickly.
> Tasks piling up might be a problem when it comes to release the resource
> associated with the {{FileStore}} in a responsive way. Since the
> {{ScheduledExecutorService}} is gracefully shut down, it might take some time
> before all the scheduled background tasks are processed and the
> {{ScheduledExecutorService}} is ready to be terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-6770) Convert oak-segment-tar to OSGi R6 annotations
[ https://issues.apache.org/jira/browse/OAK-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-6770: --- Fix Version/s: (was: 1.9.0) > Convert oak-segment-tar to OSGi R6 annotations > -- > > Key: OAK-6770 > URL: https://issues.apache.org/jira/browse/OAK-6770 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Robert Munteanu >Assignee: Francesco Mari >Priority: Minor > Labels: osgi > Fix For: 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7296) Build Jackrabbit Oak #1273 failed
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381672#comment-16381672
]
Marcel Reutegger commented on OAK-7296:
---
This is caused by a change to the Jenkins build I did yesterday. I enabled the
{{rat}} profile in an attempt to fail the build when a license header is
missing. Turns out this has some undesirable side effect. The profile also
narrows the scope of tests to a single class, which obviously isn't enough for
the test coverage check.
> Build Jackrabbit Oak #1273 failed
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-5506) reject item names with unpaired surrogates early
[
https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-5506:
---
Labels: resilience (was: )
> reject item names with unpaired surrogates early
>
>
> Key: OAK-5506
> URL: https://issues.apache.org/jira/browse/OAK-5506
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: core, jcr, segment-tar
>Affects Versions: 1.5.18
>Reporter: Julian Reschke
>Priority: Minor
> Labels: resilience
> Fix For: 1.10
>
> Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff,
> OAK-5506-bench.diff, OAK-5506-jcr-level.diff, OAK-5506-name-conversion.diff,
> OAK-5506-segment.diff, OAK-5506-segment2.diff, OAK-5506-segment3.diff,
> OAK-5506.diff, ValidNamesTest.java
>
>
> Apparently, the following node name is accepted:
>{{"foo\ud800"}}
> but a subsequent {{getPath()}} call fails:
> {noformat}
> javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not
> exist anymore
> at
> org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86)
> at
> org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34)
> at
> org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615)
> at
> org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205)
> at
> org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112)
> at
> org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140)
> at
> org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106)
> at
> org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271)
> at
> org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat}
> (test case follows)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7296) Build Jackrabbit Oak #1273 failed
[
https://issues.apache.org/jira/browse/OAK-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger updated OAK-7296:
--
Description:
No description is provided
The build Jackrabbit Oak #1273 has failed.
First failed run: [Jackrabbit Oak
#1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
{noformat}
[WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
expected minimum is 0.77
[ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
(default-check) on project oak-core: Coverage checks have not been met. See log
for details. -> [Help 1]
{noformat}
was:
No description is provided
The build Jackrabbit Oak #1273 has failed.
First failed run: [Jackrabbit Oak
#1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> Build Jackrabbit Oak #1273 failed
> -
>
> Key: OAK-7296
> URL: https://issues.apache.org/jira/browse/OAK-7296
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #1273 has failed.
> First failed run: [Jackrabbit Oak
> #1273|https://builds.apache.org/job/Jackrabbit%20Oak/1273/] [console
> log|https://builds.apache.org/job/Jackrabbit%20Oak/1273/console]
> {noformat}
> [WARNING] Rule violated for bundle oak-core: lines covered ratio is 0.24, but
> expected minimum is 0.77
> [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.9:check
> (default-check) on project oak-core: Coverage checks have not been met. See
> log for details. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
