[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882202#comment-15882202 ] Chetan Mehrotra commented on OAK-5827: -- I think we can change hash algorithm and see affect on blobId etc. It can be safely done on existing setup also and would be used for only new binaries. > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882254#comment-15882254 ] Thomas Mueller commented on OAK-5827: - As the FileDataStore and S3DataStore are also used in Jackrabbit 2.x, (jackrabbit-data), I created and linked JCR-4115. > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[
https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15885607#comment-15885607
]
Davide Giannella commented on OAK-5827:
---
By looking at the patch it looks like it will require a full DS migration to
the new SHA algorithm which may be expensive on very large and remote
situation. And maybe even require a downtime.
Can we add some logic in the form of
- if saving always use SHA-256
- when looking up use SHA-256 and if not found fallback on SHA-1
I'm not familiar with the code but I think it will have to be dealt in the DS
implementation itself (JCR) and then in here we'll have to have two
{{getIdForInputStream()}} one leveraging the SHA-256 and the other the SHA-1.
If we can manage the logic above, any consumer of the DS can seamlessly apply
the new implementation without any complications.
> Don't use SHA-1 for new DataStore binaries
> --
>
> Key: OAK-5827
> URL: https://issues.apache.org/jira/browse/OAK-5827
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Thomas Mueller
> Attachments: OAK-5827.patch
>
>
> A [collision for
> SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html]
> has been published. We still use SHA-1 for the FileDataStore, and I believe
> the S3 DataStore right now. Given there is a collision, we should switch to a
> stronger algorithm, for example SHA-256, for new binaries.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887207#comment-15887207 ] Amit Jain commented on OAK-5827: [~tmueller] bq. some of the tests in oak-blob-cloud fail, because the old Jackrabbit version is used which still uses SHA-1 The S3 specific implementation depending on Jackrabbit is not enabled by default anymore. So, we can still apply this patch. But jackrabbit update is of course needed as FileDataStore is still extends from Jackrabbit > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller > Attachments: OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887211#comment-15887211 ] Amit Jain commented on OAK-5827: bq. By looking at the patch it looks like it will require a full DS migration to the new SHA algorithm No, older binaries need not be. See below bq. when looking up use SHA-256 and if not found fallback on SHA-1 The ids are looked up directly with the blob ids stored in the nodes so, awareness about how the id was constructed is required. > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller > Attachments: OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887217#comment-15887217 ] Amit Jain commented on OAK-5827: [~tmueller] I will try to add some test cases for GC other code paths using ids etc. where we can ensure that there is no id specific thing leaking into the code. Also, we might want some test cases where both type of ids are present and the test cases should be able to change the SHA algo back to SHA-1. > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller > Attachments: OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[
https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887838#comment-15887838
]
Amit Jain commented on OAK-5827:
[~tmueller]
Based on the latest patch of JCR-4115 there is no need to have this property in
Oak (except for tests) i.e we can remove it from the
{{AbstractSharedCachingDataStore}}.
But there's a test failure in {{IgnoreMissingBinariesTest#validateMigration}}
[1]. So, a part of migration might be affected or just changing the test might
do. cc/ [~tomek.rekawek]
[1]
{noformat}
validateMigration(org.apache.jackrabbit.oak.upgrade.IgnoreMissingBinariesTest)
Time elapsed: 1.166 sec <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.jackrabbit.oak.upgrade.IgnoreMissingBinariesTest.prepare(IgnoreMissingBinariesTest.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
validateMigration(org.apache.jackrabbit.oak.upgrade.IgnoreMissingBinariesTest)
Time elapsed: 1.166 sec <<< ERROR!
java.lang.NullPointerException
at
org.apache.jackrabbit.oak.upgrade.cli.container.SegmentNodeStoreContainer.close(SegmentNodeStoreContainer.java:75)
at
org.apache.jackrabbit.oak.commons.IOUtils.closeQuietly(IOUtils.java:301)
at
org.apache.jackrabbit.oak.upgrade.cli.AbstractOak2OakTest.clean(AbstractOak2OakTest.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUn
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961523#comment-15961523 ] Jasmeet Dhiman commented on OAK-5827: - I have hit the SHA-1 collision on one of my systems while installing a package. Caused by: org.apache.jackrabbit.core.data.DataStoreException: Could not add record at org.apache.jackrabbit.core.data.FileDataStore.addRecord(FileDataStore.java:235) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.writeStream(DataStoreBlobStore.java:540) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.writeBlob(DataStoreBlobStore.java:191) ... 83 more Caused by: java.io.IOException: SHA-1 collision: /opt/author/crx-quickstart/repository/repository/datastore/00/1b/74/001b74fc2168208e7af84fa74f802fdf75bff7d4 at org.apache.jackrabbit.core.data.FileDataStore.addRecord(FileDataStore.java:225) ... 85 more Error: org.apache.jackrabbit.vault.packaging.PackageException: javax.jcr.RepositoryException: Inaccessible value > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller >Assignee: Amit Jain > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, > candidate_oak_1_6 > Fix For: 1.7.0, 1.8 > > Attachments: OAK-5827b.patch, OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085539#comment-16085539 ] Julian Reschke commented on OAK-5827: - [~amitjain] - once we upgrade to Jackrabbit 2.14.2 (OAK-6442) we'll need this applied to 1.6 as well. Should I take over and do both? > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller >Assignee: Amit Jain > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, > candidate_oak_1_6 > Fix For: 1.7.0, 1.8 > > Attachments: OAK-5827b.patch, OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085545#comment-16085545 ] Amit Jain commented on OAK-5827: [~reschke] Sure! > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller >Assignee: Amit Jain > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, > candidate_oak_1_6 > Fix For: 1.7.0, 1.8 > > Attachments: OAK-5827b.patch, OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-5827) Don't use SHA-1 for new DataStore binaries
[ https://issues.apache.org/jira/browse/OAK-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087124#comment-16087124 ] Julian Reschke commented on OAK-5827: - trunk: [r1786122|http://svn.apache.org/r1786122] 1.6: [r1801921|http://svn.apache.org/r1801921] > Don't use SHA-1 for new DataStore binaries > -- > > Key: OAK-5827 > URL: https://issues.apache.org/jira/browse/OAK-5827 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Thomas Mueller >Assignee: Amit Jain > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.7.0, 1.8, 1.6.4 > > Attachments: OAK-5827b.patch, OAK-5827.patch > > > A [collision for > SHA-1|https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html] > has been published. We still use SHA-1 for the FileDataStore, and I believe > the S3 DataStore right now. Given there is a collision, we should switch to a > stronger algorithm, for example SHA-256, for new binaries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
