[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170858#comment-16170858 ] Jitendra Nath Pandey commented on HADOOP-14520: --- Fix version 2.0.6-alpha didn't look right, changed it to 2.9. [~steve_l] > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.4 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > hadoop-14520-branch-2-010.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161580#comment-16161580 ] Georgi Chalakov commented on HADOOP-14520: -- Thanks for the review Steve! > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.4 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Fix For: 2.0.6-alpha, 3.0.0-beta1 > > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > hadoop-14520-branch-2-010.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161218#comment-16161218 ] Steve Loughran commented on HADOOP-14520: - +1 for branch-2 patch committed to branch-2, except for the big logging @ debug in testing. That shouldn't be in either branch, not until needed. Maybe its time to review the azure/test/resources log4j file & comment out those logging levels that successful test runs don't need > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.4 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > hadoop-14520-branch-2-010.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159400#comment-16159400
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color}
| {color:red} hadoop-tools_hadoop-azure generated 3 new + 5 unchanged - 0 fixed
= 8 total (was 5) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
19s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886163/hadoop-14520-branch-2-010.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 302bf5eab3e0 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 1421196 |
| Default Java | 1.7.0_131 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13214/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13214/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13214/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.7.4
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
> Attachments: HADOOP-14520-006.patch,
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159350#comment-16159350 ] Georgi Chalakov commented on HADOOP-14520: -- Thanks for the review Steve! I have attached the patch for branch-2: hadoop-14520-branch-2-010.patch Results from endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net Tests run: 774, Failures: 0, Errors: 0, Skipped: 131 > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > hadoop-14520-branch-2-010.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157707#comment-16157707 ] Hudson commented on HADOOP-14520: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12811/]) HADOOP-14520. WASB: Block compaction for Azure Block Blobs. Contributed (stevel: rev 13eda5000304099d1145631f9be13ce8a00b600d) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemBlockCompaction.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureConcurrentOutOfBandIo.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/test/resources/log4j.properties * (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SyncableDataOutputStream.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobAppendStream.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > HADOOP-14520-patch-07-08.diff, HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157309#comment-16157309 ] Steve Loughran commented on HADOOP-14520: - LGTM +1 committed to trunk & branch-3. If you want to backport to branch-2 , apply the patch to branch-2, run the azure tests, and if all is well, attach the patch to this JIRA with a name like hadoop-14520-branch-2-010.patch & once submitted, the patch will be tested against branch 2 instead > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > HADOOP-14520-patch-07-08.diff, HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152546#comment-16152546 ] Steve Loughran commented on HADOOP-14520: - FWIW, moving to Call<> would be the best story, especially as it lines up for Java 8. But it would be a complicated change on what is a big enough patch as it is > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > HADOOP-14520-patch-07-08.diff, HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149720#comment-16149720
]
Hadoop QA commented on HADOOP-14520:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new
+ 84 unchanged - 2 fixed = 84 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
8s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884813/HADOOP_14520_10.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 215e83c15b7f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bac4e8c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13149/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13149/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.0.0-alpha3
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
> Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch,
> HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch,
> HADOOP_14520_08.patch, HADOOP_14520_09.patch,
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149691#comment-16149691 ] Georgi Chalakov commented on HADOOP-14520: -- Thank you for adding all these fixes. Stream capabilities looks like an useful feature. I will fix the space in last patch. Re:flush() FSDataOutputStream doesn't overwrite flush() and a normal flush() call on application level would not execute BlockBlobAppendStream::flush(). When the compaction is disabled hflush/hsync are nop and the performance of BlockBlobAppendStream is the same (or better) than before. Re:more than one append stream We take a lease on the blob, that means at any point of time you can have one append stream only. If we had more than one append stream, we cannot grantee the order of write operations. I have added hsync() call and made isclosed volatile. Re:close() I think the first exception is the best indication what went wrong. After an exception, close() is just best effort. I don't know how useful for a client would be to continue after IO related exception, but if that is necessary, the client can continue. If block compaction is enabled, the client can go and read all the data until last hflush()/hsync(). When the block compaction is disabled, we grantee nothing. We may or may not have the data stored in the service. > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149623#comment-16149623
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
11 new + 84 unchanged - 2 fixed = 95 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
5s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884788/HADOOP-14520-009.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 58361e6c6d2a 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d4417da |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13148/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13148/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13148/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13148/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type:
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149486#comment-16149486
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
22 new + 84 unchanged - 2 fixed = 106 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-azure in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
2s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884768/HADOOP-14520-008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c983acaae13e 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d4417da |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13147/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13147/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13147/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13147/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149154#comment-16149154 ] Steve Loughran commented on HADOOP-14520: - I'm reviewing this, it's just taking a while as I need to be thorough > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-05.patch, > HADOOP_14520_07.patch, HADOOP_14520_08.patch, HADOOP_14520_09.patch > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148113#comment-16148113
]
Hadoop QA commented on HADOOP-14520:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 84 unchanged - 2 fixed = 85 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884545/HADOOP_14520_09.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1001f0af9345 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4148023 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13140/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13140/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13140/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.0.0-alpha3
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
> Attachments:
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147880#comment-16147880
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 84 unchanged - 2 fixed = 85 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
12s{color} | {color:red} hadoop-azure in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
10s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884518/HADOOP_14520_08.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7c84b180781f 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fd66a24 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13138/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13138/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13138/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13138/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147827#comment-16147827 ] Georgi Chalakov commented on HADOOP-14520: -- HADOOP_14520_08.patch whitespace fixes; javadoc fixes. > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-05.patch, > HADOOP_14520_07.patch, HADOOP_14520_08.patch > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146404#comment-16146404
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
20 new + 84 unchanged - 2 fixed = 104 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
33s{color} | {color:red} hadoop-azure in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
8s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884368/HADOOP_14520_07.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e29c4a79b947 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 26fafc3 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13125/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13125/artifact/patchprocess/whitespace-eol.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13125/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13125/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13125/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
>
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146357#comment-16146357 ] Georgi Chalakov commented on HADOOP-14520: -- Results : Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 bq. if you are changing precondition check, I'd recommend StringUtils.isEmpty() for Preconditions.checkArgument(StringUtils.isNotEmpty(aKey)); Done. bq. If fields aren't updated after the constructor, best to set to final (example, compactionEnabled ?). Done. bq. How long is downloadBlockList going to take in that constructor? More specifically: if compaction is disabled, can that step be skipped? downloadBlockList is used for two purposes: 1) to check for block existence 2) to download the block list bq. If the stream needs a byte buffer, best to use ElasticByteBufferPool as a pool of buffers. Done. bq. Use StorageErrorCodeStrings as the source of string constants to check for in exception error codes. Done. bq. Rather than throw IOException(e), I'd prefer more specific (existing ones). That's PathIOException and subclasses, AzureException(e), and the java.io/java.nio ones. Done bq. When wrapping a StorageException with another IOE, always include the toString value of the wrapped exception. That way, the log message of the top level log retains the underlying problem. Done. bq. BlockBlobAppendStream.WriteRequest retry logic will retry even on RuntimeExceptions like IllegalArgumentException. Ideally they should be split into recoverable vs non-recoverable ops via a RetryPolicy. Is this an issue to address here though? Overall, with the new operatins doing retries, this may be time to embrace rety policies. Or at least create a JIRA entry on doing so. add*Command() will rethrow the last exception. That means the following write() or close() will retrow stored exception. It is not going to happen right away, but the will happen before the stream is closed() bq. I know java.io.OutputStream is marked as single-thread only, but I know of code (hello HBase!) which means that you must make some of the calls thread safe. HADOOP-11708/HADOOP-11710 covers this issue in CryptoOutputStream. At the very least, flush() must be synchronous with itself, close() & maybe write() flush() is synchronous with itself through addFlushCommand(). We do not want flush() to be synchronous with write(). We would like while a thread waits for a flush(), other threads to continue writing. bq. I'm unsure about BlockBlobAppendStream.close() waiting for up to 15 minutes for things to complete, but looking @ other blobstore clients, I can see that they are implicitly waiting without any timeout at all. And it's in the existing codebase. But: why was the time limit changed from 10 min to 15? Was this based on test failures? If so, where is the guarantee that a 15 minute wait is always sufficient. The change to 15 min was not based on test failures. I have changed the timeout back to 10 min and added a const. bq. Looking at BlockBlobAppendStream thread pooling, I think having a thread pool per output stream is expensive, especially as it has a minimum size of 4; it will ramp up fast. A pool of min=1 max=4 might be less expensive. But really, the stream should be thinking about sharing a pool common to the FS, relying on callbacks to notify it of completion rather than just awaiting pool completion and a shared writeable field. I did a some tests with YCSB and a pool of min=1, max=4. It is slower and the difference is measurable. Considering how many output stream you usually have per FS, I would like to keep min=4, max=4. The shared pool is a good idea, but I am afraid we would need bigger change and at the end I am not sure we will get significant benefits. bq. I think the access/use of lastException needs to be made stronger than just a volatile, as it means that code of the form if (lastException!=null) throw lastException isn't thread safe. I know, it's not that harmful provided lastException is never set to null, but I'd still like some isolated get/getAndSet/maybeThrow operations. Similarly, is lastException the best way to propagate failure, as it means that teardown failures are going to get reported ahead of earlier ones during the write itself. Overall, I propose using Callable WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-006.patch, HADOOP-14520-05.patch, > HADOOP_14520_07.patch > > > Block Compaction for WASB allows uploading new blocks for every
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134151#comment-16134151
]
Steve Loughran commented on HADOOP-14520:
-
I've been looking at the current page blob code, and I've seen something else I
think may need fixing; how things close
* {{PageBlobOutputStream.close()}} call is completely {{synchronized}}, even
during a long upload. Which means that until that upload completes, all calls
may block
* I don't see the flush or wrtie calls checking for the stream being open
before doing anything, meaning that the only way that the state will be checked
is internally, in {{iopool.execute()}} or elsewhere.
I think here it'd be good to move to an atomic boolean to manage closed state,
and use it as the guard to the operations.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.0.0-alpha3
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
> Attachments: HADOOP-14520-006.patch, HADOOP-14520-05.patch
>
>
> Block Compaction for WASB allows uploading new blocks for every hflush/hsync
> call. When the number of blocks is above 32000, next hflush/hsync triggers
> the block compaction process. Block compaction replaces a sequence of blocks
> with one block. From all the sequences with total length less than 4M,
> compaction chooses the longest one. It is a greedy algorithm that preserve
> all potential candidates for the next round. Block Compaction for WASB
> increases data durability and allows using block blobs instead of page blobs.
> By default, block compaction is disabled. Similar to the configuration for
> page blobs, the client needs to specify HDFS folders where block compaction
> over block blobs is enabled.
> Results for HADOOP-14520-05.patch
> tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net
> Tests run: 707, Failures: 0, Errors: 0, Skipped: 119
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133912#comment-16133912
]
Thomas Marquardt commented on HADOOP-14520:
---
I will hand this off to Georgi, as he is returning from vacation Monday. I
noticed the following while reviewing the latest patches:
1) {{writeBlockRequestInternal}} has retry logic that returns the buffer to the
pool and then retries using the buffer that it just returned.
2) {{writeBlockRequestInternal}} is currently returning a byte array originally
created by {{ByteArrayOutputStream}} to the buffer pool. If this is not clear,
look at blockCompaction where it creates {{ByteArrayOutputStreamInternal}},
then wraps the underlying {{byte[]}} in a {{ByteBuffer}} and passes it to
{{writeBlockRequestInternal}} which returns it to the pool.
3) {{blockCompaction}} can be refactored to make unit testing easy. For
example, extracting out a {{getBlockSequenceForCompaction}} function that takes
a block list as input and returns a sequence of blocks to be compacted would
allow a data driven unit test to run many different block lists thru the
algorithm.
4) I recommend the following description for the blockCompaction function:
{code:java}
/**
* Block compaction is only enabled when the number of blocks exceeds
activateCompactionBlockCount.
* The algorithm searches for the longest sequence of two or more blocks {b1,
b2, ..., bn} such that
* size(b1) + size(b2) + ... + size(bn) < maximum-block-size. It then
downloads the blocks in the
* sequence, concatenates the data to form a single block, uploads this new
block, and updates the block
* list to replace the sequence of blocks with the new block.
*/
{code}
5) I recommend renaming {{BlockBlobAppendStream.bufferSize}} to
{{maxBlockSize}}. It is the maximum size of a block.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.0.0-alpha3
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
> Attachments: HADOOP-14520-006.patch, HADOOP-14520-05.patch
>
>
> Block Compaction for WASB allows uploading new blocks for every hflush/hsync
> call. When the number of blocks is above 32000, next hflush/hsync triggers
> the block compaction process. Block compaction replaces a sequence of blocks
> with one block. From all the sequences with total length less than 4M,
> compaction chooses the longest one. It is a greedy algorithm that preserve
> all potential candidates for the next round. Block Compaction for WASB
> increases data durability and allows using block blobs instead of page blobs.
> By default, block compaction is disabled. Similar to the configuration for
> page blobs, the client needs to specify HDFS folders where block compaction
> over block blobs is enabled.
> Results for HADOOP-14520-05.patch
> tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net
> Tests run: 707, Failures: 0, Errors: 0, Skipped: 119
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130399#comment-16130399
]
Steve Loughran commented on HADOOP-14520:
-
Production code is looking pretty good, so I've just gone through the tests in
detail too now. Sorry.
1. size of buffers/compaction blocks
I'm worried about what happens when large buffers have been flushed & then a
compaction starts. The size of the buffer needed will be that of
sum(size(blocks)), won't it? I don't see any checks on those limits, such as a
decision to set a maximum size of a compacted block & break up compactions if
the total block count to compact is > that.
2. Failure handling on the compaction process. Does a failure on a compaction
download & upload in {{blockCompaction()} }need to fail the entire write
process? If it's a transient error it could be overkill. However, if it is a
sign that {{flush()}} isn't reliably working then the current behaviour is the
one to run with.
3. One thing I'd like (but which won't mandate) is for the stream to count the
#of compaction events, bytes compacted and total duration. then provide some
@VisibleForTesting @ Unstable getters, *and print them in the {{toString()}}
call. That would line things up for moving to FS-level instrumentation, and can
be used immediately .
h3. {{BlockBlobAppendStream}}:
* L349: use constant in {{StorageErrorCodeStrings}}
* Use {{org.apache.hadoop.util.DirectBufferPool}} to pool the buffers; stable
code, uses weak refs to ensure GCs will recover free buffers from the pool.
* Make sure that {{blockCompaction}} uses a buffer from the pool too; I don't
think it does right now.
* {{UploaderThreadFactory}}: idle thought: would it make sense to include the
container ID or container & key in the thread? I don't know of anything else
which does this, but it would aid thread dump diagnostics.
h3. {{SelfRenewingLease}}
L82: use the constants in {{StorageErrorCodeStrings}}
h2. Test code
* There's no concurrency test, which would be nice. Could one go into
{{TestNativeAzureFileSystemConcurrency}}
* Maybe also think about having {{TestBlockBlobInputStream}} use this stream as
its upload mechanism; insert some flushes through the loop and see what
actually happens on larger scale files. The small tests, while nice and fast,
don't check things like buffer sizing if you have large blocks to combine.
h3. {{TestNativeAzureFileSystemBlockCompaction}}
As background, I like to review tests from the following use case "its got a
transient jenkins failure and all you have is the stack trace to debug what
failed". Which means I expect tests to: preserve all stack traces, add as much
diagnostics information in asserts, including text for every simple
assertTrue/assertFalse —enough to get an idea what's wrong without pasting the
stack in the IDE to find out which specific assert actually failed.
h4. {{verifyFileData}} & {{verifyAppend}}:
I'm not actually sure these work properly if the created file is > the
generated test data, and, by swallowing exceptions, they don't actually report
underlying failures, merely trigger an assertion failure somewhere in the
calling code.
I'd replace these entirely with {{ContractTestUtils.verifyFileContents()}},
which does report failures and is widely enough used that it's considered
stable.
h4. {{testCompaction()}}
* once the verify calls rethrow all exceptions, some of the asserts here can be
cut
* there's a lot of copy-and-paste duplication fo the
write/write/write/flush/verify sequences; these should be factored out into
shared methods.
* if the stream.toString() call logs the compaction history, then includng the
stream toString in all asserts would help diagnose problems.
h4. other
* {{verifyBlockList}}: don't bother catching & asserting on exception, just
throw it all the way up & let JUnit report it.
* {{testCompactionDisabled}: use try-with-resource or
{{IOUtils.cleanupWithLogger}}.
h3. checkstyle
# Most of those "is a magic number" complaints are just about common values in
the test...if they were pulled out into some shared variables then it'd shut up
checkstyle
# there is that "15 minutes" constant in production. How about moving that up
from an inline constant to a static constant "CLOSE_UPLOAD_DELAY" or similar in
the class —so at least its obvious what the number is for/where the delay is
chosen. At some point in the future, if ever felt to be an issue, then it could
be made a config option, with all the trouble that ensues.
# javadoc is still unhappy.. I'm actually surprised that it's not complaining
about all the missing "."' chars at the end of each sentence ... maybe the
latest update to java 8.x has got javadocs complaining less. Lovely as that may
be, we have to worry about java9 too, so please: review the diff and add them
to the new javadoc comments.
# Probably a good time
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130125#comment-16130125
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
64 new + 82 unchanged - 4 fixed = 146 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} hadoop-azure in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
10s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882298/HADOOP-14520-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7a4f3a8eac5b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f04cb4 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13058/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13058/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13058/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13058/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129216#comment-16129216 ] Thomas Marquardt commented on HADOOP-14520: --- I started working on an update yesterday, since Georgi is on vacation. I'll provide an updated patch and review soon. > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-05.patch > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP-14520-05.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 707, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129186#comment-16129186
]
Steve Loughran commented on HADOOP-14520:
-
This is quite a big patch; I've had to learn my way round bits of code while
reviewing it. For that reason alone, I'm not knowledgeable enough to be the
sole reviewer. What I have done is gone through the code & tried to understand
what it's working with, comments below. The bad news is, because it's code I'm
not familiar with, (a) my comments go further than just this patch and (b) I
may be utterly wrong. Bear that in mind.
Here's my first review, though its not detailed enough.
h3. {{AzureNativeFileSystemStore}}
* It looks like {{AzureNativeFileSystemStore.getDirectorySet()}} doesn't trim
whitespace from paths. Created HADOOP-14778 to deal with it separately.
h3. {{BlockBlobAppendStream}}
* if you are changing precondition check, I'd recommend StringUtils.isEmpty()
for
{code}
Preconditions.checkArgument(StringUtils.isNotEmpty(aKey));
Preconditions.checkArgument(bufferSize >= 0);
{code}
* If fields aren't updated after the constructor, best to set to {{final}}
(example, {{compactionEnabled}} ?).
* How long is {{downloadBlockList}} going to take in that constructor? More
specifically: if compaction is disabled, can that step be skipped?
* If the stream needs a byte buffer, best to use {{ElasticByteBufferPool}} as a
pool of buffers.
h3. Exception handling, wrapping, rethrowing
* Use {{StorageErrorCodeStrings}} as the source of string constants to check
for in exception error codes.
* Rather than {{throw IOException(e)}}, I'd prefer more specific (existing
ones). That's {{PathIOException}} and subclasses, {{AzureException(e)}}, and
the java.io/java.nio ones. Whichever is closest to what's actually gone wrong.
IOEs are too generic to use in try/catch.
* When wrapping a StorageException with another IOE, always include the
toString value of the wrapped exception.
That way, the log message of the top level log retains the underlying problem.
Example from {{UploadBlockCommand}}:
{code}
throw new IOException("Encountered Exception while committing append blocks",
ex);
{code}
{code}
throw new IOException("Encountered Exception while committing append blocks: "
+ ex, ex);
{code}
* {{BlockBlobAppendStream.WriteRequest}} retry logic will retry even on
RuntimeExceptions like IllegalArgumentException. Ideally they should be split
into recoverable vs non-recoverable ops via a {{RetryPolicy}}. Is this an issue
to address here though?
Overall, with the new operatins doing retries, this may be time to embrace rety
policies. Or at least create a JIRA entry on doing so.
h3. Concurrency
# I know {{java.io.OutputStream}} is marked as single-thread only, but I know
of code (hello HBase!) which means that you must make some of the calls thread
safe. HADOOP-11708/HADOOP-11710 covers this issue in CryptoOutputStream. At the
very least, {{flush()}} must be synchronous with itself, close() & maybe write()
# I'm unsure about {{BlockBlobAppendStream.close()}} waiting for up to 15
minutes for things to complete, but looking @ other blobstore clients, I can
see that they are implicitly waiting without any timeout at all. And it's in
the existing codebase. But: why was the time limit changed from 10 min to 15?
Was this based on test failures? If so, where is the guarantee that a 15 minute
wait is always sufficient.
# Looking at {{BlockBlobAppendStream}} thread pooling, I think having a thread
pool per output stream is expensive, especially as it has a minimum size of 4;
it will ramp up fast. A pool of min=1 max=4 might be less expensive. But
really, the stream should be thinking about sharing a pool common to the FS,
relying on callbacks to notify it of completion rather than just awaiting pool
completion and a shared writeable field.
# I think the access/use of {{lastException}} needs to be made stronger than
just a {{volatile}}, as it means that code of the form {{if
(lastException!=null) throw lastException}} isn't thread safe. I know, it's not
that harmful provided lastException is never set to null, but I'd still like
some isolated get/getAndSet/maybeThrow operations.
# Similarly, is {{lastException}} the best way to propagate failure, as it
means that teardown failures are going to get reported ahead of earlier ones
during the write itself.
Overall, I propose using Callable<> over Runnable. Allows you to throw
exceptions & return things, caller gets to pick them up when it chooses to.
h3. code style
Checkstyle has a lot of complaints (which will need a resubmit to show).
* Can you do a patch without all the whitespace stripping? It makes the patch
too big & very brittle to cherrypick. I know the spaces are wrong, but trying
to strip them in a patch creates needless patch conflict. when the patch goes
in we'll
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101107#comment-16101107 ] Shane Mainali commented on HADOOP-14520: I have reviewed the latest as well and it looks good to me (my comments were already taken care of in previous patches), thanks [~Georgi]! Note that [~Georgi] also did HBase and other testing for this to validate the changes. > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-05.patch > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP-14520-05.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 707, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079722#comment-16079722
]
Hadoop QA commented on HADOOP-14520:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
82 new + 116 unchanged - 4 fixed = 198 total (was 120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12876335/HADOOP-14520-05.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d660dd8f2ed9 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ba5b056 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12746/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12746/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12746/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: Block compaction for Azure Block Blobs
>
>
> Key: HADOOP-14520
> URL: https://issues.apache.org/jira/browse/HADOOP-14520
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.0.0-alpha3
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
> Attachments:
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[
https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079508#comment-16079508
]
Hadoop QA commented on HADOOP-14520:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
80 new + 112 unchanged - 4 fixed = 192 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} hadoop-tools/hadoop-azure generated 4 new + 0
unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-azure in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Inconsistent synchronization of
org.apache.hadoop.fs.azure.BlockBlobAppendStream.activateCompactionBlockCount;
locked 50% of time Unsynchronized access at BlockBlobAppendStream.java:50% of
time Unsynchronized access at BlockBlobAppendStream.java:[line 807] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.azure.BlockBlobAppendStream.bufferSize; locked 60% of time
Unsynchronized access at BlockBlobAppendStream.java:60% of time
Unsynchronized access at BlockBlobAppendStream.java:[line 875] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.azure.BlockBlobAppendStream.outBuffer; locked 50% of time
Unsynchronized access at BlockBlobAppendStream.java:50% of time Unsynchronized
access at BlockBlobAppendStream.java:[line 413] |
| | Should
org.apache.hadoop.fs.azure.BlockBlobAppendStream$ByteArrayOutputStreamInternal
be a _static_ inner class? At BlockBlobAppendStream.java:inner class? At
BlockBlobAppendStream.java:[lines 527-532] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12876296/HADOOP-14520-4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9065c56a0341 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
