[jira] [Created] (HADOOP-15373) Clear the code of Callqueuemanager
maobaolong created HADOOP-15373: --- Summary: Clear the code of Callqueuemanager Key: HADOOP-15373 URL: https://issues.apache.org/jira/browse/HADOOP-15373 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.2.0 Reporter: maobaolong Assignee: maobaolong The CallQueueManager contain two member, putRef and takeRef. They seems shouldn't have to use AtomicReference, it can use the volatile directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15361) RawLocalFileSystem should use Java nio framework for rename
[
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429576#comment-16429576
]
Andras Bokor commented on HADOOP-15361:
---
[~ste...@apache.org],
I am bit confused how to resolve the caveats.
{quote}The compatibility is the troublespot here. How does it relate to what we
have in filesystem.md?
{quote}
There are some caveats and differences between RawLocal and HDFS that could be
affected:
* filsystem.md states that if the source does not exist we should throw
FileNotFoundException but HDFS does not throw exception and contract test also
expects only a false (FileSystemContractBaseTest#testRenameNonExistentPath).
* local filesystem is able to replace a file but HDFS does not
* if the parent folder of the destination does not exist HDFS fails but local
filesystem creates the missing directories.
What is best strategy here? Should we keep the sync with filesystem.md or
follow HDFS and the contract tests?
For me it seems filesystem.md just states what is happening, these behaviors
are not intended.
> RawLocalFileSystem should use Java nio framework for rename
> ---
>
> Key: HADOOP-15361
> URL: https://issues.apache.org/jira/browse/HADOOP-15361
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Labels: incompatibleChange
> Attachments: HADOOP-15361.01.patch
>
>
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames.
> The fallback logic is a copy-on-fail logic so when rename fails it copies the
> source then delete it.
> An additional fallback logic was needed for Windows to provide POSIX rename
> behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests
> (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not
> platform dependent and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io
> overrides the destination only if the source is also a directory but
> handleEmptyDstDirectoryOnWindows method checks only the destination. That
> means rename allows to override a directory with a file on Windows but not on
> Unix.
> File#renameTo and Files#move are not 100% compatible:
> If the source is a directory and the destination is an empty directory
> File#renameTo overrides the source but Files#move is does not. We have to use
> {{StandardCopyOption.REPLACE_EXISTING}} but it overrides the destination even
> if the source or the destination is a file. So to make them compatible we
> have to check that the either the source or the destination is a directory
> before we add the copy option.
> I think the correct strategy is
> * Where the contract test passed so far it should pass after this
> * Where the contract test failed because of Java specific think and not
> because of the fallback logic we should keep the original behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429574#comment-16429574
]
genericqa commented on HADOOP-14999:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 2m
19s{color} | {color:red} Docker failed to build yetus/hadoop:dbd69cb. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14999 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917956/HADOOP-14999-branch-2.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14450/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Major
> Attachments: HADOOP-14999-branch-2.001.patch,
> HADOOP-14999-branch-2.002.patch, HADOOP-14999.001.patch,
> HADOOP-14999.002.patch, HADOOP-14999.003.patch, HADOOP-14999.004.patch,
> HADOOP-14999.005.patch, HADOOP-14999.006.patch, HADOOP-14999.007.patch,
> HADOOP-14999.008.patch, HADOOP-14999.009.patch, HADOOP-14999.010.patch,
> HADOOP-14999.011.patch, asynchronous_file_uploading.pdf,
> diff-between-patch7-and-patch8.txt
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
> This patch reuse {{SemaphoredDelegatingExecutor}} as executor service and
> depends on HADOOP-15039.
> Attached {{asynchronous_file_uploading.pdf}} illustrated the difference
> between previous {{AliyunOSSOutputStream}} and
> {{AliyunOSSBlockOutputStream}}, i.e. this asynchronous multi-part based
> uploading mechanism.
> 1. {{AliyunOSSOutputStream}}: we need to output the whole result to local
> disk before we can upload it to OSS. This will poses two problems：
> - if the output file is too large, it will run out of the local disk.
> - if the output file is too large, task will wait long time to upload result
> to OSS before finish, wasting much compute resource.
> 2. {{AliyunOSSBlockOutputStream}}: we cut the task output into small blocks,
> i.e. some small local file, and each block will be packaged into a uploading
> task. These tasks will be submitted into {{SemaphoredDelegatingExecutor}}.
> {{SemaphoredDelegatingExecutor}} will upload this blocks in parallel, this
> will improve performance greatly.
> 3. Each task will retry 3 times to upload block to Aliyun OSS. If one of
> those tasks failed, the whole file uploading will failed, and we will abort
> current uploading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-14999:
---
Attachment: HADOOP-14999-branch-2.002.patch
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Major
> Attachments: HADOOP-14999-branch-2.001.patch,
> HADOOP-14999-branch-2.002.patch, HADOOP-14999.001.patch,
> HADOOP-14999.002.patch, HADOOP-14999.003.patch, HADOOP-14999.004.patch,
> HADOOP-14999.005.patch, HADOOP-14999.006.patch, HADOOP-14999.007.patch,
> HADOOP-14999.008.patch, HADOOP-14999.009.patch, HADOOP-14999.010.patch,
> HADOOP-14999.011.patch, asynchronous_file_uploading.pdf,
> diff-between-patch7-and-patch8.txt
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
> This patch reuse {{SemaphoredDelegatingExecutor}} as executor service and
> depends on HADOOP-15039.
> Attached {{asynchronous_file_uploading.pdf}} illustrated the difference
> between previous {{AliyunOSSOutputStream}} and
> {{AliyunOSSBlockOutputStream}}, i.e. this asynchronous multi-part based
> uploading mechanism.
> 1. {{AliyunOSSOutputStream}}: we need to output the whole result to local
> disk before we can upload it to OSS. This will poses two problems：
> - if the output file is too large, it will run out of the local disk.
> - if the output file is too large, task will wait long time to upload result
> to OSS before finish, wasting much compute resource.
> 2. {{AliyunOSSBlockOutputStream}}: we cut the task output into small blocks,
> i.e. some small local file, and each block will be packaged into a uploading
> task. These tasks will be submitted into {{SemaphoredDelegatingExecutor}}.
> {{SemaphoredDelegatingExecutor}} will upload this blocks in parallel, this
> will improve performance greatly.
> 3. Each task will retry 3 times to upload block to Aliyun OSS. If one of
> those tasks failed, the whole file uploading will failed, and we will abort
> current uploading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15365) cannot find symbol sun.misc.Cleaner
[
https://issues.apache.org/jira/browse/HADOOP-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429572#comment-16429572
]
maobaolong edited comment on HADOOP-15365 at 4/8/18 1:24 AM:
-
[~tasanuma0829] [~ajayydv] Thank you for remind me, i think the HADOOP-12760 is
same as mine, so I close this jira and mark as duplicate
was (Author: maobaolong):
[~tasanuma0829] [~ajayydv] Thank you for remind me, i think the HADOOP-12760 is
same as mine, so you can close this jira and mark as duplicate
> cannot find symbol sun.misc.Cleaner
> ---
>
> Key: HADOOP-15365
> URL: https://issues.apache.org/jira/browse/HADOOP-15365
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HADOOP-15365.001.patch
>
>
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[332,17]
> cannot find symbol
> symbol: class Cleaner
> location: package sun.misc
> [ERROR]
> /hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:[40,21]
> cannot find symbol
> symbol: class Cleaner
> location: package sun.misc
> {code}
> We should use reflection to adapt jdk8 and jdk9
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15365) cannot find symbol sun.misc.Cleaner
[
https://issues.apache.org/jira/browse/HADOOP-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong resolved HADOOP-15365.
-
Resolution: Duplicate
> cannot find symbol sun.misc.Cleaner
> ---
>
> Key: HADOOP-15365
> URL: https://issues.apache.org/jira/browse/HADOOP-15365
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HADOOP-15365.001.patch
>
>
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[332,17]
> cannot find symbol
> symbol: class Cleaner
> location: package sun.misc
> [ERROR]
> /hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:[40,21]
> cannot find symbol
> symbol: class Cleaner
> location: package sun.misc
> {code}
> We should use reflection to adapt jdk8 and jdk9
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15365) cannot find symbol sun.misc.Cleaner
[
https://issues.apache.org/jira/browse/HADOOP-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429572#comment-16429572
]
maobaolong commented on HADOOP-15365:
-
[~tasanuma0829] [~ajayydv] Thank you for remind me, i think the HADOOP-12760 is
same as mine, so you can close this jira and mark as duplicate
> cannot find symbol sun.misc.Cleaner
> ---
>
> Key: HADOOP-15365
> URL: https://issues.apache.org/jira/browse/HADOOP-15365
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HADOOP-15365.001.patch
>
>
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[332,17]
> cannot find symbol
> symbol: class Cleaner
> location: package sun.misc
> [ERROR]
> /hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:[40,21]
> cannot find symbol
> symbol: class Cleaner
> location: package sun.misc
> {code}
> We should use reflection to adapt jdk8 and jdk9
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15366) Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429314#comment-16429314
]
Hudson commented on HADOOP-15366:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13940 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13940/])
HADOOP-15366. Add a helper shutdown routine in HadoopExecutor to ensure
(msingh: rev 0b345b765370515d7222154ad5cae9b86f137a76)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/HadoopExecutors.java
> Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15366) Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HADOOP-15366:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
Thanks for the contribution [~shashikant]. I have committed this to the trunk.
> Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15366) Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HADOOP-15366:
---
Summary: Add a helper shutdown routine in HadoopExecutor to ensure clean
shutdown (was: Add a helper shutDown routine in HadoopExecutor to ensure clean
shutdown)
> Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15366) Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429292#comment-16429292
]
Mukul Kumar Singh commented on HADOOP-15366:
Thanks for working on this [~shashikant].
+1, v0 patch looks good to me. I will commit this shortly.
> Add a helper shutDown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
