[jira] [Updated] (HADOOP-12474) Fix data race when allocating server port in MiniKMS
[
https://issues.apache.org/jira/browse/HADOOP-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-12474:
---
Status: Patch Available (was: Open)
> Fix data race when allocating server port in MiniKMS
> -
>
> Key: HADOOP-12474
> URL: https://issues.apache.org/jira/browse/HADOOP-12474
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-12474.000.patch
>
>
> In {{MiniKMS}}, there is potential data race when create jetty server in
> {{createJettyServer}}. It looks like the code searches for a free port and
> then starts jetty, but maybe there's enough of a race condition between port
> location and jetty.start to cause intermittent failures.
> {code}
> ServerSocket ss = new ServerSocket((inPort < 0) ? 0 : inPort, 50,
> localhost);
> int port = ss.getLocalPort();
> Server server = new Server(0);
> if (!ssl) {
> server.getConnectors()[0].setHost(host);
> server.getConnectors()[0].setPort(port);
> } else {
> ...
> c.setPort(port);
> {code}
> We've seen test failures saying {{java.net.BindException: Address already in
> use}}, e.g.
> [https://builds.apache.org/job/PreCommit-HDFS-Build/12942/testReport/]
> As in [HADOOP-12417], we should always bind port 0 which gives us an
> ephemeral port, instead of searching a free port before starting jetty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12474) Fix data race when allocating server port in MiniKMS
[
https://issues.apache.org/jira/browse/HADOOP-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-12474:
---
Attachment: HADOOP-12474.000.patch
The v0 patch address this issue as we addressed the issue in [HADOOP-12417].
> Fix data race when allocating server port in MiniKMS
> -
>
> Key: HADOOP-12474
> URL: https://issues.apache.org/jira/browse/HADOOP-12474
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-12474.000.patch
>
>
> In {{MiniKMS}}, there is potential data race when create jetty server in
> {{createJettyServer}}. It looks like the code searches for a free port and
> then starts jetty, but maybe there's enough of a race condition between port
> location and jetty.start to cause intermittent failures.
> {code}
> ServerSocket ss = new ServerSocket((inPort < 0) ? 0 : inPort, 50,
> localhost);
> int port = ss.getLocalPort();
> Server server = new Server(0);
> if (!ssl) {
> server.getConnectors()[0].setHost(host);
> server.getConnectors()[0].setPort(port);
> } else {
> ...
> c.setPort(port);
> {code}
> We've seen test failures saying {{java.net.BindException: Address already in
> use}}, e.g.
> [https://builds.apache.org/job/PreCommit-HDFS-Build/12942/testReport/]
> As in [HADOOP-12417], we should always bind port 0 which gives us an
> ephemeral port, instead of searching a free port before starting jetty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12474) Fix data race when allocating server port in MiniKMS
Mingliang Liu created HADOOP-12474:
--
Summary: Fix data race when allocating server port in MiniKMS
Key: HADOOP-12474
URL: https://issues.apache.org/jira/browse/HADOOP-12474
Project: Hadoop Common
Issue Type: Bug
Reporter: Mingliang Liu
Assignee: Mingliang Liu
In {{MiniKMS}}, there is potential data race when create jetty server in
{{createJettyServer}}. It looks like the code searches for a free port and then
starts jetty, but maybe there's enough of a race condition between port
location and jetty.start to cause intermittent failures.
{code}
ServerSocket ss = new ServerSocket((inPort < 0) ? 0 : inPort, 50,
localhost);
int port = ss.getLocalPort();
Server server = new Server(0);
if (!ssl) {
server.getConnectors()[0].setHost(host);
server.getConnectors()[0].setPort(port);
} else {
...
c.setPort(port);
{code}
We've seen test failures saying {{java.net.BindException: Address already in
use}}, e.g.
[https://builds.apache.org/job/PreCommit-HDFS-Build/12942/testReport/]
As in [HADOOP-12417], we should always bind port 0 which gives us an ephemeral
port, instead of searching a free port before starting jetty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients
[
https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954431#comment-14954431
]
Aaron Fabbri commented on HADOOP-11684:
---
+1 on the 002 patch.
I'm attaching a 004 patch that is identical, except it adds an extra unit test
on {{BlockingThreadPoolExecutorService}}. It is optional, but demonstrates
some testing I've done. These blobstore tests don't run by default, but my
test does depend on timing, so it could theoretically fail on a heavily loaded
system. (I detect whether or not the threadpool blocks via elapsed system time
across {{submit()}} calls, and these aren't realtime systems.}
Reviewers: The idea behind the 002 approach pretty simple. The meat of the
change is just a semaphore around {{ExecutorService}} functions {{submit()}}
etc.. e.g.
{code}
+ @Override
+ public ListenableFuture submit(Runnable task) {
+try {
+ queueingPermits.acquire();
+} catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return Futures.immediateFailedCheckedFuture(e);
+}
+return super.submit(new RunnableWithPermitRelease(task));
+ }
{code}
> S3a to use thread pool that blocks clients
> --
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch,
> HADOOP-11684-003.patch, HADOOP-11684-004.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload
> wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them,
> rather than throwing an exception. F.i. something similar to
> https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11684) S3a to use thread pool that blocks clients
[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-11684: -- Attachment: HADOOP-11684-004.patch > S3a to use thread pool that blocks clients > -- > > Key: HADOOP-11684 > URL: https://issues.apache.org/jira/browse/HADOOP-11684 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch, > HADOOP-11684-003.patch, HADOOP-11684-004.patch > > > Currently, if fs.s3a.max.total.tasks are queued and another (part)upload > wants to start, a RejectedExecutionException is thrown. > We should use a threadpool that blocks clients, nicely throtthling them, > rather than throwing an exception. F.i. something similar to > https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12469) distcp should not ignore the ignoreFailures option
[
https://issues.apache.org/jira/browse/HADOOP-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated HADOOP-12469:
---
Summary: distcp should not ignore the ignoreFailures option (was: distcp
shout not ignore the ignoreFailures option)
> distcp should not ignore the ignoreFailures option
> --
>
> Key: HADOOP-12469
> URL: https://issues.apache.org/jira/browse/HADOOP-12469
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Gera Shegalov
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HADOOP-12469.000.patch
>
>
> {{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
> # via {{RetriableCommand::execute}}
> # via {{CopyMapper#copyFileWithRetry}}
> before {{CopyMapper::handleFailure}} tests
> {code}
> if (ignoreFailures && exception.getCause() instanceof
> RetriableFileCopyCommand.CopyReadException
> {code}
> which is always false.
> Orthogonally, ignoring failures should be mutually exclusive with the atomic
> option otherwise an incomplete dir is eligible for commit defeating the
> purpose.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-10775) Shell operations to fail with meaningful errors on windows if winutils.exe not found
[
https://issues.apache.org/jira/browse/HADOOP-10775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-10775:
---
Hadoop Flags: Reviewed
+1 for patch v009. I confirmed a full test run on Windows.
Whitespace issues can be fixed on commit.
I have one more very minor nitpick.
{code}
-if (Shell.WINDOWS && !Shell.isJava7OrAbove()) {
- // On Java6 on Windows, we copied the file
- Assert.assertEquals(data.length, link.length());
-} else {
- // Otherwise, the target file size is zero
- Assert.assertEquals(0, link.length());
-}
+// Otherwise, the target file size is zero
+Assert.assertEquals(0, link.length());
{code}
Now that the conditional logic is gone, the "otherwise" phrasing of the comment
doesn't make sense. I think the whole comment can be removed at this point.
Feel free to drop it during the final commit. IMO, there is no need to go
through full review of another patch revision for this minor tweak.
[~ste...@apache.org], thank you for contributing the patch.
> Shell operations to fail with meaningful errors on windows if winutils.exe
> not found
>
>
> Key: HADOOP-10775
> URL: https://issues.apache.org/jira/browse/HADOOP-10775
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: trunk-win, 2.7.1
> Environment: windows
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-10775-002.patch, HADOOP-10775-003.patch,
> HADOOP-10775-004.patch, HADOOP-10775-005.patch, HADOOP-10775-006.patch,
> HADOOP-10775-007.patch, HADOOP-10775-008.patch, HADOOP-10775-009.patch,
> HADOOP-10775.patch
>
>
> If {{winutils.exe}} can't be found {{HADOOP_HOME}} wrong/unset or other
> causes, then an error is logged -but when any of the {{Shell}} operations are
> used, an NPE is raised rather than something meaningful.
> The error message at setup time should be preserved and then raised before
> any attempt to invoke a winutils-driven process made
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[
https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953794#comment-14953794
]
Hadoop QA commented on HADOOP-11683:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 1s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766168/HADOOP-11683.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9849c8b |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7796/console |
This message was automatically generated.
> Need a plugin API to translate long principal names to local OS user names
> arbitrarily
> --
>
> Key: HADOOP-11683
> URL: https://issues.apache.org/jira/browse/HADOOP-11683
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Sunny Cheung
>Assignee: roger mak
> Attachments: HADOOP-11683.001.patch, HADOOP-11683.002.patch
>
>
> We need a plugin API to translate long principal names (e.g.
> john@example.com) to local OS user names (e.g. user123456) arbitrarily.
> For some organizations the name translation is straightforward (e.g.
> john@example.com to john_doe), and the hadoop.security.auth_to_local
> configurable mapping is sufficient to resolve this (see HADOOP-6526).
> However, in some other cases the name translation is arbitrary and cannot be
> generalized by a set of translation rules easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roger mak updated HADOOP-11683: --- Target Version/s: 2.6.2 Status: Patch Available (was: Open) submit for review. > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch, HADOOP-11683.002.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roger mak updated HADOOP-11683: --- Attachment: HADOOP-11683.002.patch Enhanced test logic. > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch, HADOOP-11683.002.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953702#comment-14953702 ] roger mak commented on HADOOP-11683: Update patch for trunk. > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roger mak updated HADOOP-11683: --- Status: Open (was: Patch Available) > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12364) Deleting pid file after stop is causing the daemons to keep restarting
[ https://issues.apache.org/jira/browse/HADOOP-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953684#comment-14953684 ] Siqi Li commented on HADOOP-12364: -- Hi [~aw], can you take a look at the latest patch, I have fixed the shell check and change double bracket to single bracket as you mentioned above > Deleting pid file after stop is causing the daemons to keep restarting > --- > > Key: HADOOP-12364 > URL: https://issues.apache.org/jira/browse/HADOOP-12364 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Siqi Li >Assignee: Siqi Li >Priority: Minor > Attachments: HADOOP-12364.v1.patch, HADOOP-12364.v2.patch, > HADOOP-12364.v3.patch, HADOOP-12364.v4.patch > > > pid files are deleting in 5 seconds after we stop the daemons. If a start > command were executed within the 5 seconds, the pid file will be overwrite > with a new pid. However, this pid file is going to be deleted by the former > stop command. This is causing the monitoring service to lose track of the > daemons, hence keep rebooting them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12469) distcp shout not ignore the ignoreFailures option
[
https://issues.apache.org/jira/browse/HADOOP-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953458#comment-14953458
]
Mingliang Liu commented on HADOOP-12469:
Thanks [~wheat9] for reviewing this patch. I should make it clear that it was
not to address the whole issue when I upload the patch. Sorry for the confusion.
As [~jira.shegalov] suggested, I'll update the patch with tests enabled. I
filed a new issue [HADOOP-12473] about the atomic option for ignoring failures.
> distcp shout not ignore the ignoreFailures option
> -
>
> Key: HADOOP-12469
> URL: https://issues.apache.org/jira/browse/HADOOP-12469
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Gera Shegalov
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HADOOP-12469.000.patch
>
>
> {{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
> # via {{RetriableCommand::execute}}
> # via {{CopyMapper#copyFileWithRetry}}
> before {{CopyMapper::handleFailure}} tests
> {code}
> if (ignoreFailures && exception.getCause() instanceof
> RetriableFileCopyCommand.CopyReadException
> {code}
> which is always false.
> Orthogonally, ignoring failures should be mutually exclusive with the atomic
> option otherwise an incomplete dir is eligible for commit defeating the
> purpose.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option
[
https://issues.apache.org/jira/browse/HADOOP-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-12473:
---
Description: In {{CopyMapper::handleFailure}}, the mapper handles failure
and will ignore it if no it's config key is on. Ignoring failures should be
mutually exclusive with the atomic option otherwise an incomplete dir is
eligible for commit defeating the purpose. (was: In
{{CopyMapper::handleFailure}}, the mapper handles failures Ignoring failures
should be mutually exclusive with the atomic option otherwise an incomplete dir
is eligible for commit defeating the purpose.)
> distcp's ignoring failures should be mutually exclusive with the atomic option
> --
>
> Key: HADOOP-12473
> URL: https://issues.apache.org/jira/browse/HADOOP-12473
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.0
>
>
> In {{CopyMapper::handleFailure}}, the mapper handles failure and will ignore
> it if no it's config key is on. Ignoring failures should be mutually
> exclusive with the atomic option otherwise an incomplete dir is eligible for
> commit defeating the purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option
[
https://issues.apache.org/jira/browse/HADOOP-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-12473:
---
Assignee: (was: Mingliang Liu)
> distcp's ignoring failures should be mutually exclusive with the atomic option
> --
>
> Key: HADOOP-12473
> URL: https://issues.apache.org/jira/browse/HADOOP-12473
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.0
>
>
> In {{CopyMapper::handleFailure}}, the mapper handles failures Ignoring
> failures should be mutually exclusive with the atomic option otherwise an
> incomplete dir is eligible for commit defeating the purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option
Mingliang Liu created HADOOP-12473:
--
Summary: distcp's ignoring failures should be mutually exclusive
with the atomic option
Key: HADOOP-12473
URL: https://issues.apache.org/jira/browse/HADOOP-12473
Project: Hadoop Common
Issue Type: Bug
Components: tools/distcp
Affects Versions: 2.7.1
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Critical
Fix For: 2.8.0
{{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
# via {{RetriableCommand::execute}}
# via {{CopyMapper#copyFileWithRetry}}
before {{CopyMapper::handleFailure}} tests
{code}
if (ignoreFailures && exception.getCause() instanceof
RetriableFileCopyCommand.CopyReadException
{code}
which is always false.
Orthogonally, ignoring failures should be mutually exclusive with the atomic
option otherwise an incomplete dir is eligible for commit defeating the purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option
[
https://issues.apache.org/jira/browse/HADOOP-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-12473:
---
Description: In {{CopyMapper::handleFailure}}, the mapper handles failures
Ignoring failures should be mutually exclusive with the atomic option otherwise
an incomplete dir is eligible for commit defeating the purpose. (was:
{{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
# via {{RetriableCommand::execute}}
# via {{CopyMapper#copyFileWithRetry}}
before {{CopyMapper::handleFailure}} tests
{code}
if (ignoreFailures && exception.getCause() instanceof
RetriableFileCopyCommand.CopyReadException
{code}
which is always false.
Orthogonally, ignoring failures should be mutually exclusive with the atomic
option otherwise an incomplete dir is eligible for commit defeating the purpose.
)
> distcp's ignoring failures should be mutually exclusive with the atomic option
> --
>
> Key: HADOOP-12473
> URL: https://issues.apache.org/jira/browse/HADOOP-12473
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.0
>
>
> In {{CopyMapper::handleFailure}}, the mapper handles failures Ignoring
> failures should be mutually exclusive with the atomic option otherwise an
> incomplete dir is eligible for commit defeating the purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953115#comment-14953115 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #484 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/484/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/CHANGES.txt > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953105#comment-14953105 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2422 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2422/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java * hadoop-common-project/hadoop-common/CHANGES.txt > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952910#comment-14952910 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #524 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/524/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10296) Incorrect null check in SwiftRestClient#buildException()
[
https://issues.apache.org/jira/browse/HADOOP-10296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952892#comment-14952892
]
Kanaka Kumar Avvaru commented on HADOOP-10296:
--
Thanks [~ajisakaa] for the review & commit
> Incorrect null check in SwiftRestClient#buildException()
>
>
> Key: HADOOP-10296
> URL: https://issues.apache.org/jira/browse/HADOOP-10296
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Ted Yu
>Assignee: Kanaka Kumar Avvaru
>Priority: Minor
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HADOOP-10296.1.patch, HADOOP-10296.2.patch
>
>
> {code}
> if (requestContentLen!=null) {
> errorText.append(" available
> ").append(availableContentRange.getValue());
> }
> {code}
> The null check should be for availableContentRange
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952875#comment-14952875 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-Yarn-trunk #1249 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1249/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/CHANGES.txt > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952857#comment-14952857 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #512 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/512/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952835#comment-14952835 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2458 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2458/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952803#comment-14952803 ] Hudson commented on HADOOP-10300: - FAILURE: Integrated in Hadoop-trunk-Commit #8611 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8611/]) HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp (yliu: rev e617cf6dd13f2bb5d7cbb15ee2cdb260ecd46cd3) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPCServerResponder.java > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10300: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952778#comment-14952778 ] Yi Liu commented on HADOOP-10300: - +1, thanks [~daryn]. > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Attachments: HADOOP-10300.patch, HADOOP-10300.patch, > HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
