[REMINDER] How to set fix versions when committing

[Andrew Wang]

Hi all,

I finished the bulk fix version update and just rebranched
branch-3.0.0-alpha1 off of trunk. So, a reminder that the procedure for
setting fix versions has changed slightly from before.

Everything is fully detailed here, the example in particular should help
clarify things:

https://hadoop.apache.org/versioning.html

The short of it though is that if a JIRA is going into trunk or
branch-3.0.0-alpha1, it should also have a 3.0.0-alpha1 or 3.0.0-alpha2
fixVersion set.

Thanks,
Andrew

Re: Turning off JIRA notifications temporarily at ~6PM PDT for bulk JIRA fix version update

[Andrew Wang]

Script has finished, JIRA notifications should be back on. If you notice
any strangeness, let me know.

I'll send a separate email blast reminding people about branches and fix
versions.

On Mon, Aug 29, 2016 at 5:56 PM, Andrew Wang 
wrote:

> Starting this now, wish me luck...
>
> On Mon, Aug 29, 2016 at 2:18 PM, Andrew Wang 
> wrote:
>
>> Hi folks,
>>
>> I'm planning to pull the trigger on a bulk JIRA update to add the
>> 3.0.0-alpha1 fix version to a bunch of JIRAs, based on the versioning
>> scheme put forth in a previous set of common-dev emails [1]. Vinod asked me
>> to hold off on the 2.8.0 update, I think he'll run it separately.
>>
>> I've talked to infra about this, and the best way of avoiding a flood of
>> notifications is to temporarily disable JIRA update notifications while the
>> bulk update is running.
>>
>> If you'd like to review the script (and most importantly, the JIRA
>> query), it's available here:
>>
>> https://github.com/umbrant/jira-update
>>
>> I've already gotten a few people to look at it and I've done some test
>> runs, so I think it's ready to go.
>>
>> I plan to run the script around 6PM PDT (3.75 hours from now), and will
>> send out an all-clear when everything is done.
>>
>> Thanks,
>> Andrew
>>
>> [1]: https://lists.apache.org/thread.html/5bcff03c5de719651c
>> 621563b2653b914ee03c352d5aee76abe58284@%3Ccommon-dev.hadoop.apache.org%3E
>>
>
>

Re: Turning off JIRA notifications temporarily at ~6PM PDT for bulk JIRA fix version update

[Andrew Wang]

Starting this now, wish me luck...

On Mon, Aug 29, 2016 at 2:18 PM, Andrew Wang 
wrote:

> Hi folks,
>
> I'm planning to pull the trigger on a bulk JIRA update to add the
> 3.0.0-alpha1 fix version to a bunch of JIRAs, based on the versioning
> scheme put forth in a previous set of common-dev emails [1]. Vinod asked me
> to hold off on the 2.8.0 update, I think he'll run it separately.
>
> I've talked to infra about this, and the best way of avoiding a flood of
> notifications is to temporarily disable JIRA update notifications while the
> bulk update is running.
>
> If you'd like to review the script (and most importantly, the JIRA query),
> it's available here:
>
> https://github.com/umbrant/jira-update
>
> I've already gotten a few people to look at it and I've done some test
> runs, so I think it's ready to go.
>
> I plan to run the script around 6PM PDT (3.75 hours from now), and will
> send out an all-clear when everything is done.
>
> Thanks,
> Andrew
>
> [1]: https://lists.apache.org/thread.html/5bcff03c5de719651c621563b2653b
> 914ee03c352d5aee76abe58284@%3Ccommon-dev.hadoop.apache.org%3E
>

[jira] [Reopened] (HADOOP-13286) add a S3A scale test to do gunzip and linecount

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-13286:
--

> add a S3A scale test to do gunzip and linecount
> ---
>
> Key: HADOOP-13286
> URL: https://issues.apache.org/jira/browse/HADOOP-13286
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13286-branch-2-001.patch
>
>
> the HADOOP-13203 patch proposal showed that there were performance problems 
> downstream which weren't surfacing in the current scale tests.
> Trying to decompress the .gz test file and then go through it with LineReader 
> models a basic use case: parse a .csv.gz data source. 
> Add this, with metric printing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13286) add a S3A scale test to do gunzip and linecount

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-13286.
--
   Resolution: Duplicate
Fix Version/s: (was: 2.8.0)

> add a S3A scale test to do gunzip and linecount
> ---
>
> Key: HADOOP-13286
> URL: https://issues.apache.org/jira/browse/HADOOP-13286
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13286-branch-2-001.patch
>
>
> the HADOOP-13203 patch proposal showed that there were performance problems 
> downstream which weren't surfacing in the current scale tests.
> Trying to decompress the .gz test file and then go through it with LineReader 
> models a basic use case: parse a .csv.gz data source. 
> Add this, with metric printing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-12976) s3a toString to be meaningful in logs

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-12976.
--
Resolution: Duplicate

> s3a toString to be meaningful in logs
> -
>
> Key: HADOOP-12976
> URL: https://issues.apache.org/jira/browse/HADOOP-12976
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Fix For: 2.8.0
>
>
> today's toString value is just the object ref; better to include the URL of 
> the FS
> Example:
> {code}
> Cleaning filesystem org.apache.hadoop.fs.s3a.S3AFileSystem@1f069dc1 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-12997) s3a to pass PositionedReadable contract tests, improve readFully perf.

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-12997.
--
   Resolution: Duplicate
Fix Version/s: (was: 2.8.0)

> s3a to pass PositionedReadable contract tests, improve readFully perf.
> --
>
> Key: HADOOP-12997
> URL: https://issues.apache.org/jira/browse/HADOOP-12997
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Fix s3a so that it passes the new tests in HADOOP-12994
> Also: optimise readFully so that instead of a sequence of seek-read-seek 
> operations, it does an opening seek and retains that position as it loops 
> through the data



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-12997) s3a to pass PositionedReadable contract tests, improve readFully perf.

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-12997:
--

> s3a to pass PositionedReadable contract tests, improve readFully perf.
> --
>
> Key: HADOOP-12997
> URL: https://issues.apache.org/jira/browse/HADOOP-12997
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Fix s3a so that it passes the new tests in HADOOP-12994
> Also: optimise readFully so that instead of a sequence of seek-read-seek 
> operations, it does an opening seek and retains that position as it loops 
> through the data



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-12976) s3a toString to be meaningful in logs

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-12976:
--

> s3a toString to be meaningful in logs
> -
>
> Key: HADOOP-12976
> URL: https://issues.apache.org/jira/browse/HADOOP-12976
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Fix For: 2.8.0
>
>
> today's toString value is just the object ref; better to include the URL of 
> the FS
> Example:
> {code}
> Cleaning filesystem org.apache.hadoop.fs.s3a.S3AFileSystem@1f069dc1 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-12762) task: null java.lang.unsupportedoperationexception: this is supposed to be overridden by subclasses.

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-12762.
--
Resolution: Cannot Reproduce

> task: null java.lang.unsupportedoperationexception: this is supposed to be 
> overridden by subclasses.
> 
>
> Key: HADOOP-12762
> URL: https://issues.apache.org/jira/browse/HADOOP-12762
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.2
>Reporter: Padma
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-12420) While trying to access Amazon S3 through hadoop-aws(Spark basically) I was getting Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.tr

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-12420:
--

> While trying to access Amazon S3 through hadoop-aws(Spark basically) I was 
> getting Exception in thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> --
>
> Key: HADOOP-12420
> URL: https://issues.apache.org/jira/browse/HADOOP-12420
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Tariq Mohammad
>Assignee: Tariq Mohammad
>Priority: Minor
>
> While trying to access data stored in Amazon S3 through Apache Spark, which  
> internally uses hadoop-aws jar I was getting the following exception :
> Exception in thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> Probable reason could be the fact that aws java sdk expects a long parameter 
> for the setMultipartUploadThreshold(long multiPartThreshold) method, but 
> hadoop-aws was using a parameter of type int(multiPartThreshold). 
> I tried using the downloaded hadoop-aws jar and the build through its maven 
> dependency, but in both the cases I encountered the same exception. Although 
> I can see private long multiPartThreshold; in hadoop-aws GitHub repo, it's 
> not getting reflected in the downloaded jar or in the jar created from maven 
> dependency.
> Following lines in the S3AFileSystem class create this difference :
> Build from trunk : 
> private long multiPartThreshold;
> this.multiPartThreshold = conf.getLong("fs.s3a.multipart.threshold", 
> 2147483647L); => Line 267
> Build through maven dependency : 
> private int multiPartThreshold;
> multiPartThreshold = conf.getInt(MIN_MULTIPART_THRESHOLD, 
> DEFAULT_MIN_MULTIPART_THRESHOLD); => Line 249



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-12420) While trying to access Amazon S3 through hadoop-aws(Spark basically) I was getting Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.tr

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-12420.
--
   Resolution: Duplicate
Fix Version/s: (was: 2.8.0)

> While trying to access Amazon S3 through hadoop-aws(Spark basically) I was 
> getting Exception in thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> --
>
> Key: HADOOP-12420
> URL: https://issues.apache.org/jira/browse/HADOOP-12420
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Tariq Mohammad
>Assignee: Tariq Mohammad
>Priority: Minor
>
> While trying to access data stored in Amazon S3 through Apache Spark, which  
> internally uses hadoop-aws jar I was getting the following exception :
> Exception in thread "main" java.lang.NoSuchMethodError: 
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> Probable reason could be the fact that aws java sdk expects a long parameter 
> for the setMultipartUploadThreshold(long multiPartThreshold) method, but 
> hadoop-aws was using a parameter of type int(multiPartThreshold). 
> I tried using the downloaded hadoop-aws jar and the build through its maven 
> dependency, but in both the cases I encountered the same exception. Although 
> I can see private long multiPartThreshold; in hadoop-aws GitHub repo, it's 
> not getting reflected in the downloaded jar or in the jar created from maven 
> dependency.
> Following lines in the S3AFileSystem class create this difference :
> Build from trunk : 
> private long multiPartThreshold;
> this.multiPartThreshold = conf.getLong("fs.s3a.multipart.threshold", 
> 2147483647L); => Line 267
> Build through maven dependency : 
> private int multiPartThreshold;
> multiPartThreshold = conf.getInt(MIN_MULTIPART_THRESHOLD, 
> DEFAULT_MIN_MULTIPART_THRESHOLD); => Line 249



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-12319) S3AFastOutputStream has no ability to apply backpressure

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-12319:
--

> S3AFastOutputStream has no ability to apply backpressure
> 
>
> Key: HADOOP-12319
> URL: https://issues.apache.org/jira/browse/HADOOP-12319
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Colin Marc
>Priority: Critical
>
> Currently, users of S3AFastOutputStream can control memory usage with a few 
> settings: {{fs.s3a.threads.core,max}}, which control the number of active 
> uploads (specifically as arguments to a {{ThreadPoolExecutor}}), and 
> {{fs.s3a.max.total.tasks}}, which controls the size of the feeding queue for 
> the {{ThreadPoolExecutor}}.
> However, a user can get an almost *guaranteed* crash if the throughput of the 
> writing job is higher than the total S3 throughput, because there is never 
> any backpressure or blocking on calls to {{write}}.
> If {{fs.s3a.max.total.tasks}} is set high (the default is 1000), then 
> {{write}} calls will continue to add data to the queue, which can eventually 
> OOM. But if the user tries to set it lower, then writes will fail when the 
> queue is full; the {{ThreadPoolExecutor}} will reject the part with 
> {{java.util.concurrent.RejectedExecutionException}}.
> Ideally, calls to {{write}} should *block, not fail* when the queue is full, 
> so as to apply backpressure on whatever the writing process is.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-12319) S3AFastOutputStream has no ability to apply backpressure

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-12319.
--
   Resolution: Duplicate
Fix Version/s: (was: 2.8.0)

> S3AFastOutputStream has no ability to apply backpressure
> 
>
> Key: HADOOP-12319
> URL: https://issues.apache.org/jira/browse/HADOOP-12319
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Colin Marc
>Priority: Critical
>
> Currently, users of S3AFastOutputStream can control memory usage with a few 
> settings: {{fs.s3a.threads.core,max}}, which control the number of active 
> uploads (specifically as arguments to a {{ThreadPoolExecutor}}), and 
> {{fs.s3a.max.total.tasks}}, which controls the size of the feeding queue for 
> the {{ThreadPoolExecutor}}.
> However, a user can get an almost *guaranteed* crash if the throughput of the 
> writing job is higher than the total S3 throughput, because there is never 
> any backpressure or blocking on calls to {{write}}.
> If {{fs.s3a.max.total.tasks}} is set high (the default is 1000), then 
> {{write}} calls will continue to add data to the queue, which can eventually 
> OOM. But if the user tries to set it lower, then writes will fail when the 
> queue is full; the {{ThreadPoolExecutor}} will reject the part with 
> {{java.util.concurrent.RejectedExecutionException}}.
> Ideally, calls to {{write}} should *block, not fail* when the queue is full, 
> so as to apply backpressure on whatever the writing process is.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-11874) s3a can throw spurious IOEs on close()

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HADOOP-11874.
--
Resolution: Duplicate

> s3a can throw spurious IOEs on close()
> --
>
> Key: HADOOP-11874
> URL: https://issues.apache.org/jira/browse/HADOOP-11874
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.8.0
>
>
> from a code review, it's clear that the issue seen in HADOOP-11851 can 
> surface in S3a, though with HADOOP-11570, it's less likely. It will only 
> happen on those cases when abort() isn't called.
> The "clean" close() code path needs to catch IOEs from the wrappedStream and 
> call abort() in that situation too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-11874) s3a can throw spurious IOEs on close()

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HADOOP-11874:
--

> s3a can throw spurious IOEs on close()
> --
>
> Key: HADOOP-11874
> URL: https://issues.apache.org/jira/browse/HADOOP-11874
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.8.0
>
>
> from a code review, it's clear that the issue seen in HADOOP-11851 can 
> surface in S3a, though with HADOOP-11570, it's less likely. It will only 
> happen on those cases when abort() isn't called.
> The "clean" close() code path needs to catch IOEs from the wrappedStream and 
> call abort() in that situation too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13557) UserGroupInformation created from a Subject incorrectly tries to renew the Keberos ticket

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-13557.

Resolution: Duplicate

Seems like a duplicate of HADOOP-13558?
Given there are already some watchers there, I'm closing this one as a dup. 
Let's follow up over there. Thanks for creating this [~tucu00].

> UserGroupInformation created from a Subject incorrectly tries to renew the 
> Keberos ticket
> -
>
> Key: HADOOP-13557
> URL: https://issues.apache.org/jira/browse/HADOOP-13557
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.2, 2.6.4, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>
> The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions 
> and if they are met it invokes the {{reloginFromKeytab()}}. The 
> {{reloginFromKeytab()}} method then fails with an {{IOException}} 
> "loginUserFromKeyTab must be done first" because there is no keytab 
> associated with the UGI.
> The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab 
> ({{isKeytab}} UGI instance variable) associated with the UGI, if there is one 
> it triggers a call to {{reloginFromKeytab()}}. The problem is that the 
> {{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned 
> {{IOException}}.
> The root of the problem seems to be when creating a UGI via the 
> {{UGI.loginUserFromSubject(Subject)}} method, this method uses the 
> {{UserGroupInformation(Subject)}} constructor, and this constructor does the 
> following to determine if there is a keytab or not.
> {code}
>   this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
> {code}
> If the {{Subject}} given had a keytab, then the UGI instance will have the 
> {{isKeytab}} set to TRUE.
> It sets the UGI instance as it would have a keytab because the Subject has a 
> keytab. This has 2 problems:
> First, it does not set the keytab file (and this, having the {{isKeytab}} set 
> to TRUE and the {{keytabFile}) set to NULL is what triggers the 
> {{IOException}} in the method {{reloginFromKeytab()}}.
> Second (and even if the first problem is fixed, this still is a problem), it 
> assumes that because the subject has a keytab it is up to UGI to to the 
> relogin using the keytab. This is incorrect if the UGI was created using the 
> {{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the 
> Subject is not the UGI, but the caller, so the caller is responsible for 
> renewing the Kerberos tickets and the UGI should not try to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13559) Remove close() within try-with-resources

[Aaron Fabbri (JIRA)]

Aaron Fabbri created HADOOP-13559:
-

 Summary: Remove close() within try-with-resources
 Key: HADOOP-13559
 URL: https://issues.apache.org/jira/browse/HADOOP-13559
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.8.0, 2.9.0
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
Priority: Minor


My colleague noticed that HADOOP-12994 introduced to places where close() was 
still called manually within a try-with-resources block.

I'll attach a patch to remove the manual close() calls. 

These extra calls to close() are probably safe, as InputStream is a Closable, 
not AutoClosable (the later does not specify close() as idempotent).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13557) UserGroupInformation created from a Subject incorrectly tries to renew the Keberos ticket

[Alejandro Abdelnur (JIRA)]

Alejandro Abdelnur created HADOOP-13557:
---

 Summary: UserGroupInformation created from a Subject incorrectly 
tries to renew the Keberos ticket
 Key: HADOOP-13557
 URL: https://issues.apache.org/jira/browse/HADOOP-13557
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.6.4, 2.7.2, 3.0.0-alpha2
Reporter: Alejandro Abdelnur


The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions and 
if they are met it invokes the {{reloginFromKeytab()}}. The 
{{reloginFromKeytab()}} method then fails with an {{IOException}} 
"loginUserFromKeyTab must be done first" because there is no keytab associated 
with the UGI.

The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab 
({{isKeytab}} UGI instance variable) associated with the UGI, if there is one 
it triggers a call to {{reloginFromKeytab()}}. The problem is that the 
{{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned 
{{IOException}}.


The root of the problem seems to be when creating a UGI via the 
{{UGI.loginUserFromSubject(Subject)}} method, this method uses the 
{{UserGroupInformation(Subject)}} constructor, and this constructor does the 
following to determine if there is a keytab or not.

{code}
  this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
{code}

If the {{Subject}} given had a keytab, then the UGI instance will have the 
{{isKeytab}} set to TRUE.

It sets the UGI instance as it would have a keytab because the Subject has a 
keytab. This has 2 problems:

First, it does not set the keytab file (and this, having the {{isKeytab}} set 
to TRUE and the {{keytabFile}) set to NULL is what triggers the {{IOException}} 
in the method {{reloginFromKeytab()}}.

Second (and even if the first problem is fixed, this still is a problem), it 
assumes that because the subject has a keytab it is up to UGI to to the relogin 
using the keytab. This is incorrect if the UGI was created using the 
{{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the 
Subject is not the UGI, but the caller, so the caller is responsible for 
renewing the Kerberos tickets and the UGI should not try to do so.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13558) UserGroupInformation created from a Subject incorrectly tries to renew the Keberos ticket

[Alejandro Abdelnur (JIRA)]

Alejandro Abdelnur created HADOOP-13558:
---

 Summary: UserGroupInformation created from a Subject incorrectly 
tries to renew the Keberos ticket
 Key: HADOOP-13558
 URL: https://issues.apache.org/jira/browse/HADOOP-13558
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.6.4, 2.7.2, 3.0.0-alpha2
Reporter: Alejandro Abdelnur


The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions and 
if they are met it invokes the {{reloginFromKeytab()}}. The 
{{reloginFromKeytab()}} method then fails with an {{IOException}} 
"loginUserFromKeyTab must be done first" because there is no keytab associated 
with the UGI.

The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab 
({{isKeytab}} UGI instance variable) associated with the UGI, if there is one 
it triggers a call to {{reloginFromKeytab()}}. The problem is that the 
{{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned 
{{IOException}}.


The root of the problem seems to be when creating a UGI via the 
{{UGI.loginUserFromSubject(Subject)}} method, this method uses the 
{{UserGroupInformation(Subject)}} constructor, and this constructor does the 
following to determine if there is a keytab or not.

{code}
  this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
{code}

If the {{Subject}} given had a keytab, then the UGI instance will have the 
{{isKeytab}} set to TRUE.

It sets the UGI instance as it would have a keytab because the Subject has a 
keytab. This has 2 problems:

First, it does not set the keytab file (and this, having the {{isKeytab}} set 
to TRUE and the {{keytabFile}) set to NULL is what triggers the {{IOException}} 
in the method {{reloginFromKeytab()}}.

Second (and even if the first problem is fixed, this still is a problem), it 
assumes that because the subject has a keytab it is up to UGI to to the relogin 
using the keytab. This is incorrect if the UGI was created using the 
{{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the 
Subject is not the UGI, but the caller, so the caller is responsible for 
renewing the Kerberos tickets and the UGI should not try to do so.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/

No changes




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_test_libhdfs_zerocopy_hdfs_static 

Failed junit tests :

   hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   hadoop.hdfs.TestRollingUpgrade 
   hadoop.hdfs.TestDecommissionWithStriped 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   CTEST:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
  [24K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [8.0K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/148/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org