Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Wilfred Spiegelenburg 
+1 (non binding)

- build from source on MacOSX 10.14.1, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- confirmed the checksum and signature
- deployed a single node cluster  (jdk 1.8u191 / centos 7.5)

Wilfred

> On 23 Nov 2018, at 23:06, Sunil G  wrote:
> 
> Hi folks,
> 
> 
> 
> Thanks to all contributors who helped in this release [1]. I have created
> 
> first release candidate (RC0) for Apache Hadoop 3.2.0.
> 
> 
> Artifacts for this RC are available here:
> 
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> 
> 
> 
> RC tag in git is release-3.2.0-RC0.
> 
> 
> 
> The maven artifacts are available via repository.apache.org at
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> 
> 
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> 
> 
> 
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
> 
> are the highlights of this release.
> 
> 1. Node Attributes Support in YARN
> 
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> 
> 3. Support service upgrade via YARN Service API and CLI
> 
> 4. HDFS Storage Policy Satisfier
> 
> 5. Support Windows Azure Storage - Blob file system in Hadoop
> 
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> 
> 7. Improvements in Router-based HDFS federation
> 
> 
> 
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> 
> I have done few testing with my pseudo cluster. My +1 to start.
> 
> 
> 
> Regards,
> 
> Sunil
> 
> 
> 
> [1]
> 
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> 
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC


Wilfred Spiegelenburg | Software Engineer
cloudera.com

[jira] [Created] (HDFS-14113) EC : Add Configuration to restrict UserDefined Policies

2018-11-28 Thread Ayush Saxena (JIRA) 
Ayush Saxena created HDFS-14113:
---

 Summary: EC : Add Configuration to restrict UserDefined Policies
 Key: HDFS-14113
 URL: https://issues.apache.org/jira/browse/HDFS-14113
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ayush Saxena
Assignee: Ayush Saxena


By default addition of erasure coding policies is enabled for users.We need to 
add configuration whether to allow addition of new User Defined policies or 
not.Which can be configured in for of a Boolean value at the server side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14112) Avoid recursive call to external authorizer for getContentSummary.

2018-11-28 Thread Jitendra Nath Pandey (JIRA) 
Jitendra Nath Pandey created HDFS-14112:
---

 Summary: Avoid recursive call to external authorizer for 
getContentSummary.
 Key: HDFS-14112
 URL: https://issues.apache.org/jira/browse/HDFS-14112
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Jitendra Nath Pandey
Assignee: Tsz Wo Nicholas Sze


HDFS-12130 optimizes permission check, and invokes permission checker 
recursively for each component of the tree, which works well for FSPermission 
checker.

But for certain external authorizers it may be more efficient to make one call 
with {{subaccess}}, because often they don't have to evaluate for each and 
every component of the path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Sunil G 
Hi Eric,

Thanks for helping in verifying the release.

Post YARN-7370, preemption configs are refreshable. I tried to test by
making some changes in capacity-scheduler.xml and invoking yarn rmadmin
-refreshQueues.
I can see the changes reflected as per logs after refresh. Could you please
help to give some more scenarios so that i can try to reproduce.
Meanwhile i ll try some other combinations as well and let you know.


reservationsContinueLooking = true
*preemptionDisabled = true*
defaultAppPriorityPerQueue = 0
priority = 0
maxLifetime = -1 seconds
defaultLifetime = -1 seconds
2018-11-29 06:25:53,792 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=sunilgovindan IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService
RESULT=SUCCESS
2018-11-29 06:25:55,900 INFO
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy:
Capacity Scheduler configuration changed, updated preemption properties to:
max_ignored_over_capacity = 0.1
natural_termination_factor = 0.2
max_wait_before_kill = 15000
monitoring_interval = 3000
*total_preemption_per_round = 0.4*
observe_only = false
lazy-preemption-enabled = false
*intra-queue-preemption.enabled = false*
*intra-queue-preemption.max-allowable-limit = 0.4*
intra-queue-preemption.minimum-threshold = 0.5
intra-queue-preemption.preemption-order-policy = USERLIMIT_FIRST
priority-utilization.underutilized-preemption.enabled = false
select_based_on_reserved_containers = false
additional_res_balance_based_on_reserved_containers = false
Preemption-to-balance-queue-enabled = false

*now i disabled preemption for default queue and made some changed in
intraqueue-preemption params.*

reservationsContinueLooking = true
*preemptionDisabled = false*
defaultAppPriorityPerQueue = 0
priority = 0
maxLifetime = -1 seconds
defaultLifetime = -1 seconds
2018-11-29 06:29:32,620 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=sunilgovindan IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService
RESULT=SUCCESS
2018-11-29 06:29:34,893 INFO
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy:
Capacity Scheduler configuration changed, updated preemption properties to:
max_ignored_over_capacity = 0.1
natural_termination_factor = 0.2
max_wait_before_kill = 15000
monitoring_interval = 3000
*total_preemption_per_round = 0.7*
observe_only = false
lazy-preemption-enabled = false
*intra-queue-preemption.enabled = true*
*intra-queue-preemption.max-allowable-limit = 0.5*
intra-queue-preemption.minimum-threshold = 0.5
intra-queue-preemption.preemption-order-policy = USERLIMIT_FIRST
priority-utilization.underutilized-preemption.enabled = false
select_based_on_reserved_containers = false
additional_res_balance_based_on_reserved_containers = false
Preemption-to-balance-queue-enabled = false

On Thu, Nov 29, 2018 at 4:19 AM Eric Payne 
wrote:

> Sunil,
>
> So, the basic symptoms are that if preemption is enabled on any queue, the
> preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all
> of the preemption-specific properties are set back to the default values.
>
> This was introduced in branch-3.1, so it is NOT new behavior for release
> 3.2.0. I am still tracking down the cause. I will open a JIRA once I do
> further investigation if there is not one already.
>
> This will be a problem for installations which use preemption and which
> use the refreshQueues feature.
>
> Thanks,
> -Eric
>
>
> On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne <
> eric.payne1...@yahoo.com> wrote:
>
>
>
>
>
> Sunil, thanks for all of the hard work on this release.
>
> I have discovered that queue refresh doesn't work in some cases. For
> example, when I change
> yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take
> effect unless I restart the RM.
>
> I am still investigating, but I thought I should bring this up asap.
>
> Thanks,
> -Eric
>
>
>
>
> On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G 
> wrote:
>
>
>
>
>
> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6.

Re: RPC connect error when using kerberos Auth

2018-11-28 Thread Wei-Chiu Chuang 
Not sure about integrity -- I've seen very few if at all installations with
integrity option enabled.
Regarding privacy -- have you made sure both client and server enabled SASL
privacy? Both sides must have consistent RPC settings for them to talk.

On Wed, Nov 28, 2018 at 12:58 AM ZongtianHou 
wrote:

> Hi,everyone
> I am using a hdfs client API to access a secured hdfs cluster. The
> kerberos have been set up successfully. When the configuration of
> hadoop.rpc.protection in core-site.xml is set to authentication, it works
> well. However, when it is set to integration or privacy, the namenode can
> not be connected, and the log of namenode give the following error. Does
> anyone know what the info mean and what lib is needed for connection in
> integration and privacy mode? Any hint will be very appreciated!!
>
> 2018-11-27 17:14:05,270 WARN SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth failed for 127.0.0.1:50769:null (Problem with callback handler)
> 2018-11-27 17:14:05,270 INFO org.apache.hadoop.ipc.Server: Socket Reader
> #1 for port 8020: readAndProcess from client 127.0.0.1 threw exception
> [javax.security.sasl.SaslException: Problem with callback handler [Caused
> by javax.security.sasl.SaslException: Client selected unsupported
> protection: 1]]
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDDS-881) Add support for transform client requests to OM into RaftClient requests

2018-11-28 Thread Hanisha Koneru (JIRA) 
Hanisha Koneru created HDDS-881:
---

 Summary: Add support for transform client requests to OM into 
RaftClient requests
 Key: HDDS-881
 URL: https://issues.apache.org/jira/browse/HDDS-881
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Hanisha Koneru
Assignee: Hanisha Koneru


When OM receives a request, we need to transform the request into Ratis server 
compatible request so that the OM's Ratis server can process that request. 
In this Jira, we just add the support to convert a client request received by 
OM into a RaftClient request. This transformed request would later be passed 
onto the OM's Ratis server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Chandni Singh 
+1 (non-binding)

- Build from source
- Launched example sleeper service
- Tested upgrade of sleeper service
- Tested cancellation of upgrade
- Tested express upgrade

On Wed, Nov 28, 2018 at 8:22 AM Peter Bacsko 
wrote:

> +1 (non-binding)
>
> - Built from source at tag 3.2.0-rc0 (Ubuntu 18.10, JDK1.8.0_191)
> - Verified checksums of hadoop-3.2.0.tar.gz
> - Installed on a 3-node physical cluster
> - Ran teragen/terasort/teravalidate
> - Ran distributed shell a couple of times
> - Checked UIs (RM, NM, DN, JHS)
>
> Peter
>
> On Wed, Nov 28, 2018 at 5:17 PM Jason Lowe  wrote:
>
> > Thanks for driving this release, Sunil!
> >
> > +1 (binding)
> >
> > - Verified signatures and digests
> > - Successfully performed a native build
> > - Deployed a single-node cluster
> > - Ran some sample jobs
> >
> > Jason
> >
> > On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:
> >
> > > Hi folks,
> > >
> > >
> > >
> > > Thanks to all contributors who helped in this release [1]. I have
> created
> > >
> > > first release candidate (RC0) for Apache Hadoop 3.2.0.
> > >
> > >
> > > Artifacts for this RC are available here:
> > >
> > > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> > >
> > >
> > >
> > > RC tag in git is release-3.2.0-RC0.
> > >
> > >
> > >
> > > The maven artifacts are available via repository.apache.org at
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> > >
> > >
> > > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm
> PST.
> > >
> > >
> > >
> > > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> > > additions
> > >
> > > are the highlights of this release.
> > >
> > > 1. Node Attributes Support in YARN
> > >
> > > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> > >
> > > 3. Support service upgrade via YARN Service API and CLI
> > >
> > > 4. HDFS Storage Policy Satisfier
> > >
> > > 5. Support Windows Azure Storage - Blob file system in Hadoop
> > >
> > > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> > >
> > > 7. Improvements in Router-based HDFS federation
> > >
> > >
> > >
> > > Thanks to Wangda, Vinod, Marton for helping me in preparing the
> release.
> > >
> > > I have done few testing with my pseudo cluster. My +1 to start.
> > >
> > >
> > >
> > > Regards,
> > >
> > > Sunil
> > >
> > >
> > >
> > > [1]
> > >
> > >
> > >
> >
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.2.0)
> > > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> > > ORDER BY fixVersion ASC
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Eric Payne 
Sunil,

So, the basic symptoms are that if preemption is enabled on any queue, the 
preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of 
the preemption-specific properties are set back to the default values.

This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. 
I am still tracking down the cause. I will open a JIRA once I do further 
investigation if there is not one already.

This will be a problem for installations which use preemption and which use the 
refreshQueues feature.

Thanks,
-Eric


On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne 
 wrote: 





Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-880) Create api for ACL handling in Ozone

2018-11-28 Thread Ajay Kumar (JIRA) 
Ajay Kumar created HDDS-880:
---

 Summary: Create api for ACL handling in Ozone
 Key: HDDS-880
 URL: https://issues.apache.org/jira/browse/HDDS-880
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: documentation
Reporter: Ajay Kumar
Assignee: Anu Engineer


The valid uri pattern for an Ozone fs uri should be 
{{o3fs://://}}.

But OzoneFileSystem accepts uri's of the form {{o3fs://.}} only.
{code:java}
# In OzoneFileSyste.java
private static final Pattern URL_SCHEMA_PATTERN =
Pattern.compile("(.+)\\.([^\\.]+)");
if (!matcher.matches()) {
  throw new IllegalArgumentException("Ozone file system url should be "
  + "in the form o3fs://bucket.volume");
}{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-879) Add InitiateMultipartUpload in ozone

2018-11-28 Thread Bharat Viswanadham (JIRA) 
Bharat Viswanadham created HDDS-879:
---

 Summary: Add InitiateMultipartUpload in ozone
 Key: HDDS-879
 URL: https://issues.apache.org/jira/browse/HDDS-879
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


This Jira is to add initiate multipart upload.

InitiateMultipart upload does 2 things:
 # Create an entry in the open table for this key
 # Add multipartInfo information for this key.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

2018-11-28 Thread Todd Lipcon (JIRA) 
Todd Lipcon created HDFS-14111:
--

 Summary: hdfsOpenFile on HDFS causes unnecessary IO from file 
offset 0
 Key: HDFS-14111
 URL: https://issues.apache.org/jira/browse/HDFS-14111
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client, libhdfs
Affects Versions: 3.2.0
Reporter: Todd Lipcon


hdfsOpenFile() calls readDirect() with a 0-length argument in order to check 
whether the underlying stream supports bytebuffer reads. With DFSInputStream, 
the read(0) isn't short circuited, and results in the DFSClient opening a block 
reader. In the case of a remote block, the block reader will actually issue a 
read of the whole block, causing the datanode to perform unnecessary IO and 
network transfers in order to fill up the client's TCP buffers. This causes 
performance degradation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10369) hdfsread crash when reading data reaches to 128M

2018-11-28 Thread Todd Lipcon (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDFS-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-10369.

Resolution: Invalid

You're mallocing a buffer of 5 bytes here, seems your C code is just broken.

> hdfsread crash when reading data reaches to 128M
> 
>
> Key: HDFS-10369
> URL: https://issues.apache.org/jira/browse/HDFS-10369
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Reporter: vince zhang
>Priority: Major
>
> see code below, it would crash after   printf("hdfsGetDefaultBlockSize2:%d, 
> ret:%d\n", hdfsGetDefaultBlockSize(fs), ret);
>   
> hdfsFile read_file = hdfsOpenFile(fs, "/testpath", O_RDONLY, 0, 0, 1); 
>   int total = hdfsAvailable(fs, read_file);
>   printf("Total:%d\n", total);
>   char* buffer = (char*)malloc(sizeof(size+1) * sizeof(char));
>   int ret = -1; 
>   int len = 0;
>   ret = hdfsSeek(fs, read_file, 134152192);
>   printf("hdfsGetDefaultBlockSize1:%d, ret:%d\n", 
> hdfsGetDefaultBlockSize(fs), ret);
>   ret = hdfsRead(fs, read_file, (void*)buffer, size);
>   printf("hdfsGetDefaultBlockSize2:%d, ret:%d\n", 
> hdfsGetDefaultBlockSize(fs), ret);
>   ret = hdfsRead(fs, read_file, (void*)buffer, size);
>   printf("hdfsGetDefaultBlockSize3:%d, ret:%d\n", 
> hdfsGetDefaultBlockSize(fs), ret);
>   return 0;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14110) NPE when serving http requests while NameNode is starting up

2018-11-28 Thread Chao Sun (JIRA) 
Chao Sun created HDFS-14110:
---

 Summary: NPE when serving http requests while NameNode is starting 
up
 Key: HDFS-14110
 URL: https://issues.apache.org/jira/browse/HDFS-14110
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.8.2
Reporter: Chao Sun
Assignee: Chao Sun


In 2.8.2 we saw this exception when a security-enabled NameNode is still 
loading edits:
{code:java}
2018-11-28 00:21:02,909 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected 
pause in JVM or host machine (eg GC): pause of approximately 2068ms GC pool 
'ParNew' had collection(s): count=1 time=2325ms 2018-11-28 00:21:05,768 WARN 
org.apache.hadoop.hdfs.web.resources.ExceptionHandler: INTERNAL_SERVER_ERROR 
java.lang.NullPointerException at 
org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:283) 
at org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:226) at 
org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:54)
 at 
org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:42)
 at 
com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
 at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
 at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
 at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
 at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
 at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
 at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
 at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
 at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
 at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:87) at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
 at 
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)
 at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
 at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
 at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at 
org.mortbay.jetty.Server.handle(Server.java:326) at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at 
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at 
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{code}
Looking at the code, this is where the NPE happened (the line with

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Eric Payne 
Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-813) [JDK11] mvn javadoc:javadoc -Phdds fails

2018-11-28 Thread Dinesh Chitlangia (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HDDS-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dinesh Chitlangia resolved HDDS-813.

Resolution: Implemented

> [JDK11] mvn javadoc:javadoc -Phdds fails
> 
>
> Key: HDDS-813
> URL: https://issues.apache.org/jira/browse/HDDS-813
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Dinesh Chitlangia
>Priority: Major
>  Labels: javadoc
> Attachments: HDDS-813.001.patch
>
>
> {{mvn javadoc:javadoc -Phdds}} fails on Java 11
> {noformat}
> [ERROR] 
> /Users/aajisaka/git/hadoop/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java:107:
>  error: bad use of '>'
> [ERROR]* @param count count must be > 0.
> [ERROR] 
> /Users/aajisaka/git/hadoop/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/LocatedContainer.java:85:
>  error: unknown tag: DatanodeInfo
> [ERROR]   * @return Set nodes that currently host the container
> [ERROR] 
> /Users/aajisaka/git/hadoop/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmLocatedBlock.java:71:
>  error: unknown tag: DatanodeInfo
> [ERROR]   * @return List nodes that currently host the block
> [ERROR] 
> /Users/aajisaka/git/hadoop/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/Auditable.java:28:
>  error: malformed HTML
> [ERROR]   * @return Map with values to be logged in audit.
> [ERROR]                 ^
> [ERROR] 
> /Users/aajisaka/git/hadoop/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/Auditable.java:28:
>  error: bad use of '>'
> [ERROR]   * @return Map with values to be logged in audit.
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Peter Bacsko 
+1 (non-binding)

- Built from source at tag 3.2.0-rc0 (Ubuntu 18.10, JDK1.8.0_191)
- Verified checksums of hadoop-3.2.0.tar.gz
- Installed on a 3-node physical cluster
- Ran teragen/terasort/teravalidate
- Ran distributed shell a couple of times
- Checked UIs (RM, NM, DN, JHS)

Peter

On Wed, Nov 28, 2018 at 5:17 PM Jason Lowe  wrote:

> Thanks for driving this release, Sunil!
>
> +1 (binding)
>
> - Verified signatures and digests
> - Successfully performed a native build
> - Deployed a single-node cluster
> - Ran some sample jobs
>
> Jason
>
> On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:
>
> > Hi folks,
> >
> >
> >
> > Thanks to all contributors who helped in this release [1]. I have created
> >
> > first release candidate (RC0) for Apache Hadoop 3.2.0.
> >
> >
> > Artifacts for this RC are available here:
> >
> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> >
> >
> >
> > RC tag in git is release-3.2.0-RC0.
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> >
> >
> >
> > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> > additions
> >
> > are the highlights of this release.
> >
> > 1. Node Attributes Support in YARN
> >
> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> >
> > 3. Support service upgrade via YARN Service API and CLI
> >
> > 4. HDFS Storage Policy Satisfier
> >
> > 5. Support Windows Azure Storage - Blob file system in Hadoop
> >
> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> >
> > 7. Improvements in Router-based HDFS federation
> >
> >
> >
> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> >
> > I have done few testing with my pseudo cluster. My +1 to start.
> >
> >
> >
> > Regards,
> >
> > Sunil
> >
> >
> >
> > [1]
> >
> >
> >
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> > ORDER BY fixVersion ASC
> >
>

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

2018-11-28 Thread Jason Lowe 
Thanks for driving this release, Sunil!

+1 (binding)

- Verified signatures and digests
- Successfully performed a native build
- Deployed a single-node cluster
- Ran some sample jobs

Jason

On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:

> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>

[jira] [Created] (HDFS-14109) Improve hdfs auditlog format and support federation friendly

2018-11-28 Thread He Xiaoqiao (JIRA) 
He Xiaoqiao created HDFS-14109:
--

 Summary: Improve hdfs auditlog format and support federation 
friendly
 Key: HDFS-14109
 URL: https://issues.apache.org/jira/browse/HDFS-14109
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: He Xiaoqiao
Assignee: He Xiaoqiao


The following auditlog format does not well meet requirement for federation 
arch currently. Since some case we need to aggregate all namespace audit log, 
if there are some common path request(e.g. /tmp, /user/ etc. some path may not 
appear in mountTable, but the path is very real), we will have no idea to split 
them that which namespace it request to. So I propose add column {{nsid}} to 
support federation more friendly.  
{quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true   
ugi=hdfs/hostn...@realm.com (auth:KERBEROS)  ip=/10.1.1.2 cmd=getfileinfo 
src=/path   dst=null        perm=null       proto=rpc       clientName=null
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-878) Do the disk failure check before ContainerSet initialization

2018-11-28 Thread Yiqun Lin (JIRA) 
Yiqun Lin created HDDS-878:
--

 Summary: Do the disk failure check before ContainerSet 
initialization
 Key: HDDS-878
 URL: https://issues.apache.org/jira/browse/HDDS-878
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Yiqun Lin
Assignee: Yiqun Lin


OzoneContainer won't handle any disk failures before initializing the 
ContainerSet. In a very extreme case, all the disk are bad and the 
OzoneContainer can still be running. We can improve this and add disk failure 
tolerance, like DataNode already did.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org