date:20180228

[jira] [Created] (HDFS-13212) RBF: Fix router location cache issue

2018-02-28 Thread weiwei wu (JIRA)

weiwei wu created HDFS-13212:


 Summary: RBF: Fix router location cache issue
 Key: HDFS-13212
 URL: https://issues.apache.org/jira/browse/HDFS-13212
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: federation, hdfs
Reporter: weiwei wu
 Attachments: HDFS-12615-001.patch

The MountTableResolver refreshEntries function have a bug when add a new mount 
table entry which already have location cache. The old location cache will 
never be invalid until this mount point change again.

Need to invalid the location cache when add the mount table entries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13211) Refactor Unit Tests for SnapshotSKipList

2018-02-28 Thread Shashikant Banerjee (JIRA)

Shashikant Banerjee created HDFS-13211:
--

 Summary: Refactor Unit Tests for SnapshotSKipList
 Key: HDFS-13211
 URL: https://issues.apache.org/jira/browse/HDFS-13211
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: snapshots
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee


HDFS-13102 implements the DiffList interface for storing Directory Diffs using 
SkipList.

This Jira proposes to refactor the unit tests for HDFS-13102.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13210) Fix the typo in MiniDFSCluster class

2018-02-28 Thread Yiqun Lin (JIRA)

Yiqun Lin created HDFS-13210:


 Summary: Fix the typo in MiniDFSCluster class 
 Key: HDFS-13210
 URL: https://issues.apache.org/jira/browse/HDFS-13210
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0
Reporter: Yiqun Lin


There is a typo {{SimilatedFSDataset}} in {{MiniDFSCluster#injectBlocks}}.
In line2748 and line2749:
{code}
public void injectBlocks(int dataNodeIndex,
  Iterable blocksToInject, String bpid) throws IOException {
if (dataNodeIndex < 0 || dataNodeIndex > dataNodes.size()) {
  throw new IndexOutOfBoundsException();
}
final DataNode dn = dataNodes.get(dataNodeIndex).datanode;
final FsDatasetSpi dataSet = DataNodeTestUtils.getFSDataset(dn);
if (!(dataSet instanceof SimulatedFSDataset)) {
  throw new IOException("injectBlocks is valid only for 
SimilatedFSDataset");
}
...
}
{code}
{{SimilatedFSDataset}} should be {{SimulatedFSDataset}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2018-02-28 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/150/

[Feb 28, 2018 2:25:49 AM] (yqlin) HDFS-13194. CachePool permissions incorrectly 
checked. Contributed by
[Feb 28, 2018 9:19:03 PM] (stevel) Revert "HADOOP-15090. Add ADL 
troubleshooting doc."




-1 overall


The following subsystems voted -1:
docker


Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-7240 to trunk

2018-02-28 Thread sanjay Radia

Andrew, thanks for your response.

1) Wrt to NN on top of HDSL. You raised the issue of FSN lock separation . This 
was a key issue we discussed heavily in the past in the context of “Show the 
community a way to connect NN into the the new block layer”. We heard you 
clearly and thought deeply and showed how NN can be put on top of  WITHOUT 
removing the FSN.  We described this in detail  in HDFS-10419 and also  in the 
summary of the DISCUSSION thread:
  Milestone 1 (no removal of FSN) gives almost 2x scalability and does not 
require separation of FSN lock and that milestone 2 which removes the FSN lock 
gives 2x scalability. 

You have conveniently ignored this. Let me reemphasize: Removing the FSN lock 
is not necessary for NN/HDFS to benefit from HDSL and you get almost the same 
scalability benefit. Hence the FSN local issue is moot. 

2) You have also conveniently ignored our arguments that there is benefit in 
keeping HDSL and HDFS together that are in the vote and discussion thread 
summary:
  A) Side by side usage and resulting operational concerns
>>"In the short term and medium term, the new system and HDFS
>> will be used side-by-side by users. ……  
>> During this time, sharing the DN daemon and admin functions
>> between the two systems is operationally important”

   B) Sharing code 
>>"Need to easily share the block layer code between the two systems
>> when used side-by-side. Areas where sharing code is desired over time: 
>>  - Sharing new block layer’s  new netty based protocol engine
>> for old HDFS DNs (a long time sore issue for HDFS block layer). 
>> - Shallow data copy from old system to new system is practical
>> only if within same project and daemon otherwise have to deal
>> with security setting and coordinations across daemons.
>> Shallow copy is useful as customer migrate from old to new.
>> - Shared disk scheduling in the future"

3) You argue for separate project from 2 conflicting arguments: (1) Separate 
then merge later, what’s the hurry.  (2) keep seperate and focus on non-HDFS 
storage use cases. The HDFS community members built HDSL to address HDFS 
scalability; they were  not trying go after object store users or market (ceph 
etc). As explained multiple times OzoneFS is an intermediate step to stabilize 
HDSL but of immediate value for apps such as Hive and Spark. So even if there 
might be value in being separate (your motivation 2)  and go after a new 
storage use cases, the HDFS community members that built HDSL want to focus on 
improving HDFS; you may not agree with that but the engineers that are writing 
the code should be able to drive the direction.  Further look at the Security 
design we posted  - shows a Hadoop/HDFS focus not a focus for some other object 
store market: it fits into the Hadoop security model, especially supporting the 
use case of Jobs and the resulting need to support delegation tokens. 

4) You argue that the  HDSL and OzoneFS modules are separate and therefore one 
should go as a separate project. * Looks like one can’t win here. Damned if you 
do and Damned if you don’t. In the discussion with the Cloudera team one of the 
issues raised was that there a lot of new code and it will destabilized HDFS. 
We explained that  we have kept the code in separate modules so that it will 
not impact current HDFS stability, and that features like HDSL’s  new protocol 
engine will be plugged into the old HDFS block layer only after stabilization. 
You argue for stability and hence separate modules and then use it against to 
push it out as a separate project.

sanjay

> On Feb 28, 2018, at 12:10 AM, Andrew Wang  wrote:
> 
> Resending since the formatting was messed up, let's try plain text this
> time:
> 
> Hi Jitendra and all,
> 
> Thanks for putting this together. I caught up on the discussion on JIRA and
> document at HDFS-10419, and still have the same concerns raised earlier
> about merging the Ozone branch to trunk.
> 
> To recap these questions/concerns at a very high level:
> 
> * Wouldn't Ozone benefit from being a separate project?
> * Why should it be merged now?
> 
> I still believe that both Ozone and Hadoop would benefit from Ozone being a
> separate project, and that there is no pressing reason to merge Ozone/HDSL
> now.
> 
> The primary reason I've heard for merging is that the Ozone is that it's at
> a stage where it's ready for user feedback. Second, that it needs to be
> merged to start on the NN refactoring for HDFS-on-HDSL.
> 
> First, without HDFS-on-HDSL support, users are testing against the Ozone
> object storage interface. Ozone and HDSL themselves are implemented as
> separate masters and new functionality bolted onto the datanode. It also
> doesn't look like HDFS in terms of API or featureset; yes, it speaks
> FileSystem, but so do many out-of-tree storage systems like S3, Ceph,
> Swift, ADLS etc. Ozone/HDSL does not support popular HDFS features like
> erasure coding, encryption,

[jira] [Created] (HDFS-13209) DistributedFileSystem.create should allow an option to provide StoragePolicy

2018-02-28 Thread Jean-Marc Spaggiari (JIRA)

Jean-Marc Spaggiari created HDFS-13209:
--

 Summary: DistributedFileSystem.create should allow an option to 
provide StoragePolicy
 Key: HDFS-13209
 URL: https://issues.apache.org/jira/browse/HDFS-13209
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: hdfs
Affects Versions: 3.0.0
Reporter: Jean-Marc Spaggiari


DistributedFileSystem.create allows to get a FSDataOutputStream. The stored 
file and related blocks will used the directory based StoragePolicy.

 

However, sometime, we might need to keep all files in the same directory 
(consistency constraint) but might want some of them on SSD (small, in my case) 
until they are processed and merger/removed. Then they will go on the default 
policy.

 

When creating a file, it will be useful to have an option to specify a 
different StoragePolicy...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13208) RBF: Mount path not available after ADD-REMOVE-ADD

2018-02-28 Thread Wei Yan (JIRA)

Wei Yan created HDFS-13208:
--

 Summary: RBF: Mount path not available after ADD-REMOVE-ADD
 Key: HDFS-13208
 URL: https://issues.apache.org/jira/browse/HDFS-13208
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Wei Yan


To reproduce this issue, run the following commands at Router 1:
{code:java}
$ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1
$ hdfs dfsrouteradmin -rm /test1
$ hdfs dfsrouteradmin -add /test1{code}
"hdfs dfs -ls hdfs://Router1:8020/test1" works well after step 1. But after 
step 3 when we add /test1 back, Router 1 still returns "No such file or 
directory". 

But after step 3, when we run cmd "hdfs dfs -ls hdfs://Router2:8020/test1" 
talking to another Router, it works well.

>From Router logs, I can see StateStoreZookeeperImpl and MountTableResolver are 
>updated correctly and in time. Not find the root case yet, still looking into 
>it.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

2018-02-28 Thread Chris Douglas

+1

Let's get this done. We've had many false starts on a native HDFS
client. This is a good base to build on. -C

On Wed, Feb 28, 2018 at 9:55 AM, Jim Clampffer
 wrote:
> Hi everyone,
>
> I'd like to start a thread to discuss merging the HDFS-8707 aka libhdfs++
> into trunk.  I sent originally sent a similar email out last October but it
> sounds like it was buried by discussions about other feature merges that
> were going on at the time.
>
> libhdfs++ is an HDFS client written in C++ designed to be used in
> applications that are written in non-JVM based languages.  In its current
> state it supports kerberos authenticated reads from HDFS and has been used
> in production clusters for over a year so it has had a significant amount
> of burn-in time.  The HDFS-8707 branch has been around for about 2 years
> now so I'd like to know people's thoughts on what it would take to merge
> current branch and handling writes and encrypted reads in a new one.
>
> Current notable features:
>   -A libhdfs/libhdfs3 compatible C API that allows libhdfs++ to serve as a
> drop-in replacement for clients that only need read support (until
> libhdfs++ also supports writes).
>   -An asynchronous C++ API with synchronous shims on top if the client
> application wants to do blocking operations.  Internally a single thread
> (optionally more) uses select/epoll by way of boost::asio to watch
> thousands of sockets without the overhead of spawning threads to emulate
> async operation.
>   -Kerberos/SASL authentication support
>   -HA namenode support
>   -A set of utility programs that mirror the HDFS CLI utilities e.g.
> "./hdfs dfs -chmod".  The major benefit of these is the tool startup time
> is ~3 orders of magnitude faster (<1ms vs hundreds of ms) and occupies a
> lot less memory since it isn't dealing with the JVM.  This makes it
> possible to do things like write a simple bash script that stats a file,
> applies some rules to the result, and decides if it should move it in a way
> that scales to thousands of files without being penalized with O(N) JVM
> startups.
>   -Cancelable reads.  This has proven to be very useful in multiuser
> applications that (pre)fetch large blocks of data but need to remain
> responsive for interactive users.  Rather than waiting for a large and/or
> slow read to finish it will return immediately and the associated resources
> (buffer, file descriptor) become available for the rest of the application
> to use.
>
> There are a couple known issues: the doc build isn't integrated with the
> rest of hadoop and the public API headers aren't being exported when
> building a distribution.  A short term solution for missing docs is to go
> through the libhdfs(3) compatible API and use the libhdfs docs.  Other than
> a few modifications to the pom files to integrate the build and the changes
> are isolated to a new directory so the chance of causing any regressions in
> the rest of the code is minimal.
>
> Please share your thoughts, thanks!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13207) Add blockpool used for DFSAdmin Report command

2018-02-28 Thread Bharat Viswanadham (JIRA)

Bharat Viswanadham created HDFS-13207:
-

 Summary: Add blockpool used for DFSAdmin Report command 
 Key: HDFS-13207
 URL: https://issues.apache.org/jira/browse/HDFS-13207
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


Add block pool used for dfsadmin -report command. 

This will be helpful on a federated cluster to know DFS used by a particular 
namespace.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-02-28 Thread Ted Yu (JIRA)

Ted Yu created HDFS-13206:
-

 Summary: IllegalStateException: Unable to finalize edits file
 Key: HDFS-13206
 URL: https://issues.apache.org/jira/browse/HDFS-13206
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ted Yu


I noticed the following in hbase test output running against hadoop3:
{code}
2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): 
Error: finalize log segment 1, 658 failed for (journal 
JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
 stream=null))
java.lang.IllegalStateException: Unable to finalize edits file 
/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
  at 
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
  at 
org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
  at 
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
  at 
org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
  at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
  at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
  at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
  at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
  at 
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
  at 
org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
  at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
  at 
org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
  at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
  at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
  at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
  at 
org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

2018-02-28 Thread Jim Clampffer

Hi everyone,

I'd like to start a thread to discuss merging the HDFS-8707 aka libhdfs++
into trunk.  I sent originally sent a similar email out last October but it
sounds like it was buried by discussions about other feature merges that
were going on at the time.

libhdfs++ is an HDFS client written in C++ designed to be used in
applications that are written in non-JVM based languages.  In its current
state it supports kerberos authenticated reads from HDFS and has been used
in production clusters for over a year so it has had a significant amount
of burn-in time.  The HDFS-8707 branch has been around for about 2 years
now so I'd like to know people's thoughts on what it would take to merge
current branch and handling writes and encrypted reads in a new one.

Current notable features:
  -A libhdfs/libhdfs3 compatible C API that allows libhdfs++ to serve as a
drop-in replacement for clients that only need read support (until
libhdfs++ also supports writes).
  -An asynchronous C++ API with synchronous shims on top if the client
application wants to do blocking operations.  Internally a single thread
(optionally more) uses select/epoll by way of boost::asio to watch
thousands of sockets without the overhead of spawning threads to emulate
async operation.
  -Kerberos/SASL authentication support
  -HA namenode support
  -A set of utility programs that mirror the HDFS CLI utilities e.g.
"./hdfs dfs -chmod".  The major benefit of these is the tool startup time
is ~3 orders of magnitude faster (<1ms vs hundreds of ms) and occupies a
lot less memory since it isn't dealing with the JVM.  This makes it
possible to do things like write a simple bash script that stats a file,
applies some rules to the result, and decides if it should move it in a way
that scales to thousands of files without being penalized with O(N) JVM
startups.
  -Cancelable reads.  This has proven to be very useful in multiuser
applications that (pre)fetch large blocks of data but need to remain
responsive for interactive users.  Rather than waiting for a large and/or
slow read to finish it will return immediately and the associated resources
(buffer, file descriptor) become available for the rest of the application
to use.

There are a couple known issues: the doc build isn't integrated with the
rest of hadoop and the public API headers aren't being exported when
building a distribution.  A short term solution for missing docs is to go
through the libhdfs(3) compatible API and use the libhdfs docs.  Other than
a few modifications to the pom files to integrate the build and the changes
are isolated to a new directory so the chance of causing any regressions in
the rest of the code is minimal.

Please share your thoughts, thanks!

[jira] [Created] (HDFS-13205) Incorrect path is passed to checkPermission during authorization of file under a snapshot (specifically under a subdir) after original subdir is deleted

2018-02-28 Thread Raghavender Rao Guruvannagari (JIRA)

Raghavender Rao Guruvannagari created HDFS-13205:


 Summary: Incorrect path is passed to checkPermission during 
authorization of file under a snapshot (specifically under a subdir) after 
original subdir is deleted
 Key: HDFS-13205
 URL: https://issues.apache.org/jira/browse/HDFS-13205
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 2.7.4
Reporter: Raghavender Rao Guruvannagari


Steps to reproduce the issue.

+As 'hdfs' superuser+ 
-- Create a folder (/hdptest/test) with 700 permissions and ( 
/hdptest/test/mydir) with 755.

--HDFS Ranger policy is defined  with RWX for user "test" on /hdptest/test/ 
recursively.

 --Allow snapshot on the directory  /hdptest/test/mydir:

 
{code:java}
#su - test
[test@node1 ~]$ hdfs dfs -ls /hdptest/test/mydir
[test@node1 ~]$ hdfs dfs -mkdir /hdptest/test/mydir/test
[test@node1 ~]$ hdfs dfs -put /etc/passwd /hdptest/test/mydir/test
[test@node1 ~]$ hdfs lsSnapshottableDir
drwxr-xr-x 0 test hdfs 0 2018-01-25 14:22 1 65536 /hdptest/test/mydir
 
{code}
 

-->Create Snapshot :

 

 
{code:java}
[test@node1 ~]$ hdfs dfs -createSnapshot /hdptest/test/mydir
Created snapshot /hdptest/test/mydir/.snapshot/s20180125-135430.953
{code}
 

 

-->Verifying that snapshot directory has the current files from directory and 
verify the file is accessible  .snapshot path:

 

 
{code:java}
[test@node1 ~]$ hdfs dfs -ls -R 
/hdptest/test/mydir/.snapshot/s20180125-135430.953
drwxr-xr-x   - test hdfs  0 2018-01-25 13:53 
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test
-rw-r--r--   3 test hdfs   3227 2018-01-25 13:53 
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd
[test@node1 ~]$ hdfs dfs -cat 
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
ehdpzepp:x:1016:496::/home/ehdpzepp:/bin/bash
zepptest:x:1017:496::/home/zepptest:/bin/bash
{code}
 

 

-->Remove the file from main directory and verified that file is still 
accessible:

 

 
{code:java}
[test@node1 ~]$ hdfs dfs -rm /hdptest/test/mydir/test/passwd
18/01/25 13:55:06 INFO fs.TrashPolicyDefault: Moved: 
'hdfs://rangerSME/hdptest/test/mydir/test/passwd' to trash at: 
hdfs://rangerSME/user/test/.Trash/Current/hdptest/test/mydir/test/passwd
[test@node1 ~]$ hdfs dfs -cat 
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
{code}
 

 

-->Remove the parent directory of the file which was deleted, now accessing the 
same file under .snapshot dir fails with permission denied error

 

 
{code:java}
[test@node1 ~]$ hdfs dfs -rm -r /hdptest/test/mydir/test
18/01/25 13:55:25 INFO fs.TrashPolicyDefault: Moved: 
'hdfs://rangerSME/hdptest/test/mydir/test' to trash at: 
hdfs://rangerSME/user/test/.Trash/Current/hdptest/test/mydir/test1516888525269
[test@node1 ~]$ hdfs dfs -cat 
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
cat: Permission denied: user=test, access=EXECUTE, 
inode="/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd":hdfs:hdfs:drwxr-x---
 
{code}
 

Ranger policies are not honored in this case for .snapshot directories/files 
after main directory is deleted under snapshotable directory.

 

Workaround is to provide execute permission at HDFS level for the parent folder

 
{code:java}
#su - hdfs
#hdfs dfs -chmod 701 /hdptest/test
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-02-28 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/

[Feb 27, 2018 2:48:52 AM] (yqlin) HDFS-13184. RBF: Improve the unit test 
TestRouterRPCClientRetries.
[Feb 27, 2018 3:38:29 PM] (arp) HADOOP-15178. Generalize NetUtils#wrapException 
to handle other
[Feb 27, 2018 4:48:03 PM] (inigoiri) HDFS-13193. Various Improvements for 
BlockTokenSecretManager.
[Feb 27, 2018 4:53:00 PM] (inigoiri) HDFS-13192. Change the code order in 
getFileEncryptionInfo to avoid
[Feb 27, 2018 6:15:43 PM] (arp) HADOOP-14959. 
DelegationTokenAuthenticator.authenticate() to wrap
[Feb 27, 2018 6:18:07 PM] (arp) HDFS-13181. DiskBalancer: Add an configuration 
for valid plan hours .
[Feb 27, 2018 6:27:18 PM] (arp) MAPREDUCE-7061. SingleCluster setup document 
needs to be updated.
[Feb 27, 2018 9:19:16 PM] (wangda) YARN-7893. Document the FPGA isolation 
feature. (Zhankun Tang via
[Feb 27, 2018 9:19:24 PM] (wangda) YARN-7959. Add .vm extension to 
PlacementConstraints.md to ensure proper
[Feb 27, 2018 10:33:57 PM] (billie) YARN-7446. Remove --user flag when running 
privileged mode docker
[Feb 27, 2018 11:28:41 PM] (szetszwo) HDFS-13143. SnapshotDiff - 
snapshotDiffReport might be inconsistent if
[Feb 28, 2018 1:39:02 AM] (inigoiri) HDFS-13199. RBF: Fix the hdfs router page 
missing label icon issue.




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 
   hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.fs.http.server.TestHttpFSServerWebServer 
   hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
   hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/whitespace-tabs.txt
  [288K]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [328K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [20K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/706/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache

[jira] [Created] (HDFS-13204) RBF: optimize name service safe mode icon

2018-02-28 Thread liuhongtong (JIRA)

liuhongtong created HDFS-13204:
--

 Summary: RBF: optimize name service safe mode icon
 Key: HDFS-13204
 URL: https://issues.apache.org/jira/browse/HDFS-13204
 Project: Hadoop HDFS
  Issue Type: Wish
Reporter: liuhongtong
 Attachments: image-2018-02-28-17-46-35-127.png, 
image-2018-02-28-17-47-44-156.png, image-2018-02-28-17-51-33-913.png

In federationhealth webpage，the safe mode icon in the picture below may induce 
users the name service is maintaining.

!image-2018-02-28-17-51-33-913.png!

In fact, if the name service is in safe mode, users can't do most of  
operations. So I think the icon below may be more reasonable.

!image-2018-02-28-17-46-35-127.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-7240 to trunk

2018-02-28 Thread Andrew Wang

Resending since the formatting was messed up, let's try plain text this
time:

Hi Jitendra and all,

Thanks for putting this together. I caught up on the discussion on JIRA and
document at HDFS-10419, and still have the same concerns raised earlier
about merging the Ozone branch to trunk.

To recap these questions/concerns at a very high level:

* Wouldn't Ozone benefit from being a separate project?
* Why should it be merged now?

I still believe that both Ozone and Hadoop would benefit from Ozone being a
separate project, and that there is no pressing reason to merge Ozone/HDSL
now.

The primary reason I've heard for merging is that the Ozone is that it's at
a stage where it's ready for user feedback. Second, that it needs to be
merged to start on the NN refactoring for HDFS-on-HDSL.

First, without HDFS-on-HDSL support, users are testing against the Ozone
object storage interface. Ozone and HDSL themselves are implemented as
separate masters and new functionality bolted onto the datanode. It also
doesn't look like HDFS in terms of API or featureset; yes, it speaks
FileSystem, but so do many out-of-tree storage systems like S3, Ceph,
Swift, ADLS etc. Ozone/HDSL does not support popular HDFS features like
erasure coding, encryption, high-availability, snapshots, hflush/hsync (and
thus HBase), or APIs like WebHDFS or NFS. This means that Ozone feels like
a new, different system that could reasonably be deployed and tested
separately from HDFS. It's unlikely to replace many of today's HDFS
deployments, and from what I understand, Ozone was not designed to do this.

Second, the NameNode refactoring for HDFS-on-HDSL by itself is a major
undertaking. The discussion on HDFS-10419 is still ongoing so it’s not
clear what the ultimate refactoring will be, but I do know that the earlier
FSN/BM refactoring during 2.x was very painful (introducing new bugs and
making backports difficult) and probably should have been deferred to a new
major release instead. I think this refactoring is important for the
long-term maintainability of the NN and worth pursuing, but as a Hadoop 4.0
item. Merging HDSL is also not a prerequisite for starting this
refactoring. Really, I see the refactoring as the prerequisite for
HDFS-on-HDSL to be possible.

Finally, I earnestly believe that Ozone/HDSL itself would benefit from
being a separate project. Ozone could release faster and iterate more
quickly if it wasn't hampered by Hadoop's release schedule and security and
compatibility requirements. There are also publicity and community
benefits; it's an opportunity to build a community focused on the novel
capabilities and architectural choices of Ozone/HDSL. There are examples of
other projects that were "incubated" on a branch in the Hadoop repo before
being spun off to great success.

In conclusion, I'd like to see Ozone succeeding and thriving as a separate
project. Meanwhile, we can work on the HDFS refactoring required to
separate the FSN and BM and make it pluggable. At that point (likely in the
Hadoop 4 timeframe), we'll be ready to pursue HDFS-on-HDSL integration.

Best,
Andrew

On Tue, Feb 27, 2018 at 11:23 PM, Andrew Wang 
wrote:

>
>
>
>
>
>
>
>
>
> *Hi Jitendra and all,Thanks for putting this together. I caught up on the
> discussion on JIRA and document at HDFS-10419, and still have the same
> concerns raised earlier
> 
> about merging the Ozone branch to trunk.To recap these questions/concerns
> at a very high level:* Wouldn't Ozone benefit from being a separate
> project?* Why should it be merged now?I still believe that both Ozone and
> Hadoop would benefit from Ozone being a separate project, and that there is
> no pressing reason to merge Ozone/HDSL now.The primary reason I've heard
> for merging is that the Ozone is that it's at a stage where it's ready for
> user feedback. Second, that it needs to be merged to start on the NN
> refactoring for HDFS-on-HDSL.First, without HDFS-on-HDSL support, users are
> testing against the Ozone object storage interface. Ozone and HDSL
> themselves are implemented as separate masters and new functionality bolted
> onto the datanode. It also doesn't look like HDFS in terms of API or
> featureset; yes, it speaks FileSystem, but so do many out-of-tree storage
> systems like S3, Ceph, Swift, ADLS etc. Ozone/HDSL does not support popular
> HDFS features like erasure coding, encryption, high-availability,
> snapshots, hflush/hsync (and thus HBase), or APIs like WebHDFS or NFS. This
> means that Ozone feels like a new, different system that could reasonably
> be deployed and tested separately from HDFS. It's unlikely to replace many
> of today's HDFS deployments, and from what I understand, Ozone was not
> designed to do this.Second, the NameNode refactoring for HDFS-on-HDSL by
> itself is a major undertaking.

[jira] [Created] (HDFS-13212) RBF: Fix router location cache issue

[jira] [Created] (HDFS-13211) Refactor Unit Tests for SnapshotSKipList

[jira] [Created] (HDFS-13210) Fix the typo in MiniDFSCluster class

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

Re: [VOTE] Merging branch HDFS-7240 to trunk

[jira] [Created] (HDFS-13209) DistributedFileSystem.create should allow an option to provide StoragePolicy

[jira] [Created] (HDFS-13208) RBF: Mount path not available after ADD-REMOVE-ADD

Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

[jira] [Created] (HDFS-13207) Add blockpool used for DFSAdmin Report command

[jira] [Created] (HDFS-13206) IllegalStateException: Unable to finalize edits file

[DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

[jira] [Created] (HDFS-13205) Incorrect path is passed to checkPermission during authorization of file under a snapshot (specifically under a subdir) after original subdir is deleted

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (HDFS-13204) RBF: optimize name service safe mode icon

Re: [VOTE] Merging branch HDFS-7240 to trunk

15 matches

Site Navigation

Mail list logo

Footer information