[jira] [Reopened] (HDDS-676) Enable Read from open Containers via Standalone Protocol

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer reopened HDDS-676:
---

There is an conflict when porting to 0.3 branch, can you please take this to 
that branch if needed?

> Enable Read from open Containers via Standalone Protocol
> 
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch, 
> HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, 
> HDDS-676.006.patch, HDDS-676.007.patch, HDDS-676.008.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open 
> containers in OM as well datanode, Ozone Client reads can through Standalone 
> protocol not necessarily requiring Ratis. Client should verify the BCSID of 
> the container which has the data block , which should always be greater than 
> or equal to the BCSID of the block to be read and the existing block BCSID 
> should exactly match that of the block to be read. As a part of this, Client 
> can try to read from a replica with a supplied BCSID and failover to the next 
> one in case the block does ont exist on one replica.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-712) Use x-amz-storage-class to specify replication type and replication factor

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDDS-712:
---

 Summary: Use x-amz-storage-class to specify replication type and 
replication factor
 Key: HDDS-712
 URL: https://issues.apache.org/jira/browse/HDDS-712
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


 

This has been a comment in the Jira in HDDS-693 from 

@DefaultValue("STAND_ALONE") @QueryParam("replicationType")

Just an opportunistic comment. Not part of this patch, this query param will 
not be sent by S3 hence this will always default to Stand_Alone. At some point 
we need to move to RATIS, Perhaps we have to read this via x-amz-storage-class.

*I propose below solution for this:*

Currently, in code we take query params replicationType and replicationFactor 
and default them to Stand alone and 1. But these query params cannot be passed 
from aws cli.

We want to use x-amz-storage-class header and pass the values. By default for 
S3 If you don't specify this it defaults to Standard. So, in Ozone over S3 
also, as we want to default to Ratis and replication factor three by default.

We can use the mapping Standard to RATIS and REDUCED_REDUNDANCY to Stand alone.

 

There are 2 more values 

STANDARD_IA and ONEZONE_IA these need to be considered later how we want to use 
them. Intially we are considering to use Standard and Reduced_Redundancy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-711) Add robot tests for scm cli commands

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDDS-711:
---

 Summary: Add robot tests for scm cli commands
 Key: HDDS-711
 URL: https://issues.apache.org/jira/browse/HDDS-711
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Bharat Viswanadham


Integration tests for scmcli are removed as part of HDDS-379.

This Jira is opened to add robot tests for scmcli commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-710) Make Block Commit Sequence (BCS) opaque to clients

[Arpit Agarwal (JIRA)]

Arpit Agarwal created HDDS-710:
--

 Summary: Make Block Commit Sequence (BCS) opaque to clients
 Key: HDDS-710
 URL: https://issues.apache.org/jira/browse/HDDS-710
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Client
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal


An immutable block is identified by the following:
- Container ID
- Local Block ID
- BCS (Block Commit Sequence ID)

All of these values are currently exposed to the client. Instead we can have a 
composite block ID that hides these details from the client. A first thought is 
a naive implementation that generates a 192-bit (3x64-bit) block ID.

Proposed by [~anu] in HDDS-676.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14017) ObserverReadProxyProviderWithIPFailover does not quite work

[Chen Liang (JIRA)]

Chen Liang created HDFS-14017:
-

 Summary: ObserverReadProxyProviderWithIPFailover does not quite 
work
 Key: HDFS-14017
 URL: https://issues.apache.org/jira/browse/HDFS-14017
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang


Currently {{ObserverReadProxyProviderWithIPFailover}} extends 
{{ObserverReadProxyProvider}}, and the only difference is changing the proxy 
factory to use {{IPFailoverProxyProvider}}. However this is not enough because 
when calling constructor of {{ObserverReadProxyProvider}} in super(...), the 
follow line:
{code}
nameNodeProxies = getProxyAddresses(uri,
HdfsClientConfigKeys.DFS_NAMENODE_RPC_ADDRESS_KEY);
{code}
will try to resolve the all configured NN addresses to do configured failover. 
But in the case of IPFailover, this does not really apply. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14016) ObserverReadProxyProvider can never enable observer read except in tests

[Chen Liang (JIRA)]

Chen Liang created HDFS-14016:
-

 Summary: ObserverReadProxyProvider can never enable observer read 
except in tests
 Key: HDFS-14016
 URL: https://issues.apache.org/jira/browse/HDFS-14016
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang


Currently in {{ObserverReadProxyProvider#invoke}}, only when 
{{observerReadEnabled && isRead(method)}} is true, the code will check whether 
to talk to Observer. Otherwise always talk to active. The issue here is that 
currently it can only be set through {{setObserverReadEnabled}}, which is used 
by tests only. So observer read is always disabled in deployment and no way to 
enable it. We may want to either expose a configuration key, or hard code it to 
true so it can only be changed for testing purpose, or simply remove this 
variable. This is closely related to HDFS-13923.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage

[Daniel Templeton (JIRA)]

Daniel Templeton created HDFS-14015:
---

 Summary: Improve error handling in hdfsThreadDestructor in native 
thread local storage
 Key: HDFS-14015
 URL: https://issues.apache.org/jira/browse/HDFS-14015
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton


In the hdfsThreadDestructor() function, we ignore the return value from the 
DetachCurrentThread() call.  We are seeing cases where a native thread dies 
while holding a JVM monitor, and it doesn't release the monitor.  We're hoping 
that logging this error instead of ignoring it will shed some light on the 
issue.  In any case, it's good programming practice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/

[Oct 21, 2018 4:50:10 PM] (eyang) HADOOP-15821. Addendum patch to fix Hadoop 
Registry parent project. 




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   module:hadoop-common-project/hadoop-registry 
   Exceptional return value of 
java.util.concurrent.ExecutorService.submit(Callable) ignored in 
org.apache.hadoop.registry.server.dns.RegistryDNS.addNIOTCP(InetAddress, int) 
At RegistryDNS.java:ignored in 
org.apache.hadoop.registry.server.dns.RegistryDNS.addNIOTCP(InetAddress, int) 
At RegistryDNS.java:[line 900] 
   Exceptional return value of 
java.util.concurrent.ExecutorService.submit(Callable) ignored in 
org.apache.hadoop.registry.server.dns.RegistryDNS.addNIOUDP(InetAddress, int) 
At RegistryDNS.java:ignored in 
org.apache.hadoop.registry.server.dns.RegistryDNS.addNIOUDP(InetAddress, int) 
At RegistryDNS.java:[line 926] 
   Exceptional return value of 
java.util.concurrent.ExecutorService.submit(Callable) ignored in 
org.apache.hadoop.registry.server.dns.RegistryDNS.serveNIOTCP(ServerSocketChannel,
 InetAddress, int) At RegistryDNS.java:ignored in 
org.apache.hadoop.registry.server.dns.RegistryDNS.serveNIOTCP(ServerSocketChannel,
 InetAddress, int) At RegistryDNS.java:[line 850] 

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.resourcemanager.TestRMAdminService 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   hadoop.mapreduce.v2.hs.server.TestHSAdminServer 
   hadoop.streaming.TestMultipleCachefiles 
   hadoop.streaming.TestSymLink 
   hadoop.streaming.TestMultipleArchiveFiles 
   hadoop.streaming.TestStreamingBadRecords 
   hadoop.streaming.TestFileArgs 
   hadoop.mapred.gridmix.TestGridmixSubmission 
   hadoop.mapred.gridmix.TestLoadJob 
   hadoop.mapred.gridmix.TestSleepJob 
   hadoop.mapred.gridmix.TestDistCacheEmulation 
   hadoop.tools.TestDistCh 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-compile-javac-root.txt
  [296K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-patch-pylint.txt
  [40K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-patch-shellcheck.txt
  [68K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/branch-findbugs-hadoop-common-project_hadoop-registry-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/934/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [4.0K]
   

Re: Hadoop 3.2 Release Plan proposal

[Steve Loughran]

its in.

good catch!

On 20 Oct 2018, at 01:35, Wei-Chiu Chuang 
mailto:weic...@cloudera.com>> wrote:

Thanks Sunil G for driving the release,
I filed HADOOP-15866 for a 
compat fix. If any one has cycle please review it, as I think it is needed for 
3.2.0.

On Thu, Oct 18, 2018 at 4:43 AM Sunil G 
mailto:sun...@apache.org>> wrote:
Hi Folks,

As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.

I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.

- Sunil


On Tue, Oct 16, 2018 at 8:59 AM Sunil G 
mailto:sun...@apache.org>> wrote:

> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender 
> mailto:jonben...@stripe.com>>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G 
>> mailto:sun...@apache.org>> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter 
>>> mailto:rkan...@cloudera.com>>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > 3.2.0-SNAPSHOT
>>> > ./pom.xml:3.2.0-SNAPSHOT
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> mailto:fab...@cloudera.com.invalid>>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G 
>>> >> mailto:sun...@apache.org>> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G 
>>> >> > mailto:sun...@apache.org>> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G 
>>> >> > > mailto:sun...@apache.org>>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G 
>>> >> > >> mailto:sun...@apache.org>>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > 

[jira] [Created] (HDDS-709) Modify Close Container handling sequence on datanodes

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDDS-709:


 Summary: Modify Close Container handling sequence on datanodes
 Key: HDDS-709
 URL: https://issues.apache.org/jira/browse/HDDS-709
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee


With quasi closed container state for handling majority node failures, the 
close container handling sequence in Datanodes need to change. Once the 
datanodes receive a close container command from SCM, the open container 
replicas individually be marked in the closing state. In a closing state, only 
the transactions coming from the Ratis leader  are allowed , all other write 
transaction will fail. A close container transaction will be queued via Ratis 
on the leader which will be replayed to the followers which makes it transition 
to CLOSED/QUASI CLOSED state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14014) Unable to change the state of DN to maintenance using dfs.hosts.maintenance

[venkata ram kumar ch (JIRA)]

venkata ram kumar ch created HDFS-14014:
---

 Summary: Unable to change the state of DN  to maintenance using 
dfs.hosts.maintenance
 Key: HDFS-14014
 URL: https://issues.apache.org/jira/browse/HDFS-14014
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: venkata ram kumar ch


 

hdfs-site.xml configurations :


 dfs.namenode.maintenance.replication.min
 1
 
 
 dfs.namenode.hosts.provider.classname
 
org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager
 
 
 dfs.hosts.maintenance
 /opt/lifeline2/install/hadoop/namenode/etc/hadoop/maintenance
 


 

maintenance file :

{ "hostName": "vm1", "port": 50076, "adminState": "IN_MAINTENANCE", 
"maintenanceExpireTimeInMS" : 1540204025000}

Command : 

/hadoop/namenode/bin # ./hdfs dfsadmin -refreshNodes
2018-10-22 17:45:54,286 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes where 
applicable
Refresh nodes failed for vm1:65110
Refresh nodes failed for vm2:65110
refreshNodes: 2 exceptions 
[org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): (No such 
file or directory)
 at java.io.FileInputStream.open0(Native Method)
 at java.io.FileInputStream.open(FileInputStream.java:195)
 at java.io.FileInputStream.(FileInputStream.java:138)
 at java.io.FileInputStream.(FileInputStream.java:93)
 at 
org.apache.hadoop.hdfs.util.CombinedHostsFileReader.readFile(CombinedHostsFileReader.java:75)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager.refresh(CombinedHostFileManager.java:215)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager.refresh(CombinedHostFileManager.java:210)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.refreshHostsReader(DatanodeManager.java:1195)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.refreshNodes(DatanodeManager.java:1177)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.refreshNodes(FSNamesystem.java:4488)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.refreshNodes(NameNodeRpcServer.java:1270)
 at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.refreshNodes(ClientNamenodeProtocolServerSideTranslatorPB.java:913)
 at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
, org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): (No 
such file or directory)
 at java.io.FileInputStream.open0(Native Method)
 at java.io.FileInputStream.open(FileInputStream.java:195)
 at java.io.FileInputStream.(FileInputStream.java:138)
 at java.io.FileInputStream.(FileInputStream.java:93)
 at 
org.apache.hadoop.hdfs.util.CombinedHostsFileReader.readFile(CombinedHostsFileReader.java:75)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager.refresh(CombinedHostFileManager.java:215)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager.refresh(CombinedHostFileManager.java:210)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.refreshHostsReader(DatanodeManager.java:1195)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.refreshNodes(DatanodeManager.java:1177)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.refreshNodes(FSNamesystem.java:4488)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.refreshNodes(NameNodeRpcServer.java:1270)
 at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.refreshNodes(ClientNamenodeProtocolServerSideTranslatorPB.java:913)
 at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)

 

[jira] [Created] (HDDS-708) Validate BCSID while reading blocks from containers in datanodes

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDDS-708:


 Summary: Validate BCSID while reading blocks from containers in 
datanodes
 Key: HDDS-708
 URL: https://issues.apache.org/jira/browse/HDDS-708
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee


Ozone client while making a getBlock call during reading data , should read the 
bcsId from OzoneManager for the block and the same needs to be validated in 
Datanode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14013) HDFS ZKFC on standby NN not starting with credentials stored in hdfs

[Krzysztof Adamski (JIRA)]

Krzysztof Adamski created HDFS-14013:


 Summary: HDFS ZKFC on standby NN not starting with credentials 
stored in hdfs
 Key: HDFS-14013
 URL: https://issues.apache.org/jira/browse/HDFS-14013
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 3.1.1
Reporter: Krzysztof Adamski
 Attachments: hadoop-hdfs-zkfc-server1.log

HDFS ZKFailoverController is not starting correctly on strandby NameNode when 
credential provideris stored in hdfs. Removing credential provider entry from 
core-site helps. See full exception stack attached.

It looks like if it only checks the credentials on the same host namenode, not 
redirects to the active one. It may make sense to delay credential check after 
active namenode is elected and redirect to the active namenode as well.

 
 
{code:java}
2018-10-22 08:17:09,251 FATAL tools.DFSZKFailoverController 
(DFSZKFailoverController.java:main(197)) - DFSZKFailOverController exiting due 
to earlier exception java.io.IOException: Configuration problem with provider 
path. 2018-10-22 08:17:09,252 DEBUG util.ExitUtil 
(ExitUtil.java:terminate(209)) - Exiting with status 1: java.io.IOException: 
Configuration problem with provider path. 1: java.io.IOException: Configuration 
problem with provider path. at 
org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:265) at 
org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:199)
 Caused by: java.io.IOException: Configuration problem with provider path. 
at 
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2363)
 at 
org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2282) 
at 
org.apache.hadoop.security.SecurityUtil.getZKAuthInfos(SecurityUtil.java:732)   
  at 
org.apache.hadoop.ha.ZKFailoverController.initZK(ZKFailoverController.java:343) 
at 
org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:194)  
   at 
org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:60)
 at 
org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:175)  
   at 
org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:171)  
   at java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:360) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
 at 
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
 at 
org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:171)    
 at 
org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:195)
 Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): 
Operation category READ is not supported in state standby. Visit 
https://s.apache.org/sbnn-error at 
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1951)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1427)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3100)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1154)
 at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
 at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at 
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at 
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) 
 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at 
org.apache.hadoop.ipc.Client.call(Client.java:1443) at 
org.apache.hadoop.ipc.Client.call(Client.java:1353) at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
 at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at