[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120129#comment-17120129 ] zZtai commented on HDFS-15098: -- [~lindongdong] > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.download Bouncy Castle Crypto APIs from bouncycastle.org > [https://bouncycastle.org/download/bcprov-ext-jdk15on-165.jar] > 2.Configure JDK > Place bcprov-ext-jdk15on-165.jar in $JAVA_HOME/jre/lib/ext directory, > add "security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider" > to $JAVA_HOME/jre/lib/security/java.security file > 3.Configure Hadoop KMS > 4.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 > 2.configure Bouncy Castle Crypto on JDK -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120128#comment-17120128 ] zZtai commented on HDFS-15098: -- [~lindongdong] > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.download Bouncy Castle Crypto APIs from bouncycastle.org > [https://bouncycastle.org/download/bcprov-ext-jdk15on-165.jar] > 2.Configure JDK > Place bcprov-ext-jdk15on-165.jar in $JAVA_HOME/jre/lib/ext directory, > add "security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider" > to $JAVA_HOME/jre/lib/security/java.security file > 3.Configure Hadoop KMS > 4.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 > 2.configure Bouncy Castle Crypto on JDK -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Comment: was deleted (was: [~lindongdong] ) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.download Bouncy Castle Crypto APIs from bouncycastle.org > [https://bouncycastle.org/download/bcprov-ext-jdk15on-165.jar] > 2.Configure JDK > Place bcprov-ext-jdk15on-165.jar in $JAVA_HOME/jre/lib/ext directory, > add "security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider" > to $JAVA_HOME/jre/lib/security/java.security file > 3.Configure Hadoop KMS > 4.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 > 2.configure Bouncy Castle Crypto on JDK -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120129#comment-17120129 ] zZtai edited comment on HDFS-15098 at 5/30/20, 7:13 AM: [~lindongdong] as [~weichiu] mentioned , the existing crypto implementation should falls back to a Java implementation if openssl is not loaded, bouncycastle provides these capabilities . was (Author: zztai): [~lindongdong] > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.download Bouncy Castle Crypto APIs from bouncycastle.org > [https://bouncycastle.org/download/bcprov-ext-jdk15on-165.jar] > 2.Configure JDK > Place bcprov-ext-jdk15on-165.jar in $JAVA_HOME/jre/lib/ext directory, > add "security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider" > to $JAVA_HOME/jre/lib/security/java.security file > 3.Configure Hadoop KMS > 4.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 > 2.configure Bouncy Castle Crypto on JDK -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-15379) DataStreamer should reset thread interrupted state in createBlockOutputStream
[
https://issues.apache.org/jira/browse/HDFS-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ludun updated HDFS-15379:
-
Comment: was deleted
(was: @assignee)
> DataStreamer should reset thread interrupted state in createBlockOutputStream
> -
>
> Key: HDFS-15379
> URL: https://issues.apache.org/jira/browse/HDFS-15379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 2.7.7, 3.1.3
>Reporter: ludun
>Priority: Major
> Attachments: HDFS-15379.001.patch
>
>
> In createBlockOutputStream if thread was interrupted becuase timeout to
> conenct to DataNode.
> {code}2020-05-27 18:32:53,310 | DEBUG | Connecting to datanode
> xx.xx.xx.xx:25009 | DataStreamer.java:251
> 2020-05-27 18:33:50,457 | INFO | Exception in createBlockOutputStream
> blk_1115121199_41386360 | DataStreamer.java:1854
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:40370
> remote=/xx.xx.xx.xx:25009]. 615000 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551)
> at
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1826)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1743)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
> then abandonBlockrpc to namenode also failed due to interrupted exception
> immediately.
> {code}2020-05-27 18:33:50,461 | DEBUG | Connecting to xx/xx.xx.xx.xx:25000 |
> Client.java:814
> 2020-05-27 18:33:50,462 | DEBUG | Failed to connect to server:
> xx/xx.xx.xx.xx:25000: try once and fail. | Client.java:956
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:436)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1613)
> at org.apache.hadoop.ipc.Client.call(Client.java:1444)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.abandonBlock(ClientNamenodeProtocolTranslatorPB.java:509)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy11.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1748)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15379) DataStreamer should reset thread interrupted state in createBlockOutputStream
[
https://issues.apache.org/jira/browse/HDFS-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120132#comment-17120132
]
ludun commented on HDFS-15379:
--
@assignee
> DataStreamer should reset thread interrupted state in createBlockOutputStream
> -
>
> Key: HDFS-15379
> URL: https://issues.apache.org/jira/browse/HDFS-15379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 2.7.7, 3.1.3
>Reporter: ludun
>Priority: Major
> Attachments: HDFS-15379.001.patch
>
>
> In createBlockOutputStream if thread was interrupted becuase timeout to
> conenct to DataNode.
> {code}2020-05-27 18:32:53,310 | DEBUG | Connecting to datanode
> xx.xx.xx.xx:25009 | DataStreamer.java:251
> 2020-05-27 18:33:50,457 | INFO | Exception in createBlockOutputStream
> blk_1115121199_41386360 | DataStreamer.java:1854
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:40370
> remote=/xx.xx.xx.xx:25009]. 615000 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551)
> at
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1826)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1743)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
> then abandonBlockrpc to namenode also failed due to interrupted exception
> immediately.
> {code}2020-05-27 18:33:50,461 | DEBUG | Connecting to xx/xx.xx.xx.xx:25000 |
> Client.java:814
> 2020-05-27 18:33:50,462 | DEBUG | Failed to connect to server:
> xx/xx.xx.xx.xx:25000: try once and fail. | Client.java:956
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:436)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1613)
> at org.apache.hadoop.ipc.Client.call(Client.java:1444)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.abandonBlock(ClientNamenodeProtocolTranslatorPB.java:509)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy11.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1748)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoo
[jira] [Commented] (HDFS-15379) DataStreamer should reset thread interrupted state in createBlockOutputStream
[
https://issues.apache.org/jira/browse/HDFS-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120133#comment-17120133
]
ludun commented on HDFS-15379:
--
[~brahma]
> DataStreamer should reset thread interrupted state in createBlockOutputStream
> -
>
> Key: HDFS-15379
> URL: https://issues.apache.org/jira/browse/HDFS-15379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 2.7.7, 3.1.3
>Reporter: ludun
>Priority: Major
> Attachments: HDFS-15379.001.patch
>
>
> In createBlockOutputStream if thread was interrupted becuase timeout to
> conenct to DataNode.
> {code}2020-05-27 18:32:53,310 | DEBUG | Connecting to datanode
> xx.xx.xx.xx:25009 | DataStreamer.java:251
> 2020-05-27 18:33:50,457 | INFO | Exception in createBlockOutputStream
> blk_1115121199_41386360 | DataStreamer.java:1854
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:40370
> remote=/xx.xx.xx.xx:25009]. 615000 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551)
> at
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1826)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1743)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
> then abandonBlockrpc to namenode also failed due to interrupted exception
> immediately.
> {code}2020-05-27 18:33:50,461 | DEBUG | Connecting to xx/xx.xx.xx.xx:25000 |
> Client.java:814
> 2020-05-27 18:33:50,462 | DEBUG | Failed to connect to server:
> xx/xx.xx.xx.xx:25000: try once and fail. | Client.java:956
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:436)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1613)
> at org.apache.hadoop.ipc.Client.call(Client.java:1444)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.abandonBlock(ClientNamenodeProtocolTranslatorPB.java:509)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy11.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1748)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoo
[jira] [Commented] (HDFS-14960) TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology
[
https://issues.apache.org/jira/browse/HDFS-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120154#comment-17120154
]
Ayush Saxena commented on HDFS-14960:
-
Thanx [~Jim_Brennan] for the patch.
Minor nit :
{code:java}
Assert.assertEquals("Balancer did not exit with NO_MOVE_PROGRESS",
{code}
{{assertEquals}} is already imported, this can be just {{assertEquals}} rather
than {{Assert.assertEquals}}
Apart from that LGTM
> TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology
>
>
> Key: HDFS-14960
> URL: https://issues.apache.org/jira/browse/HDFS-14960
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.3
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Attachments: HDFS-14960.001.patch, HDFS-14960.002.patch,
> HDFS-14960.003.patch, HDFS-14960.004.patch, HDFS-14960.005.patch
>
>
> As reported in HDFS-14958, TestBalancerWithNodeGroup was succeeding even
> though it was using DFSNetworkTopology instead of
> NetworkTopologyWithNodeGroup.
> [~inigoiri] rightly suggested that this indicates the test is not very good -
> it should fail when run without NetworkTopologyWithNodeGroup.
> We should improve this test.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
tomscut created HDFS-15380:
--
Summary: Could not fetch real remote ip in
RouterWebHdfsMethods.java
Key: HDFS-15380
URL: https://issues.apache.org/jira/browse/HDFS-15380
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.1.0
Reporter: tomscut
Fix For: 3.4.0
Attachments: PATCH
We plan to add audit log for hdfs router, then we fetch remote ip via
Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
"REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
in construction method RouterWebHdfsMethods() and init(). When we call method
Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
"localhost/127.0.0.1" via InetAddress.getByName().
So we can change this field "REMOTE_ADDRESS" to a String value, just like
NamenodeWebHdfsMethods does.
{code:java}
2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods REMOTE_ADDRESS:
14.39.39.28, current thread: qtp476579021-10902020-05-27 19:15:18,827 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:init(150)) - init
REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
19:15:18,836 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr REMOTE_ADDRESS:
null, current thread: IPC Server handler 75 on 2020-05-27 19:15:18,837 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
thread: IPC Server handler 75 on
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Fix Version/s: (was: 3.4.0)
> Could not fetch real remote ip in RouterWebHdfsMethods.java
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: PATCH
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
> in construction method RouterWebHdfsMethods() and init(). When we call method
> Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
> ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
> "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Attachment: HDFS-15380.001.patch
> Could not fetch real remote ip in RouterWebHdfsMethods.java
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch, PATCH
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
> in construction method RouterWebHdfsMethods() and init(). When we call method
> Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
> ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
> "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Attachment: (was: PATCH)
> Could not fetch real remote ip in RouterWebHdfsMethods.java
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
> in construction method RouterWebHdfsMethods() and init(). When we call method
> Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
> ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
> "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Description:
We plan to add audit log for hdfs router, then we fetch remote ip via
Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
"REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
in construction method RouterWebHdfsMethods() and init(). When we call method
Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
"localhost/127.0.0.1" via InetAddress.getByName().
So we can change this field "REMOTE_ADDRESS" to a String value, just like
NamenodeWebHdfsMethods does.
I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
shown below:
{code:java}
2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods REMOTE_ADDRESS:
14.39.39.28, current thread: qtp476579021-10902020-05-27 19:15:18,827 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:init(150)) - init
REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
19:15:18,836 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr REMOTE_ADDRESS:
null, current thread: IPC Server handler 75 on 2020-05-27 19:15:18,837 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
thread: IPC Server handler 75 on
{code}
was:
We plan to add audit log for hdfs router, then we fetch remote ip via
Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
"REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
in construction method RouterWebHdfsMethods() and init(). When we call method
Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
"localhost/127.0.0.1" via InetAddress.getByName().
So we can change this field "REMOTE_ADDRESS" to a String value, just like
NamenodeWebHdfsMethods does.
{code:java}
2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods REMOTE_ADDRESS:
14.39.39.28, current thread: qtp476579021-10902020-05-27 19:15:18,827 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:init(150)) - init
REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
19:15:18,836 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr REMOTE_ADDRESS:
null, current thread: IPC Server handler 75 on 2020-05-27 19:15:18,837 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
thread: IPC Server handler 75 on
{code}
> Could not fetch real remote ip in RouterWebHdfsMethods.java
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-109020
[jira] [Updated] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Attachment: (was: HDFS-15380.001.patch)
> Could not fetch real remote ip in RouterWebHdfsMethods.java
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) Could not fetch real remote ip in RouterWebHdfsMethods.java
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Attachment: HDFS-15380.001.patch
> Could not fetch real remote ip in RouterWebHdfsMethods.java
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15380:
---
Summary: RBF: Could not fetch real remote IP in RouterWebHdfsMethods (was:
Could not fetch real remote ip in RouterWebHdfsMethods.java)
> RBF: Could not fetch real remote IP in RouterWebHdfsMethods
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120337#comment-17120337
]
Íñigo Goiri commented on HDFS-15380:
This approach was added by HDFS-12512.
I don't think there's any strong requirement from that JIRA but worth taking a
look and making sure we are not missing anything.
> RBF: Could not fetch real remote IP in RouterWebHdfsMethods
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
> 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-10902020-05-27 19:15:18,836 INFO
> router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
> getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on 2020-05-27
> 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15380:
---
Description:
We plan to add audit log for hdfs router, then we fetch remote ip via
Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
"REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
in construction method RouterWebHdfsMethods() and init(). When we call method
Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
"localhost/127.0.0.1" via InetAddress.getByName().
So we can change this field "REMOTE_ADDRESS" to a String value, just like
NamenodeWebHdfsMethods does.
I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
shown below:
{code:java}
2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods REMOTE_ADDRESS:
14.39.39.28, current thread: qtp476579021-1090
2020-05-27 19:15:18,827 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
current thread: qtp476579021-1090
2020-05-27 19:15:18,836 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr REMOTE_ADDRESS:
null, current thread: IPC Server handler 75 on
2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr REMOTE_ADDRESS:
null, current thread: IPC Server handler 75 on
2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
thread: IPC Server handler 75 on
{code}
was:
We plan to add audit log for hdfs router, then we fetch remote ip via
Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
"REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field, setting
in construction method RouterWebHdfsMethods() and init(). When we call method
Server.getRemoteIp() to fetch remote ip, the thread would be changed, so the
ThreadLocal field "REMOTE_ADDRESS" is null, and would be passed to
"localhost/127.0.0.1" via InetAddress.getByName().
So we can change this field "REMOTE_ADDRESS" to a String value, just like
NamenodeWebHdfsMethods does.
I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
shown below:
{code:java}
2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods REMOTE_ADDRESS:
14.39.39.28, current thread: qtp476579021-10902020-05-27 19:15:18,827 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:init(150)) - init
REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-10902020-05-27
19:15:18,836 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr REMOTE_ADDRESS:
null, current thread: IPC Server handler 75 on 2020-05-27 19:15:18,837 INFO
router.RouterWebHdfsMethods (RouterWebHdfsMethods.java:getRemoteAddr(170)) -
getRemoteAddr REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
(RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
thread: IPC Server handler 75 on
{code}
> RBF: Could not fetch real remote IP in RouterWebHdfsMethods
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-1090
> 2020-05-27 19:15:18,827 INFO router.RouterWebHdfsMethods
> (Router
[jira] [Updated] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15380:
---
Status: Patch Available (was: Open)
> RBF: Could not fetch real remote IP in RouterWebHdfsMethods
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-1090
> 2020-05-27 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-1090
> 2020-05-27 19:15:18,836 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri reassigned HDFS-15380:
--
Assignee: tomscut
> RBF: Could not fetch real remote IP in RouterWebHdfsMethods
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-1090
> 2020-05-27 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-1090
> 2020-05-27 19:15:18,836 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13179) TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120339#comment-17120339
]
Íñigo Goiri commented on HDFS-13179:
Yes, let's revert and use something like [^HDFS-13179-branch-2.10.003.patch].
> TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails
> intermittently
> --
>
> Key: HDFS-13179
> URL: https://issues.apache.org/jira/browse/HDFS-13179
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Gabor Bota
>Assignee: Ahmed Hussein
>Priority: Critical
> Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: HDFS-13179-branch-2.10.003.patch, HDFS-13179.001.patch,
> HDFS-13179.002.patch, HDFS-13179.003.patch, test runs.zip
>
>
> The error caused by TimeoutException because the test is waiting to ensure
> that the file is replicated to DISK storage but the replication can't be
> finished to DISK during the 30s timeout in ensureFileReplicasOnStorageType(),
> but the file is still on RAM_DISK - so there is no data loss.
> Adding the following to TestLazyPersistReplicaRecovery.java:56 essentially
> fixes the flakiness.
> {code:java}
> try {
> ensureFileReplicasOnStorageType(path1, DEFAULT);
> }catch (TimeoutException t){
> LOG.warn("We got \"" + t.getMessage() + "\" so trying to find data on
> RAM_DISK");
> ensureFileReplicasOnStorageType(path1, RAM_DISK);
> }
> }
> {code}
> Some thoughts:
> * Successful and failed tests run similar to the point when datanode
> restarts. Restart line is the following in the log: LazyPersistTestCase -
> Restarting the DataNode
> * There is a line which only occurs in the failed test: *addStoredBlock:
> Redundant addStoredBlock request received for blk_1073741825_1001 on node
> 127.0.0.1:49455 size 5242880*
> * This redundant request at BlockManager#addStoredBlock could be the main
> reason for the test fail. Something wrong with the gen stamp? Corrupt
> replicas?
> =
> Current fail ratio based on my test of TestLazyPersistReplicaRecovery:
> 1000 runs, 34 failures (3.4% fail)
> Failure rate analysis:
> TestLazyPersistReplicaRecovery.testDnRestartWithSavedReplicas: 3.4%
> 33 failures caused by: {noformat}
> java.util.concurrent.TimeoutException: Timed out waiting for condition.
> Thread diagnostics: Timestamp: 2018-01-05 11:50:34,964 "IPC Server handler 6
> on 39589"
> {noformat}
> 1 failure caused by: {noformat}
> java.net.BindException: Problem binding to [localhost:56729]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery.testDnRestartWithSavedReplicas(TestLazyPersistReplicaRecovery.java:49)
> Caused by: java.net.BindException: Address already in use at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery.testDnRestartWithSavedReplicas(TestLazyPersistReplicaRecovery.java:49)
> {noformat}
> =
> Example stacktrace:
> {noformat}
> Timed out waiting for condition. Thread diagnostics:
> Timestamp: 2017-11-01 10:36:49,499
> "Thread-1" prio=5 tid=13 runnable
> java.lang.Thread.State: RUNNABLE
> at java.lang.Thread.dumpThreads(Native Method)
> at java.lang.Thread.getAllStackTraces(Thread.java:1610)
> at
> org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87)
> at
> org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
> at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:369)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.ensureFileReplicasOnStorageType(LazyPersistTestCase.java:140)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery.testDnRestartWithSavedReplicas(TestLazyPersistReplicaRecovery.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14901) RBF: Add Encryption Zone related ClientProtocol APIs
[ https://issues.apache.org/jira/browse/HDFS-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120340#comment-17120340 ] Íñigo Goiri commented on HDFS-14901: We've been having this proposal for a while, yes, we should extend the RPC header. Not sure if it's incompatible itself, we should make it backwards compatible for sure. > RBF: Add Encryption Zone related ClientProtocol APIs > > > Key: HDFS-14901 > URL: https://issues.apache.org/jira/browse/HDFS-14901 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14901.001.patch, HDFS-14901.002.patch, > HDFS-14901.003.patch > > > Currently listEncryptionZones,reencryptEncryptionZone,listReencryptionStatus > these APIs are not implemented in Router. > This JIRA is intend to implement above mentioned APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15246) ArrayIndexOfboundsException in BlockManager CreateLocatedBlock
[ https://issues.apache.org/jira/browse/HDFS-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120342#comment-17120342 ] Íñigo Goiri commented on HDFS-15246: No point on catching the exception and fail(), just surface the exception. Can we assert that the snapshot was created, that the file has been truncated and something about the block locations? > ArrayIndexOfboundsException in BlockManager CreateLocatedBlock > -- > > Key: HDFS-15246 > URL: https://issues.apache.org/jira/browse/HDFS-15246 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15246-testrepro.patch, HDFS-15246.001.patch, > HDFS-15246.002.patch > > > java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 > > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1362) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1501) > at > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2047) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120354#comment-17120354
]
Hadoop QA commented on HDFS-15380:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
8s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
58s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29385/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15380 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13004418/HDFS-15380.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9a78f3eb4733 4.15.0-101-generic #102-Ubuntu SMP Mon May 11
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 19f26a020e2 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29385/testReport/ |
| Max. process+thread count | 3168 (vs. ulimit of 5500) |
| mo
