[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191925#comment-14191925 ] Hudson commented on YARN-2677: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1943 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1943/]) YARN-2677 registry punycoding of usernames doesn't fix all usernames to be DNS-valid (stevel) (stevel: rev 81fe8e414748161f537e6902021d63928f8635f1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/client/binding/TestRegistryOperationUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/operations/TestRegistryOperations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistryOperationsService.java * hadoop-yarn-project/CHANGES.txt > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.6.0 > > Attachments: YARN-2677-001.patch, YARN-2677-002.patch > > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191794#comment-14191794 ] Hudson commented on YARN-2677: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1918 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1918/]) YARN-2677 registry punycoding of usernames doesn't fix all usernames to be DNS-valid (stevel) (stevel: rev 81fe8e414748161f537e6902021d63928f8635f1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/client/binding/TestRegistryOperationUtils.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/operations/TestRegistryOperations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistryOperationsService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.6.0 > > Attachments: YARN-2677-001.patch, YARN-2677-002.patch > > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191719#comment-14191719 ] Hudson commented on YARN-2677: -- FAILURE: Integrated in Hadoop-Yarn-trunk #729 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/729/]) YARN-2677 registry punycoding of usernames doesn't fix all usernames to be DNS-valid (stevel) (stevel: rev 81fe8e414748161f537e6902021d63928f8635f1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/client/binding/TestRegistryOperationUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistryOperationsService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/operations/TestRegistryOperations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java * hadoop-yarn-project/CHANGES.txt > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.6.0 > > Attachments: YARN-2677-001.patch, YARN-2677-002.patch > > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190971#comment-14190971 ] Hudson commented on YARN-2677: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6399 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6399/]) YARN-2677 registry punycoding of usernames doesn't fix all usernames to be DNS-valid (stevel) (stevel: rev 81fe8e414748161f537e6902021d63928f8635f1) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/operations/TestRegistryOperations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistryOperationsService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/client/binding/TestRegistryOperationUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.6.0 > > Attachments: YARN-2677-001.patch, YARN-2677-002.patch > > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190151#comment-14190151 ] Hadoop QA commented on YARN-2677: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678180/YARN-2677-002.patch against trunk revision 179cab8. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5639//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5639//console This message is automatically generated. > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-2677-001.patch, YARN-2677-002.patch > > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167690#comment-14167690 ] Hadoop QA commented on YARN-2677: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12674272/YARN-2677-001.patch against trunk revision d3d3d47. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5368//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-YARN-Build/5368//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5368//console This message is automatically generated. > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-2677-001.patch > > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167486#comment-14167486 ] Steve Loughran commented on YARN-2677: -- it also looks like somehow the full kerberos name creeps in —that needs to be name-shortened the way UGI does {code} 14/10/10 17:35:45 ERROR main.ServiceLauncher: Exception: `/users/hb...@hwqe.hortonworks.com/services/org-apache-slider/hbcluster24': Invalid Path element "hb...@hwqe.hortonworks.com" org.apache.hadoop.registry.client.exceptions.InvalidPathnameException: `/users/hb...@apache.org/services/org-apache-slider/hbcluster24': Invalid Path element "hbase@@APACHE.ORG" at org.apache.hadoop.registry.client.binding.RegistryPathUtils.validateElementsAsDNS(RegistryPathUtils.java:78) at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.validatePath(RegistryOperationsService.java:91) at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.delete(RegistryOperationsService.java:151) at org.apache.slider.server.services.yarnregistry.YarnRegistryViewForProviders.putService(YarnRegistryViewForProviders.java:138) at org.apache.slider.server.appmaster.SliderAppMaster.registerServiceInstance(SliderAppMaster.java:1002) at org.apache.slider.server.appmaster.SliderAppMaster.createAndRunCluster(SliderAppMaster.java:846) at org.apache.slider.server.appmaster.SliderAppMaster.runService(SliderAppMaster.java:507) {code} > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167392#comment-14167392 ] Steve Loughran commented on YARN-2677: -- Proposed fix is # retain the restriction in the specification # retain the punycoding in the code # do NOT validate the hostname against DNS rules. If and when a switch to DNS exports takes place, the names will have to be escaped into DNS. The punycoding encoding guarantees that for a REST API, URLs paths will always be ASCII > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname chars, so stopping users with DNS-illegal names > (e.g. with an underscore in them) from being able to register -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2677) registry punycoding of usernames doesn't fix all usernames to be DNS-valid
[ https://issues.apache.org/jira/browse/YARN-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167385#comment-14167385 ] Steve Loughran commented on YARN-2677: -- SLIDER-500 picked this up: {code} ServiceRecord{description='Slider Application Master'; external endpoints: {Endpoint{api='org.apache.slider.appmaster', addressType='host/port', protocolType='hadoop/protobuf', addresses=[ [ "ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal" "50767" ] ] }; Endpoint{api='org.apache.http.UI', addressType='uri', protocolType='webui', addresses=[ [ "http://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:40805"; ] ] }; Endpoint{api='org.apache.slider.management', addressType='uri', protocolType='REST', addresses=[ [ "http://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:40805/ws/v1/slider/mgmt"; ] ] }; Endpoint{api='org.apache.slider.publisher', addressType='uri', protocolType='REST', addresses=[ [ "http://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:40805/ws/v1/slider/publisher"; ] ] }; Endpoint{api='org.apache.slider.registry', addressType='uri', protocolType='REST', addresses=[ [ "http://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:40805/ws/v1/slider/registry"; ] ] }; Endpoint{api='org.apache.slider.publisher.configurations', addressType='uri', protocolType='REST', addresses=[ [ "http://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:40805/ws/v1/slider/publisher/slider"; ] ] }; }; internal endpoints: {Endpoint{api='org.apache.slider.agents.secure', addressType='uri', protocolType='REST', addresses=[ [ "https://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:55485/ws/v1/slider/agents"; ] ] }; Endpoint{api='org.apache.slider.agents.oneway', addressType='uri', protocolType='REST', addresses=[ [ "https://ts-oracle6-uns-accumulo-slider-1010-4.cs1cloud.internal:55688/ws/v1/slider/agents"; ] ] }; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1412950151590_0009" }}Exception: `/users/hrt_qa/services/org-apache-slider/test-slider-accumulo-continuous-ingest': Invalid Path element "hrt_qa"14/10/10 15:44:53 ERROR main.ServiceLauncher: Exception: `/users/hrt_qa/services/org-apache-slider/test-slider-accumulo-continuous-ingest': Invalid Path element "hrt_qa"org.apache.hadoop.registry.client.exceptions.InvalidPathnameException: `/users/hrt_qa/services/org-apache-slider/test-slider-accumulo-continuous-ingest': Invalid Path element "hrt_qa" at org.apache.hadoop.registry.client.binding.RegistryPathUtils.validateElementsAsDNS(RegistryPathUtils.java:78) at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.validatePath(RegistryOperationsService.java:91) at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.delete(RegistryOperationsService.java:151) at org.apache.slider.server.services.yarnregistry.YarnRegistryViewForProviders.putService(YarnRegistryViewForProviders.java:138) at org.apache.slider.server.appmaster.SliderAppMaster.registerServiceInstance(SliderAppMaster.java:1002) at org.apache.slider.server.appmaster.SliderAppMaster.createAndRunCluster(SliderAppMaster.java:846) at org.apache.slider.server.appmaster.SliderAppMaster.runService(SliderAppMaster.java:507) at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:186) at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:471) at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:401) at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:626) at org.apache.slider.server.appmaster.SliderAppMaster.main(SliderAppMaster.java:2124)14/10/10 15:44:53 INFO util.ExitUtil: Exiting with status 3214/10/10 15:44:53 INFO mortbay.log: Shutdown hook executing14/10/10 15:44:53 INFO mortbay.log: Stopped SslSelectChannelConnector@0.0.0.0:5548514/10/10 15:44:53 INFO mortbay.log: Stopped SslSelectChannelConnector@0.0.0.0:55688 {code} > registry punycoding of usernames doesn't fix all usernames to be DNS-valid > -- > > Key: YARN-2677 > URL: https://issues.apache.org/jira/browse/YARN-2677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > The registry has a restriction "DNS-valid names only" to retain the future > option of DNS exporting of the registry. > to handle complex usernames, it punycodes the username first, using Java's > {{java.net.IDN}} class. > This turns out to only map high unicode-> ASCII, and does nothing for > ascii-but-invalid-hostname char