[jira] [Created] (HADOOP-14240) Configuration#get return value optimization
Jonathan Eagles created HADOOP-14240: Summary: Configuration#get return value optimization Key: HADOOP-14240 URL: https://issues.apache.org/jira/browse/HADOOP-14240 Project: Hadoop Common Issue Type: Improvement Reporter: Jonathan Eagles Assignee: Jonathan Eagles The string array return value can be more efficiently determined and some general redundancies can be removed to improve the speed for Configuration.get. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14239) S3A Retry Multiple S3 Key Deletion
Kazuyuki Tanimura created HADOOP-14239: -- Summary: S3A Retry Multiple S3 Key Deletion Key: HADOOP-14239 URL: https://issues.apache.org/jira/browse/HADOOP-14239 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1 Environment: EC2, AWS Reporter: Kazuyuki Tanimura When fs.s3a.multiobjectdelete.enable == true, It tries to delete multiple S3 keys at once. Although this is a great feature, it becomes problematic when AWS fails deleting some S3 keys out of the deletion list. The aws-java-sdk internally retries to delete them, but it does not help because it simply retries the same list of S3 keys including the successfully deleted ones. In that case, all successive retries fail deleting previously deleted keys since they do not exist any more. Eventually it throws an Exception and leads to a job failure entirely. Luckily, the AWS API reports which keys it failed to delete. We should retry only for the keys that failed to be deleted from S3A -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
Tsuyoshi Ozawa created HADOOP-14238: --- Summary: [Umbrella] Rechecking Guava's object is not exposed to user-facing API Key: HADOOP-14238 URL: https://issues.apache.org/jira/browse/HADOOP-14238 Project: Hadoop Common Issue Type: Improvement Reporter: Tsuyoshi Ozawa This is reported by [~hitesh] on HADOOP-10101. At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Thanks Allen, I already add it back in JIRA's 2.8.0 release date. However, I met another issue in uploading our 2.8.0 release bits to SVN: hadoop-2.8.0.tar.gz get uploaded failed with following exception: Adding (bin) hadoop-2.8.0/hadoop-2.8.0.tar.gz Transmitting file data .svn: E175002: Commit failed (details follow): svn: E175002: PUT request on '/repos/dist/!svn/txr/18902-g1s/release/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz' failed Other (smaller) files get uploaded successfully. Once I suspected it could due to no space, so I removed some old release bits (like: 2.5.2) but still no luck. So may be the size of hadoop-2.8.0.tar.gz become a problem here? I already reported the issue in https://issues.apache.org/jira/browse/INFRA-13749, but will appreciate if someone got idea on how to get through this. Thanks, Junping From: Allen Wittenauer Sent: Thursday, March 23, 2017 5:27 PM To: Junping Du Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3) Just a heads up. Looks like some removed the Finish Date off of 2.8.0 in JIRA. It needs to be put back to match what is in the artifacts that we voted on. - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Instances
Kazuyuki Tanimura created HADOOP-14237: -- Summary: S3A Support Shared Instance Profile Credentials Across All Instances Key: HADOOP-14237 URL: https://issues.apache.org/jira/browse/HADOOP-14237 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1 Environment: EC2, AWS Reporter: Kazuyuki Tanimura When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails getting the instance profile credentials, eventually all jobs on the cluster fail. Since a number of S3A clients (all mappers and reducers) try to get the credentials, the AWS credential endpoint starts responding 5xx and 4xx error codes. SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, but it still does not share the credentials with other EC2 instances / processes. This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
Mingliang Liu created HADOOP-14236: -- Summary: S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store Key: HADOOP-14236 URL: https://issues.apache.org/jira/browse/HADOOP-14236 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu After running integration test {{ITestS3AFileSystemContract}}, I found the following items are not cleaned up in DynamoDB: {code} parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir, child=subdir parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir, child=file2 {code} At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we need to be careful when cleaning up test data. Then I found it’s a bug in the code of integrating S3Guard with S3AFileSystem: for rename we miss sub-directory items to put (dest) and delete (src). The reason is that in S3A, we delete those fake directory objects if they are not necessary, e.g. non-empty. So when we list the objects to rename, the object summaries will only return _file_ objects. This has two consequences after rename: # there will be left items for src path in metadata store - left-overs will confuse {{get(Path)}} which should return null # we are not persisting the whole subtree for dest path to metadata store - this will break the DynamoDBMetadataStore invariant: _if a path exists, all its ancestors will also exist in the table_. Existing tests are not complaining about this though. If this is a real bug, let’s address it here. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14235) S3A Path does not understand colon (:) when globbing
Kazuyuki Tanimura created HADOOP-14235: -- Summary: S3A Path does not understand colon (:) when globbing Key: HADOOP-14235 URL: https://issues.apache.org/jira/browse/HADOOP-14235 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1 Reporter: Kazuyuki Tanimura S3 paths, colons (:) are valid character in S3 paths. However, the Java URI class, which is used in the Path class, does not allow it. This becomes a problem particularly when we are globbing S3 paths. The globber thinks paths with colons are invalid paths and throws URISyntaxException. The reason is we are sharing Globber.java with all other Fs. Some of the rules for regular Fs are not applicable to S3 just like this colon as an example. Same issue is reported here https://issues.apache.org/jira/browse/SPARK-20061 The good news is I have a one line fix that I am about to send a pull request. However, for a right fix, we should separate the S3 globber from the Globber.java as proposed at https://issues.apache.org/jira/browse/HADOOP-13371 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14234) Improve ADLS FileSystem tests with JUnit4
John Zhuge created HADOOP-14234: --- Summary: Improve ADLS FileSystem tests with JUnit4 Key: HADOOP-14234 URL: https://issues.apache.org/jira/browse/HADOOP-14234 Project: Hadoop Common Issue Type: Improvement Components: fs/adl, test Affects Versions: 2.8.0 Reporter: John Zhuge Priority: Minor HADOOP-14180 switches FileSystem contract tests to JUnit4 and makes various enhancements. Improve ADLS FileSystem contract tests based on that. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/ [Mar 23, 2017 12:54:54 PM] (stevel) HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in [Mar 23, 2017 4:28:10 PM] (raviprak) HADOOP-14213. Move Configuration runtime check for hadoop-site.xml to [Mar 23, 2017 6:33:29 PM] (liuml07) HADOOP-14135. Remove URI parameter in AWSCredentialProvider [Mar 23, 2017 7:13:23 PM] (cdouglas) HADOOP-14196. Azure Data Lake doc is missing required config entry. [Mar 23, 2017 7:17:27 PM] (cdouglas) HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John [Mar 23, 2017 8:34:03 PM] (liuml07) HADOOP-14214. DomainSocketWatcher::add()/delete() should not self [Mar 23, 2017 8:57:08 PM] (jzhuge) HADOOP-14195. CredentialProviderFactory$getProviders is not thread-safe. [Mar 24, 2017 6:31:27 AM] (jzhuge) HDFS-10649. Remove unused PermissionStatus#applyUMask. Contributed by -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestShellBasedUnixGroupsMapping hadoop.hdfs.TestEncryptedTransfer hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-compile-root.txt [136K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-compile-root.txt [136K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-compile-root.txt [136K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [244K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [68K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/267/artifact/out/patch-unit-hadoop-mapreduce-projec
Re: HADOOP-14163 proposal for new hadoop.apache.org
Thanks again for working on this Marton! Based on my read of the blog post you linked, we should have the git branch ready before asking infra to switch it over. I can do a more detailed review on the JIRA once you rev, and can help with the INFRA ticket once it's ready. We'll also have to update BUILDING.txt and the wiki instructions as part of this. Best, Andrew On Fri, Mar 24, 2017 at 3:06 AM, Marton Elek wrote: > > > Thank you all of the feedbacks, I fixed all of them (except one, see the > comment below) and updated the http://hadoop.anzix.net preview site. > > So the next steps: > > 0. Let me know if you have any comment about the latest version > > 1. I wait for the 2.8.0 announcement, and migrate the new announcement as > well. (wouldn't like to complicate the 2.8.0 with the site change) > > 2. I like the suggestion of Owen to move the site to a specific git > branch. I wouldn't like to pending on it if it's too much time, but if any > of the commiters could pick it up, I would wait for it. > > I tested it, and seems to be easy: > > git svn clone https://svn.apache.org/repos/asf/hadoop/common/site/main > cd main > git remote add elek g...@github.com:elek/hadoop.git > git push elek master:asf-site > > According to the blog entry, an INFRA issue should be opened (I guess by a > commiter or maybe a pmc member): > > https://blogs.apache.org/infra/entry/git_based_websites_available > > 3. After that I can submit the new site as a regular patch against the > asf-site branch. > > 4. If it's merged, I can update the release wiki pages > > Marton > > ps: > > The only suggested item which is not implemented is the short version > names in the documentation menu (2.7 instead of 2.7.3). > > I think there are two forces: usability of the site and the simplicity of > the site generation. Ideally a new release could be added to the site as > easy as possible (that was one of the motivation of the migration). > > While a new tag could be added to the header of the markdown files (eg: > versionLine: 3.0), it requires multiple files update during a new release. > And if something would be missed, there could be displayed multiple "2.7" > menu item (one for 2.7.3 and for 2.7.4). So the current method is not so > nice, but much more bug-safe. > > I prefer to keep the current/content in this step (if possible) and if the > site is migrated we can submit new patches (hopefully against a git branch) > in the normal way and further improve the site. > > > > From: Owen O'Malley > Sent: Monday, March 13, 2017 6:15 PM > To: Marton Elek > Cc: common-dev@hadoop.apache.org > Subject: Re: HADOOP-14163 proposal for new hadoop.apache.org > > Thanks for addressing this. Getting rid of Hadoop's use of forrest is a > good thing. > > In terms of content, the documentation links should be sorted by number > with only the latest from each minor release line (eg. 3.0, 2.7, 2.6). > > The download page points to the mirrors for checksums and signatures. It > should use the direct links, such as > > https://dist.apache.org/repos/dist/release/hadoop/common/ > hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.asc > https://dist.apache.org/repos/dist/release/hadoop/common/ > hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.mds > > Speaking of which, Hadoop's dist directory is huge and should be heavily > pruned. We should probably take it down to just hadoop-2.6.5, hadoop-2.7.3, > and hadoop-3.0.0-alpha2. > > You might also want to move us to git-pubsub so that we can use a branch in > our source code git repository to publish the html. Typically this uses the > asf-site branch. > > .. Owen > > On Mon, Mar 13, 2017 at 7:28 AM, Marton Elek > wrote: > > > > > Hi, > > > > In the previous thread the current forrest based hadoop site is > identified > > as one of the pain points of the release process. > > > > I created a new version of the site with exactly the same content. > > > > As it uses newer site generator (hugo), now: > > > > 1. It’s enough to create one new markdown file per release, and all the > > documentation/download links will be automatically added. > > 2. It requires only one single binary to render. > > > > > > A preview version is temporary hosted at > > > > http://hadoop.anzix.net/ > > > > to make it easier to review. > > > > > > For more details, you can check my comments on the issue > > https://issues.apache.org/jira/browse/HADOOP-14163 > > > > I would be thankful to get any feedback/review. > > > > Cheers, > > Marton > > > > > > > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
[jira] [Created] (HADOOP-14233) Don't pre-construct PreCondition.check failure message in Configuration#set
Jonathan Eagles created HADOOP-14233: Summary: Don't pre-construct PreCondition.check failure message in Configuration#set Key: HADOOP-14233 URL: https://issues.apache.org/jira/browse/HADOOP-14233 Project: Hadoop Common Issue Type: Improvement Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14232) RumenToSLS: rackName may contains slashes
[ https://issues.apache.org/jira/browse/HADOOP-14232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Vaudour reopened HADOOP-14232: - > RumenToSLS: rackName may contains slashes > - > > Key: HADOOP-14232 > URL: https://issues.apache.org/jira/browse/HADOOP-14232 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.6.0 >Reporter: Julien Vaudour >Priority: Minor > > Rack name may contain slashes, so hostName will contain several slashes. > Separation between rack name and hostName is the last slash. > For example: /platform1/pod1/rack1/node1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14232) RumenToSLS: rackName may contains slashes
[ https://issues.apache.org/jira/browse/HADOOP-14232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Vaudour resolved HADOOP-14232. - Resolution: Fixed > RumenToSLS: rackName may contains slashes > - > > Key: HADOOP-14232 > URL: https://issues.apache.org/jira/browse/HADOOP-14232 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.6.0 >Reporter: Julien Vaudour >Priority: Minor > > Rack name may contain slashes, so hostName will contain several slashes. > Separation between rack name and hostName is the last slash. > For example: /platform1/pod1/rack1/node1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14232) RumenToSLS: rackName may contains slashes
[ https://issues.apache.org/jira/browse/HADOOP-14232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Vaudour resolved HADOOP-14232. - Resolution: Won't Fix > RumenToSLS: rackName may contains slashes > - > > Key: HADOOP-14232 > URL: https://issues.apache.org/jira/browse/HADOOP-14232 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.6.0 >Reporter: Julien Vaudour >Priority: Minor > > Rack name may contain slashes, so hostName will contain several slashes. > Separation between rack name and hostName is the last slash. > For example: /platform1/pod1/rack1/node1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14232) RumenToSLS: rackName may contains slashes
Julien Vaudour created HADOOP-14232: --- Summary: RumenToSLS: rackName may contains slashes Key: HADOOP-14232 URL: https://issues.apache.org/jira/browse/HADOOP-14232 Project: Hadoop Common Issue Type: Bug Components: tools Affects Versions: 2.6.0 Reporter: Julien Vaudour Priority: Minor Rack name may contain slashes, so hostName will contain several slashes. Separation between rack name and hostName is the last slash. For example: /platform1/pod1/rack1/node1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14231) Using parentheses is not allowed in auth_to_local regex
Andras Bokor created HADOOP-14231: - Summary: Using parentheses is not allowed in auth_to_local regex Key: HADOOP-14231 URL: https://issues.apache.org/jira/browse/HADOOP-14231 Project: Hadoop Common Issue Type: Improvement Reporter: Andras Bokor Assignee: Andras Bokor Priority: Minor I tried to set the following property for auth_to_local property: {code}"RULE:[2:$1]((n|d)n)s/.*/hdfs//{code} but I got the following exception: {code}Exception in thread "main" java.util.regex.PatternSyntaxException: Unclosed group near index 9 (nn|dn|jn{code} I found that this occurs because {{ruleParser}} in {{org.apache.hadoop.security.authentication.util.KerberosName}} excludes closing parentheses. I do not really see the value of excluding parentheses (do I miss something?) so I would remove this restriction to be able to use more regex functionalities. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14230) TestAdlFileSystemContractLive fails to clean up
John Zhuge created HADOOP-14230: --- Summary: TestAdlFileSystemContractLive fails to clean up Key: HADOOP-14230 URL: https://issues.apache.org/jira/browse/HADOOP-14230 Project: Hadoop Common Issue Type: Sub-task Components: fs/adl, test Affects Versions: 2.8.0 Reporter: John Zhuge Assignee: John Zhuge Priority: Minor TestAdlFileSystemContractLive fails to clean up test directories after the tests. This is the leftover after {{testListStatus}}: {nonformat} $ bin/hadoop fs -ls -R / drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge/FileSystemContractBaseTest drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge/FileSystemContractBaseTest/testListStatus drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge/FileSystemContractBaseTest/testListStatus/a drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge/FileSystemContractBaseTest/testListStatus/b drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge/FileSystemContractBaseTest/testListStatus/c drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:17 /user/jzhuge/FileSystemContractBaseTest/testListStatus/c/1 {noformat} This is the leftover after {{testMkdirsFailsForSubdirectoryOfExistingFile}}: {noformat} $ bin/hadoop fs -ls -R / drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:22 /user drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:22 /user/jzhuge drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:22 /user/jzhuge/FileSystemContractBaseTest drwxr-xr-x - ADLSAccessApp loginapp 0 2017-03-24 08:22 /user/jzhuge/FileSystemContractBaseTest/testMkdirsFailsForSubdirectoryOfExistingFile -rw-r--r-- 1 ADLSAccessApp loginapp 2048 2017-03-24 08:22 /user/jzhuge/FileSystemContractBaseTest/testMkdirsFailsForSubdirectoryOfExistingFile/file {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/ [Mar 23, 2017 12:54:54 PM] (stevel) HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in [Mar 23, 2017 4:28:10 PM] (raviprak) HADOOP-14213. Move Configuration runtime check for hadoop-site.xml to [Mar 23, 2017 6:33:29 PM] (liuml07) HADOOP-14135. Remove URI parameter in AWSCredentialProvider [Mar 23, 2017 7:13:23 PM] (cdouglas) HADOOP-14196. Azure Data Lake doc is missing required config entry. [Mar 23, 2017 7:17:27 PM] (cdouglas) HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John [Mar 23, 2017 8:34:03 PM] (liuml07) HADOOP-14214. DomainSocketWatcher::add()/delete() should not self [Mar 23, 2017 8:57:08 PM] (jzhuge) HADOOP-14195. CredentialProviderFactory$getProviders is not thread-safe. [Mar 24, 2017 6:31:27 AM] (jzhuge) HDFS-10649. Remove unused PermissionStatus#applyUMask. Contributed by -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.net.TestDNS hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-compile-javac-root.txt [184K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [136K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/355/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14229) hadoop.security.auth_to_local example is incorrect in the documentation
Andras Bokor created HADOOP-14229: - Summary: hadoop.security.auth_to_local example is incorrect in the documentation Key: HADOOP-14229 URL: https://issues.apache.org/jira/browse/HADOOP-14229 Project: Hadoop Common Issue Type: Bug Reporter: Andras Bokor Assignee: Andras Bokor Priority: Trivial Let's see jhs as example: {code}RULE:[2:$1@$0](jhs/.*@.*REALM.TLD)s/.*/mapred/{code} That means principal has 2 components (jhs/myhost@REALM). The second column converts this to jhs@REALM. So the regex will not match on this since regex expects / in the principal. I am not sure what was the original intend here so I suggest two solutions: {code}RULE:[2:$1/$2@$0](jhs/.*@REALM.TLD)s/.*/mapred/{code} or {code}RULE:[2:$1@$0](j...@realm.tld)s/.*/mapred/{code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: HADOOP-14163 proposal for new hadoop.apache.org
Thank you all of the feedbacks, I fixed all of them (except one, see the comment below) and updated the http://hadoop.anzix.net preview site. So the next steps: 0. Let me know if you have any comment about the latest version 1. I wait for the 2.8.0 announcement, and migrate the new announcement as well. (wouldn't like to complicate the 2.8.0 with the site change) 2. I like the suggestion of Owen to move the site to a specific git branch. I wouldn't like to pending on it if it's too much time, but if any of the commiters could pick it up, I would wait for it. I tested it, and seems to be easy: git svn clone https://svn.apache.org/repos/asf/hadoop/common/site/main cd main git remote add elek g...@github.com:elek/hadoop.git git push elek master:asf-site According to the blog entry, an INFRA issue should be opened (I guess by a commiter or maybe a pmc member): https://blogs.apache.org/infra/entry/git_based_websites_available 3. After that I can submit the new site as a regular patch against the asf-site branch. 4. If it's merged, I can update the release wiki pages Marton ps: The only suggested item which is not implemented is the short version names in the documentation menu (2.7 instead of 2.7.3). I think there are two forces: usability of the site and the simplicity of the site generation. Ideally a new release could be added to the site as easy as possible (that was one of the motivation of the migration). While a new tag could be added to the header of the markdown files (eg: versionLine: 3.0), it requires multiple files update during a new release. And if something would be missed, there could be displayed multiple "2.7" menu item (one for 2.7.3 and for 2.7.4). So the current method is not so nice, but much more bug-safe. I prefer to keep the current/content in this step (if possible) and if the site is migrated we can submit new patches (hopefully against a git branch) in the normal way and further improve the site. From: Owen O'Malley Sent: Monday, March 13, 2017 6:15 PM To: Marton Elek Cc: common-dev@hadoop.apache.org Subject: Re: HADOOP-14163 proposal for new hadoop.apache.org Thanks for addressing this. Getting rid of Hadoop's use of forrest is a good thing. In terms of content, the documentation links should be sorted by number with only the latest from each minor release line (eg. 3.0, 2.7, 2.6). The download page points to the mirrors for checksums and signatures. It should use the direct links, such as https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.asc https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.mds Speaking of which, Hadoop's dist directory is huge and should be heavily pruned. We should probably take it down to just hadoop-2.6.5, hadoop-2.7.3, and hadoop-3.0.0-alpha2. You might also want to move us to git-pubsub so that we can use a branch in our source code git repository to publish the html. Typically this uses the asf-site branch. .. Owen On Mon, Mar 13, 2017 at 7:28 AM, Marton Elek wrote: > > Hi, > > In the previous thread the current forrest based hadoop site is identified > as one of the pain points of the release process. > > I created a new version of the site with exactly the same content. > > As it uses newer site generator (hugo), now: > > 1. It’s enough to create one new markdown file per release, and all the > documentation/download links will be automatically added. > 2. It requires only one single binary to render. > > > A preview version is temporary hosted at > > http://hadoop.anzix.net/ > > to make it easier to review. > > > For more details, you can check my comments on the issue > https://issues.apache.org/jira/browse/HADOOP-14163 > > I would be thankful to get any feedback/review. > > Cheers, > Marton > > > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14228) Fix getNumMapTasks() documentation in JobConf
Joe Mészáros created HADOOP-14228: - Summary: Fix getNumMapTasks() documentation in JobConf Key: HADOOP-14228 URL: https://issues.apache.org/jira/browse/HADOOP-14228 Project: Hadoop Common Issue Type: Bug Components: contrib/cloud, documentation Reporter: Joe Mészáros Priority: Minor The original description of the getNumMapTasks() method in JobConf was invalid, because it referenced to the number of reducer tasks instead of the map tasks. from: Get configured the number of reduce tasks for this job. to: Get the configured number of map tasks for this job. It was maybe the result of a tricky copy-paste ;-) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org