[jira] [Created] (HADOOP-13632) Daemonization does not check process liveness before renicing
Andrew Wang created HADOOP-13632:
Summary: Daemonization does not check process liveness before
renicing
Key: HADOOP-13632
URL: https://issues.apache.org/jira/browse/HADOOP-13632
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0-alpha1
Reporter: Andrew Wang
If you try to daemonize a process that is incorrectly configured, it will die
quite quickly. However, the daemonization function will still try to renice it
even if it's down, leading to something like this for my namenode:
{noformat}
-> % bin/hdfs --daemon start namenode
ERROR: Cannot set priority of namenode process 12036
{noformat}
It'd be more user-friendly instead of this renice error, we said that the
process couldn't be started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13631) S3Guard: implement move() for LocalMetadataStore, add unit tests
Aaron Fabbri created HADOOP-13631: - Summary: S3Guard: implement move() for LocalMetadataStore, add unit tests Key: HADOOP-13631 URL: https://issues.apache.org/jira/browse/HADOOP-13631 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri Building on HADOOP-13573 and HADOOP-13452, implement move() in LocalMetadataStore and associated MetadataStore unit tests. (Making this a separate JIRA to break up work into decent-sized and reviewable chunks.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13630) split up AWS index.md
Steve Loughran created HADOOP-13630: --- Summary: split up AWS index.md Key: HADOOP-13630 URL: https://issues.apache.org/jira/browse/HADOOP-13630 Project: Hadoop Common Issue Type: Sub-task Components: documentation, fs/s3 Affects Versions: 2.9.0 Reporter: Steve Loughran The AWS index.md file is too big, too much written by developers as we go along, not for end users. I propose splitting it into its own docs * Intro * S3A * S3N * S3 (branch-2 only, obviously) * testing * maybe in future: something on effective coding against object stores, though that could go toplevel, as it applies to all I propose waiting for HADOOP-13560 to be in, as that changes the docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/ [Sep 19, 2016 6:17:03 PM] (wang) HDFS-10868. Remove stray references to DFS_HDFS_BLOCKS_METADATA_ENABLED. [Sep 19, 2016 8:31:35 PM] (jlowe) YARN-5540. Scheduler spends too much time looking at empty priorities. [Sep 19, 2016 10:16:47 PM] (cnauroth) HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by [Sep 20, 2016 4:44:42 AM] (xiao) HDFS-10875. Optimize du -x to cache intermediate result. Contributed by [Sep 20, 2016 7:03:31 AM] (jianhe) YARN-3140. Improve locks in AbstractCSQueue/LeafQueue/ParentQueue. -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ipc.TestRPCWaitForProxy hadoop.hdfs.TestWriteReadStripedFile hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.tools.TestDFSAdminWithHA hadoop.hdfs.server.namenode.TestNestedEncryptionZones hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestRMAdminService hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.TestRMFailover hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.mapred.TestMROpportunisticMaps compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-compile-root.txt [308K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-compile-root.txt [308K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-compile-root.txt [308K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/100/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K]
[jira] [Created] (HADOOP-13629) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
Sean Mackrory created HADOOP-13629: -- Summary: Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers Key: HADOOP-13629 URL: https://issues.apache.org/jira/browse/HADOOP-13629 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13367) Support more types of Store in S3Native File System
[ https://issues.apache.org/jira/browse/HADOOP-13367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13367. - Resolution: Not A Problem Fix Version/s: 2.8.0 resolving as the Aliyun work addresses the specific need > Support more types of Store in S3Native File System > --- > > Key: HADOOP-13367 > URL: https://issues.apache.org/jira/browse/HADOOP-13367 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: liu chang >Priority: Minor > Fix For: 2.8.0 > > > There are a lot of object storage services whose protocol is similar to S3. > We could add more types of NativeFileSystemStore to support those services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13628) Support to retrieve specific property from configuration via REST API
Weiwei Yang created HADOOP-13628:
Summary: Support to retrieve specific property from configuration
via REST API
Key: HADOOP-13628
URL: https://issues.apache.org/jira/browse/HADOOP-13628
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 2.7.3
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Currently we can use rest API to retrieve all configuration properties per
daemon, but unable to get a specific property by name. This causes extra parse
work at client side when dealing with Hadoop configurations, and also it's
quite over head to send all configuration in a http response over network.
Propose to support following a {{name}} parameter in the http request,
{code}
curl --header "Accept:application/json"
http://${RM_HOST}/conf?name=yarn.nodemanager.aux-services
{"property"{"key":"yarn.resourcemanager.hostname","value":"${RM_HOST}","isFinal":false,"resource":"yarn-site.xml"}}
{code}
This change is fully backwards compatible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13627) Have an explicit KerberosAuthException for UGI to throw, text from public constants
Steve Loughran created HADOOP-13627:
---
Summary: Have an explicit KerberosAuthException for UGI to throw,
text from public constants
Key: HADOOP-13627
URL: https://issues.apache.org/jira/browse/HADOOP-13627
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.7.3
Reporter: Steve Loughran
Assignee: Xiao Chen
UGI creates simple IOEs on failure, making it impossible to catch them, ignore
them, have smart retry logic around them, etc.
# Have an explicit exception like {{KerberosAuthException extends IOException}}
to raise instead. We can't use {{AuthenticationException}} as that doesn't
extend IOE.
# move {{UGI}}, {{SecurityUtil}} and things related off simple IOEs and into
the new one
# review exceptions raised and consider if they can provide more information
# for the strings that get created, put them as public static constants, so
that tests can look for them explicitly —tests that don't break if the text is
changed.
# maybe, {{getUGIFromTicketCache}} to throw this rather than an RTE if no login
principals were found (it throws IOEs on login failures, after all)
# keep KDiag in sync with this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/ [Sep 19, 2016 9:03:06 AM] (varunsaxena) YARN-5577. [Atsv2] Document object passing in infofilters with an [Sep 19, 2016 9:08:01 AM] (jianhe) YARN-3141. Improve locks in [Sep 19, 2016 6:17:03 PM] (wang) HDFS-10868. Remove stray references to DFS_HDFS_BLOCKS_METADATA_ENABLED. [Sep 19, 2016 8:31:35 PM] (jlowe) YARN-5540. Scheduler spends too much time looking at empty priorities. [Sep 19, 2016 10:16:47 PM] (cnauroth) HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by [Sep 20, 2016 4:44:42 AM] (xiao) HDFS-10875. Optimize du -x to cache intermediate result. Contributed by -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestLargeBlockReport hadoop.hdfs.server.datanode.TestIncrementalBlockReports hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.TestApplicationClientProtocolOnHA hadoop.yarn.applications.distributedshell.TestDistributedShell cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-javadoc-root.txt [3.1M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [40K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/170/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
