Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/ [May 18, 2017 2:44:39 PM] (stevel) HADOOP-11572. s3a delete() operation fails during a concurrent delete of [May 18, 2017 5:19:42 PM] (sunilg) YARN-6560. SLS doesn't honor node total resource specified in [May 18, 2017 10:35:22 PM] (yzhang) HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar [May 19, 2017 4:27:46 AM] (cdouglas) HADOOP-11869. Suppress ParameterNumber checkstyle violations for -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.server.namenode.ha.TestBootstrapStandby hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.namenode.TestReconstructStripedBlocks hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestSafeModeWithStripedFile hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestAMRMClient hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.yarn.sls.TestSLSRunner Timed out junit tests : org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-mvninstall-root.txt [492K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [1.1M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/319/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K]
[jira] [Created] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances
Wei-Chiu Chuang created HADOOP-14441:
Summary: LoadBalancingKMSClientProvider#addDelegationTokens should
add delegation tokens from all KMS instances
Key: HADOOP-14441
URL: https://issues.apache.org/jira/browse/HADOOP-14441
Project: Hadoop Common
Issue Type: Bug
Components: kms
Affects Versions: 2.7.0
Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang
LoadBalancingKMSClientProvider only gets delegation token from one KMS
instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for
{{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
{quote}
/**
* The implementer of this class will take a renewer and add all
* delegation tokens associated with the renewer to the
* Credentials object if it is not already present,
...
**/
{quote}
This bug doesn't pop up very often, because HDFS clients such as MapReduce
unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
We have a custom client that accesses HDFS/KMS-HA using delegation token, and
we were puzzled why it always throws "Failed to find any Kerberos tgt"
exceptions talking to one KMS but not the other. Turns out that client couldn't
talk to the KMS because {{FileSystem#addDelegationTokens}} only gets one KMS
delegation token at a time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/408/ [May 18, 2017 4:59:34 AM] (vvasudev) Validate docker image name before launching container. [May 18, 2017 5:07:16 AM] (vvasudev) Sanitize arguments before launching Docker containers. [May 18, 2017 2:44:39 PM] (stevel) HADOOP-11572. s3a delete() operation fails during a concurrent delete of [May 18, 2017 5:19:42 PM] (sunilg) YARN-6560. SLS doesn't honor node total resource specified in [May 18, 2017 10:35:22 PM] (yzhang) HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar [May 19, 2017 4:27:46 AM] (cdouglas) HADOOP-11869. Suppress ParameterNumber checkstyle violations for -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-common-project/hadoop-minikdc Possible null pointer dereference in org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:[line 368] FindBugs : module:hadoop-common-project/hadoop-auth org.apache.hadoop.security.authentication.server.MultiSchemeAuthenticationHandler.authenticate(HttpServletRequest, HttpServletResponse) makes inefficient use of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:[line 192] FindBugs : module:hadoop-common-project/hadoop-common org.apache.hadoop.crypto.CipherSuite.setUnknownValue(int) unconditionally sets the field unknownValue At CipherSuite.java:unknownValue At CipherSuite.java:[line 44] org.apache.hadoop.crypto.CryptoProtocolVersion.setUnknownValue(int) unconditionally sets the field unknownValue At CryptoProtocolVersion.java:unknownValue At CryptoProtocolVersion.java:[line 67] Possible null pointer dereference in org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:[line 118] Possible null pointer dereference in org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, File, Path, File) due to return value of called method Dereferenced at RawLocalFileSystem.java:org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, File, Path, File) due to return value of called method Dereferenced at RawLocalFileSystem.java:[line 387] Return value of org.apache.hadoop.fs.permission.FsAction.or(FsAction) ignored, but method has no side effect At FTPFileSystem.java:but method has no side effect At FTPFileSystem.java:[line 421] Useless condition:lazyPersist == true at this point At CommandWithDestination.java:[line 502] org.apache.hadoop.io.DoubleWritable.compareTo(DoubleWritable) incorrectly handles double value At DoubleWritable.java: At DoubleWritable.java:[line 78] org.apache.hadoop.io.DoubleWritable$Comparator.compare(byte[], int, int, byte[], int, int) incorrectly handles double value At DoubleWritable.java:int) incorrectly handles double value At DoubleWritable.java:[line 97] org.apache.hadoop.io.FloatWritable.compareTo(FloatWritable) incorrectly handles float value At FloatWritable.java: At FloatWritable.java:[line 71] org.apache.hadoop.io.FloatWritable$Comparator.compare(byte[], int, int, byte[], int, int) incorrectly handles float value At FloatWritable.java:int) incorrectly handles float value At FloatWritable.java:[line 89] Possible null pointer dereference in org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return value of called method Dereferenced at IOUtils.java:org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return value of called method Dereferenced at IOUtils.java:[line 350] org.apache.hadoop.io.erasurecode.ECSchema.toString() makes inefficient use of keySet iterator instead of entrySet iterator At ECSchema.java:keySet iterator instead of entrySet iterator At ECSchema.java:[line 193] Possible bad parsing of shift operation in org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:operation in org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:[line 398] org.apache.hadoop.metrics2.lib.DefaultMetricsFactory.setInstance(MutableMetricsFactory) unconditionally sets the field mmfImpl At DefaultMetricsFactory.java:mmfImpl At
[jira] [Created] (HADOOP-14440) Add metrics for connections dropped
Eric Badger created HADOOP-14440: Summary: Add metrics for connections dropped Key: HADOOP-14440 URL: https://issues.apache.org/jira/browse/HADOOP-14440 Project: Hadoop Common Issue Type: Improvement Reporter: Eric Badger Assignee: Eric Badger Will be useful to figure out when the NN is getting overloaded with more connections than it can handle -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14387) new Configuration().get() fails if core-site.xml isn't on the classpath
[
https://issues.apache.org/jira/browse/HADOOP-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14387.
-
Resolution: Cannot Reproduce
> new Configuration().get() fails if core-site.xml isn't on the classpath
> ---
>
> Key: HADOOP-14387
> URL: https://issues.apache.org/jira/browse/HADOOP-14387
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 3.0.0-alpha3
> Environment: test run in downstream project with no core-site in
> test/resources
>Reporter: Steve Loughran
>Assignee: Jonathan Eagles
>Priority: Blocker
> Attachments: HADOOP-14387.1.patch
>
>
> If you try to create a config via {{new Configuration()}} and there isn't a
> {{core-site.xml}} on the CP, you get a stack trace. Previously it'd just skip
> the failure to load.
> This is a regression which breaks downstream apps that don't need a core-site
> to run, but do want to load core-default
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some mapping code
Steve Loughran created HADOOP-14439:
---
Summary: regression: secret stripping from S3x URIs breaks some
mapping code
Key: HADOOP-14439
URL: https://issues.apache.org/jira/browse/HADOOP-14439
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 2.8.0
Environment: Spark 2.1
Reporter: Steve Loughran
Priority: Minor
Surfaced in SPARK-20799
Spark is listing the contents of a path with getFileStatus(path), then looking
up the path value doing a lookup of the contents.
Apparently the lookup is failing to find files if you have a secret in the key,
{{s3a://key:secret@bucket/path}}.
Presumably this is because the stripped values aren't matching.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14438) Make ADLS doc of setting up client key up to date
Mingliang Liu created HADOOP-14438:
--
Summary: Make ADLS doc of setting up client key up to date
Key: HADOOP-14438
URL: https://issues.apache.org/jira/browse/HADOOP-14438
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/adl
Reporter: Mingliang Liu
In the doc {{hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md}},
we have such a statement:
{code:title=Note down the properties you will need to auth}
...
- Resource: Always https://management.core.windows.net/ , for all customers
{code}
Is the {{Resource}} useful here? It seems not necessary to me.
{code:title=Adding the service principal to your ADL Account}
- ...
- Select Users under Settings
...
{code}
According to the portal, it should be "Access control (IAM)" under "Settings"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
