Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/903/ No changes -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine Unread field:FSBasedSubmarineStorageImpl.java:[line 39] Found reliance on default encoding in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component):in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component): new java.io.FileWriter(File) At YarnServiceJobSubmitter.java:[line 192] org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component) may fail to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:[line 192] is not discharged org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String, int, String) concatenates strings using + in a loop At YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 72] Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/diff-compile-javac-root.txt [304K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/diff-checkstyle-root.txt [17M] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine-warnings.html [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/890/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K]
[jira] [Created] (HADOOP-15780) S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets
Aaron Fabbri created HADOOP-15780: - Summary: S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets Key: HADOOP-15780 URL: https://issues.apache.org/jira/browse/HADOOP-15780 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.2.0 Reporter: Aaron Fabbri Our general policy for S3Guard is this: All modifiers of a bucket that is configured for use with S3Guard, must use S3Guard. Otherwise, the MetadataStore will not be properly updated as the S3 bucket changes and problems will arise. There are limited circumstances in which may be safe to have an external (non-s3guard) process writing data. There are also scenarios where it definitely breaks things. I think we should start by documenting the cases that this works / does not work for. After we've enumerated that, we can suggest enhancements as needed to make this sort of configuration easier to use. To get the ball rolling, some things that do not work: - Deleting a path *p* with S3Guard, then writing a new file at path *p* without S3guard (will still have delete marker in S3Guard, making the file appear to be deleted but still visible in S3 due to false "eventual consistency") (as [~ste...@apache.org] and I have discussed) - When fs.s3a.metadatastore.authoritative is true, adding files to directories without S3Guard, then listing with S3Guard may exclude externally-written files from listings. (Note, there are also S3A interop issues with other non-S3A clients even without S3Guard, due to the unique way S3A interprets empty directory markers). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15779) S3guard: add inconsistency detection metrics
Aaron Fabbri created HADOOP-15779: - Summary: S3guard: add inconsistency detection metrics Key: HADOOP-15779 URL: https://issues.apache.org/jira/browse/HADOOP-15779 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.2.0 Reporter: Aaron Fabbri S3Guard uses a trailing log of metadata changes made to an S3 bucket to add consistency to the eventually-consistent AWS S3 service. We should add some metrics that are incremented when we detect inconsistency (eventual consistency) in S3. I'm thinking at least two counters: (1) getFileStatus() (HEAD) inconsistency detected, and (2) listing inconsistency detected. We may want to further separate into categories (present / not present etc.) This is related to Auth. Mode and TTL work that is ongoing, so let me outline how I think this should all evolve: This should happen after HADOOP-15621 (TTL for dynamo MetadataStore), since that will change *when* we query both S3 and the MetadataStore (e.g. Dynamo) for metadata. There I suggest that: # Prune time is different than TTL. Prune time: "how long until inconsistency is no longer a problem" . TTL time "how long a MetadataStore entry is considered authoritative before refresh" # Prune expired: delete entries (when hadoop CLI prune command is run). TTL Expired: entries become non-authoritative. # Prune implemented in each MetadataStore, but TTL filtering happens in S3A. Once we have this, S3A will be consulting both S3 and MetadataStore depending on configuration and/or age of the entry in the MetadataStore. Today HEAD/getFileStatus() is always short-circuit (skips S3 if MetadataStore returns results). I think S3A should consult both when TTL is stale, and invoke a callback on inconsistency that increments the new metrics. For listing, we already are comparing both sources of truth (except when S3A auth mode is on and a directory is marked authoritative in MS), so it would be pretty simple to invoke a callback on inconsistency and bump some metrics. Comments / suggestions / questions welcomed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15778) ABFS: Fix client side throttling for read
Sneha Varma created HADOOP-15778: Summary: ABFS: Fix client side throttling for read Key: HADOOP-15778 URL: https://issues.apache.org/jira/browse/HADOOP-15778 Project: Hadoop Common Issue Type: Sub-task Reporter: Sneha Varma 1. The content length for ReadFile in updateMetrics of AbfsClientThrottlingIntercept is incorrect for cases when the request fails. It is currently equal to the number of bytes that are read whereas it should be equal to the number of bytes requested. 2. sendingRequest of AbfsClientThrottlingIntercept at AbfsRestOperation should be called irrespective of whether the request has body. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15777) Add support of getSnapshotDiff for all the FS inherit from FileSystem
Xun REN created HADOOP-15777: Summary: Add support of getSnapshotDiff for all the FS inherit from FileSystem Key: HADOOP-15777 URL: https://issues.apache.org/jira/browse/HADOOP-15777 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 3.3.0 Reporter: Xun REN As proposed by [~jojochuang] in the Jira HDFS-13916, I create this Jira which depends on HADOOP-15691 in order to add support of getSnapshotDiff for all the File Systems inherit from the parent class FileSystem. This method is used to get difference between two snapshots, which is mostly used to do DistCP with "-diff" option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15776) The key length used in "KeyGenerator.init()" should be configurable
zhoushulin created HADOOP-15776:
---
Summary: The key length used in "KeyGenerator.init()" should be
configurable
Key: HADOOP-15776
URL: https://issues.apache.org/jira/browse/HADOOP-15776
Project: Hadoop Common
Issue Type: Bug
Components: conf
Reporter: zhoushulin
In mapreduce, the key length used in KeyGenerator.init() is configured with
configuration option "mapreduce.job.encrypted-intermediate-data-key-size-bits"
as follows:
{code:java|title=/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java}
protected void initJobCredentialsAndUGI(Configuration conf) {
...
int keyLen = conf.getInt(
MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS,
MRJobConfig
.DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS);
KeyGenerator keyGen =
KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO);
keyGen.init(keyLen);
encryptedSpillKey = keyGen.generateKey().getEncoded();
...
}
{code}
The same usage is also in mapred as follows:
{code:java|title=/org/apache/hadoop/mapred/LocalJobRunner.java}
public Job(JobID jobid, String jobSubmitDir) throws IOException { ...
int keyLen = conf.getInt(
MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS,
MRJobConfig
.DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS);
KeyGenerator keyGen =
KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO);
keyGen.init(keyLen);
...
}
{code}
Also, in hadoop-common, there is a configration option
"hadoop.security.key.default.bitlength", it is used in KeyProvider.java to
initiate KeyGenerator as follows:
{code:java|title=/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java}
public static final String HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_KEY =
"hadoop.security.key.default.bitlength";
/** Defalt value for HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_KEY. */
public static final int HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_DEFAULT = 128;
{code}
{code:java|title=/org/apache/hadoop/crypto/key/KeyProvider.java}
public Options(Configuration conf) {
cipher = conf.get(DEFAULT_CIPHER_NAME, DEFAULT_CIPHER);
bitLength = conf.getInt(DEFAULT_BITLENGTH_NAME, DEFAULT_BITLENGTH);
}
public KeyVersion createKey(String name, Options options)
throws NoSuchAlgorithmException, IOException {
byte[] material = generateKey(options.getBitLength(), options.getCipher());
return createKey(name, material, options);
}
protected byte[] generateKey(int size, String algorithm)
throws NoSuchAlgorithmException {
algorithm = getAlgorithm(algorithm);
KeyGenerator keyGenerator = KeyGenerator.getInstance(algorithm);
keyGenerator.init(size);
byte[] key = keyGenerator.generateKey().getEncoded();
return key;
}
...
}
{code}
However, in other two usage of "KeyGenerator.init()" in mapreduce and
hadoop-common, the key length is hard-coded as 64. Also, in the evolving
history, this value is changed from "20" to "64". So, in the perspective of
flexibility and security, these two hard coded value in "KeyGenerator.init()"
should be configurable.
{code:java|title=/org/apache/hadoop/mapreduce/JobSubmitter.java}
class JobSubmitter {
...
private static final int SHUFFLE_KEY_LENGTH = 64;
...
JobStatus submitJobInternal(Job job, Cluster cluster)
throws ClassNotFoundException, InterruptedException, IOException {
...
keyGen = KeyGenerator.getInstance(SHUFFLE_KEYGEN_ALGORITHM);
keyGen.init(SHUFFLE_KEY_LENGTH);
...
}
...
}
{code}
{code:java|title=/org/apache/hadoop/security/token/SecretManager.java}
public abstract class SecretManager {
...
private static final int KEY_LENGTH = 64;
...
private final KeyGenerator keyGen;
{
try {
keyGen = KeyGenerator.getInstance(DEFAULT_HMAC_ALGORITHM);
keyGen.init(KEY_LENGTH);
} catch (NoSuchAlgorithmException nsa) {
throw new IllegalArgumentException("Can't find " + DEFAULT_HMAC_ALGORITHM
+
" algorithm.");
}
}
...
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/ [Sep 19, 2018 3:05:25 AM] (xyao) HDDS-488. Handle chill mode exception from SCM in OzoneManager. [Sep 19, 2018 3:23:50 AM] (xiao) HDFS-13833. Improve BlockPlacementPolicyDefault's consider load logic. [Sep 19, 2018 4:18:15 AM] (arp) HDDS-497. Suppress license warnings for error log files. Contributed by [Sep 19, 2018 6:21:29 AM] (xyao) HDDS-506. Fields in AllocateScmBlockResponseProto should be optional. [Sep 19, 2018 10:12:20 AM] (weichiu) HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" [Sep 19, 2018 11:31:07 AM] (wwei) YARN-8771. CapacityScheduler fails to unreserve when cluster resource [Sep 19, 2018 1:22:08 PM] (nanda) HDDS-476. Add Pipeline reports to make pipeline active on SCM restart. [Sep 19, 2018 2:16:25 PM] (nanda) HDDS-502. Exception in OM startup when running unit tests. Contributed [Sep 19, 2018 3:11:05 PM] (elek) HDDS-458. numberofKeys is 0 for all containers even when keys are [Sep 19, 2018 4:13:44 PM] (nanda) HDDS-461. Container remains in CLOSING state in SCM forever. Contributed [Sep 19, 2018 4:30:25 PM] (inigoiri) HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by [Sep 19, 2018 4:48:55 PM] (sunilg) YARN-8757. [Submarine] Add Tensorboard component when --tensorboard is [Sep 19, 2018 5:16:11 PM] (eyang) YARN-8791. Trim docker inspect output for line feed for STOPSIGNAL [Sep 19, 2018 6:21:50 PM] (nanda) HDDS-460. Replication manager failed to import container data. [Sep 19, 2018 7:52:18 PM] (nanda) HDDS-507. EventQueue should be shutdown on SCM shutdown. Contributed by [Sep 19, 2018 8:00:30 PM] (inigoiri) HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when [Sep 19, 2018 8:03:19 PM] (aengineer) HDDS-509. TestStorageContainerManager is flaky. Contributed by Xiaoyu [Sep 19, 2018 8:22:37 PM] (cliang) HADOOP-15726. Create utility to limit frequency of log statements. [Sep 19, 2018 8:48:27 PM] (arp) HADOOP-15772. Remove the 'Path ... should be specified as a URI' [Sep 19, 2018 9:35:29 PM] (bharat) HDDS-513. Check if the EventQueue is not closed before executing [Sep 19, 2018 9:44:51 PM] (jlowe) YARN-8784. DockerLinuxContainerRuntime prevents access to distributed -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine Found reliance on default encoding in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component):in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component): new java.io.FileWriter(File) At YarnServiceJobSubmitter.java:[line 208] org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String, int, String) concatenates strings using + in a loop At YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 92] Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.client.api.impl.TestAMRMProxy cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-compile-javac-root.txt [300K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/902/artifact/out/diff-patch-shelldocs.txt [16K] whitespace:
[jira] [Created] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
Akira Ajisaka created HADOOP-15775: -- Summary: [JDK9] Add "--add-modules java.activation" to java option in unit tests Key: HADOOP-15775 URL: https://issues.apache.org/jira/browse/HADOOP-15775 Project: Hadoop Common Issue Type: Sub-task Components: test Reporter: Akira Ajisaka Many unit tests fail due to missing java.activation module. We need to configure maven surefire plugin to add "--add-modules java.activation" option to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
