[jira] [Resolved] (HADOOP-18215) Enhance WritableName to be able to return aliases for classes that use serializers
[
https://issues.apache.org/jira/browse/HADOOP-18215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault resolved HADOOP-18215.
Fix Version/s: 3.4.0
3.3.9
Release Note: If you have a SequenceFile with an old key or value class
which has been renamed, you can use WritableName.addName to add an alias class.
This functionality previously existed, but only worked for classes which extend
Writable. It now works for any class, notably key or value classes which use
io.serializations.
Resolution: Fixed
Pushed to trunk and branch-3.3. Thanks all!
> Enhance WritableName to be able to return aliases for classes that use
> serializers
> --
>
> Key: HADOOP-18215
> URL: https://issues.apache.org/jira/browse/HADOOP-18215
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> WritableName allows users shim in aliases for writables, in the case where a
> SequenceFile was written with a Writable class that has since been renamed or
> moved to another package. However, this requires that the aliased class
> extend Writable.
> Separately it's possible to configure jobs with keys and values which don't
> actually extend Writable. Instead they are meant to be
> serialized/deserialized using the serialization classes defined in
> {{io.serializations}} config.
> Unfortunately, the current implementation does not support these key/value
> classes. All we need to do to support this is remove the
> {{.asSubclass(Writable.class)}} as is already the case for the default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/ No changes -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.namenode.TestFsck hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.namenode.TestAuditLogger hadoop.hdfs.server.namenode.TestAuditLogs hadoop.streaming.TestSymLink hadoop.streaming.TestFileArgs hadoop.tools.mapred.TestCopyMapper hadoop.tools.mapred.TestUniformSizeInputFormat hadoop.tools.mapred.lib.TestDynamicInputFormat hadoop.fs.s3a.TestS3AAWSCredentialsProvider hadoop.fs.s3a.TestBucketConfiguration cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-compile-javac-root.txt [376K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/xml.txt [28K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [612K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-streaming.txt [48K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [80K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-workload.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-blockgen.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-archives.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/96/artifact/out/patch-unit-hadoop-tools_hadoop-rumen.txt [0]
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/940/ No changes ERROR: File 'out/email-report.txt' does not exist - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1138/ [Feb 15, 2023, 4:16:04 PM] (github) HDFS-16761. Namenode UI for Datanodes page not loading if any data node is down (#5390) [Feb 15, 2023, 6:28:42 PM] (github) HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads (#5274) [Feb 15, 2023, 10:38:41 PM] (github) YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) [Feb 15, 2023, 10:40:34 PM] (github) YARN-11425. [Federation] Router Supports SubClusterCleaner. (#5326) [Feb 16, 2023, 1:25:50 AM] (github) HDFS-16914. Add some logs for updateBlockForPipeline RPC. (#5381) -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:root Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] Failed junit tests : hadoop.hdfs.server.namenode.TestFSNamesystemLockReport hadoop.hdfs.server.namenode.TestAuditLogs hadoop.hdfs.server.namenode.TestAuditLogger hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.namenode.TestFsck hadoop.hdfs.TestRollingUpgrade hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestJobHistoryEvents hadoop.mapreduce.v2.hs.TestJobHistoryParsing hadoop.mapreduce.v2.hs.TestJobHistoryServer hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ipc.TestIPC hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.namenode.ha.TestPipelinesFailover hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.fs.viewfs.TestViewFileSystemHdfs hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.yarn.server.resourcemanager.TestClientRMService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/diff-compile-javac-root.txt [508K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-mvnsite-root.txt [620K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-javadoc-root.txt [68K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [244K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [1.8M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/939/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K]
