[
https://issues.apache.org/jira/browse/HADOOP-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096861#comment-18096861
]
ASF GitHub Bot commented on HADOOP-19932:
-----------------------------------------
Copilot commented on code in PR #8585:
URL: https://github.com/apache/hadoop/pull/8585#discussion_r3597412987
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestTextCommand.java:
##########
@@ -513,22 +514,20 @@ private static byte[] generateEmptyAvroBinaryData() {
private static void createEmptySequenceFile(String fileName, Configuration
conf)
throws IOException {
- conf.set("io.serializations",
"org.apache.hadoop.io.serializer.JavaSerialization");
Path path = new Path(fileName);
SequenceFile.Writer writer = SequenceFile.createWriter(conf,
SequenceFile.Writer.file(path),
- SequenceFile.Writer.keyClass(String.class),
SequenceFile.Writer.valueClass(String.class));
+ SequenceFile.Writer.keyClass(Text.class),
SequenceFile.Writer.valueClass(Text.class));
writer.close();
}
private static void createNonWritableSequenceFile(String fileName,
Configuration conf)
throws IOException {
- conf.set("io.serializations",
"org.apache.hadoop.io.serializer.JavaSerialization");
Path path = new Path(fileName);
try (SequenceFile.Writer writer = SequenceFile.createWriter(conf,
- SequenceFile.Writer.file(path),
SequenceFile.Writer.keyClass(String.class),
- SequenceFile.Writer.valueClass(String.class))) {
- writer.append("Key1", "Value1");
- writer.append("Key2", "Value2");
+ SequenceFile.Writer.file(path),
SequenceFile.Writer.keyClass(Text.class),
+ SequenceFile.Writer.valueClass(Text.class))) {
+ writer.append(new Text("Key1"), new Text("Value1"));
+ writer.append(new Text("Key2"), new Text("Value2"));
}
Review Comment:
`createNonWritableSequenceFile` now writes `Text` keys/values (i.e.,
Writable types), so it no longer creates a “non-writable” SequenceFile as the
helper name, `SEQUENCE_FILENAME` ("NonWritableSequenceFile"), and related test
names imply. This makes the tests misleading and likely drops coverage for the
original non-Writable SequenceFile path.
Consider either (a) renaming the helper/test/file constants to reflect that
this is a normal Writable SequenceFile, or (b) reworking the helper to actually
generate a SequenceFile whose key/value types are not Writable (using an
alternative supported serialization).
> Prune obsolete classes in org.apache.hadoop.io.serializer
> ---------------------------------------------------------
>
> Key: HADOOP-19932
> URL: https://issues.apache.org/jira/browse/HADOOP-19932
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 3.5.1, 3.6.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
>
> There's a collection of classes in org.apache.hadoop.io.serializer which try
> to bridge between Writable and Serializable.
> None of this code has been touched for a decade and may not be used; get rid
> of the cruft.
> Although tagged LimitedPrivate MapReduce, it may get used downstream in
> places like spark; which may notice if cut.
> so prune carefully.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
