offermannu commented on a change in pull request #3232:
URL: https://github.com/apache/hbase/pull/3232#discussion_r635830006
##########
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
##########
@@ -135,9 +135,7 @@ public CellWritableComparable(Cell kv) {
@Override
public void write(DataOutput out) throws IOException {
- out.writeInt(PrivateCellUtil.estimatedSerializedSizeOfKey(kv));
- out.writeInt(0);
- PrivateCellUtil.writeFlatKey(kv, out);
+ KeyValueUtil.write(new KeyValue(kv), out);
Review comment:
I don't know the implementation enough to evaluate the alternatives. As
far as I can tell `CellWritableComparable.readFields` uses kind of standard
deserialization methods while `CellWritableComparable.write` implements its own
serialization methods.
But before we find out what is the best way we should ask ourselfs if this
feature is used at all. Since the code is broken for at least 4 years it seems
to me that nobody uses the large import feature.
I ran into this problem because we had to import 2,5 billion records into a
new HBase/Hadoop cluster. The direct import failed because the regionserver was
flooded with write requests and we got lots of `RegionTooBusyException`s. So we
tried the bulk import and ran into this bugs which I tried to fix.
Unfortunately it didn't work either because YARN produced too many cache files
for local sorting and the disks ran full - but that's another story.
Finally we found a way to get the direct import working again which means I
don't have a testing environment anymore.
IMHO if the feature is not used at all and there is no test available, it
would be smartest to delete the feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
