[
https://issues.apache.org/jira/browse/HADOOP-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18073220#comment-18073220
]
ASF GitHub Bot commented on HADOOP-1593:
----------------------------------------
bhattmanish98 commented on code in PR #8400:
URL: https://github.com/apache/hadoop/pull/8400#discussion_r3073562184
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBuffer.java:
##########
@@ -48,6 +54,13 @@ class ReadBuffer {
private boolean isLastByteConsumed = false;
private boolean isAnyByteConsumed = false;
private AtomicInteger refCount = new AtomicInteger(0);
+ private BufferType bufferType = BufferType.NORMAL;
+ // list of combined file ranges for vectored read.
+ private List<CombinedFileRange> vectoredUnits = new ArrayList<>();
+ // Allocator used for vectored fan-out; captured at queue time */
+ private IntFunction<ByteBuffer> allocator;
+ // Tracks whether fanOut has already been executed
+ private final AtomicInteger fanOutDone = new AtomicInteger(0);
Review Comment:
Would it be better to keep fanOutDone as AtomicBoolean instead of
AtomicInteger? We don't have to compare the value in isFanOutDone() in that
case.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBuffer.java:
##########
@@ -199,4 +212,67 @@ public void setAnyByteConsumed(boolean isAnyByteConsumed) {
public boolean isFullyConsumed() {
return isFirstByteConsumed() && isLastByteConsumed();
}
+
+ void addVectoredUnit(CombinedFileRange u) {
Review Comment:
Java doc for all the newly created methods
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java:
##########
@@ -285,5 +310,94 @@ protected void
testMimicFullUseAndAddFailedBuffer(ReadBuffer buf) {
completedReadList.add(buf);
}
+ /**
+ * Finds an existing {@link ReadBuffer} for the given stream whose buffered
+ * range covers the specified logical offset.
+ *
+ * <p>The search is performed in the read-ahead queue, in-progress list,
+ * and completed-read list, in that order.
+ *
+ * @param stream the {@link AbfsInputStream} associated with the read request
+ *
+ * @return a matching {@link ReadBuffer} if one exists, or {@code null}
otherwise
+ */
+ ReadBuffer findQueuedBuffer(final AbfsInputStream stream,
+ long requestedOffset) {
+ ReadBuffer buffer;
+ buffer = findInList(getReadAheadQueue(), stream, requestedOffset);
+ if (buffer != null) {
+ return buffer;
+ }
+ buffer = findInList(getInProgressList(), stream, requestedOffset);
+ if (buffer != null) {
+ return buffer;
+ }
+ return findInList(getCompletedReadList(), stream, requestedOffset);
+ }
+
+ /**
+ * Searches the given collection of {@link ReadBuffer}s for one that belongs
+ * to the specified stream and whose buffered range covers the given offset.
+ *
+ * @param buffers the collection of {@link ReadBuffer}s to search
+ * @param stream the {@link AbfsInputStream} associated with the read request
+ *
+ * @return the matching {@link ReadBuffer}, or {@code null} if none is found
+ */
+ ReadBuffer findInList(final Collection<ReadBuffer> buffers,
+ final AbfsInputStream stream, long requestedOffset) {
+ for (ReadBuffer buffer : buffers) {
+ if (buffer.getStream() == stream
Review Comment:
can buffer be null, if yes it will result into null pointer exception here.
> FsShell should work with paths in non-default FileSystem
> --------------------------------------------------------
>
> Key: HADOOP-1593
> URL: https://issues.apache.org/jira/browse/HADOOP-1593
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Reporter: Doug Cutting
> Assignee: Mahadev Konar
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.17.0
>
> Attachments: patch_1593.patch, patch_1593_1.patch
>
>
> If the default filesystem is, e.g., hdfs://foo:8888/, one should still be
> able to do 'bin/hadoop fs -ls hdfs://bar:9999/' or 'bin/hadoop fs -ls
> s3://cutting/foo'. Currently these generate a filesystem mismatch exception.
> This is because FsShell assumes that all paths are in the default
> FileSystem. Rather, the default filesystem should only be used for paths
> that do not specify a FileSystem. This would easily be accomplished by using
> Path#getFileSystem().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
