[
https://issues.apache.org/jira/browse/HADOOP-17945?focusedWorklogId=658351&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-658351
]
ASF GitHub Bot logged work on HADOOP-17945:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 30/Sep/21 13:40
Start Date: 30/Sep/21 13:40
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on a change in pull request
#3501:
URL: https://github.com/apache/hadoop/pull/3501#discussion_r719416755
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java
##########
@@ -229,30 +235,44 @@ public T fromInstance(T instance) throws IOException {
/**
* Load from a Hadoop filesystem.
- * There's a check for data availability after the file is open, by
- * raising an EOFException if stream.available == 0.
- * This allows for a meaningful exception without the round trip overhead
- * of a getFileStatus call before opening the file. It may be brittle
- * against an FS stream which doesn't return a value here, but the
- * standard filesystems all do.
- * JSON parsing and mapping problems
- * are converted to IOEs.
* @param fs filesystem
* @param path path
* @return a loaded object
- * @throws IOException IO or JSON parse problems
+ * @throws PathIOException JSON parse problem
+ * @throws IOException IO problems
*/
public T load(FileSystem fs, Path path) throws IOException {
- try (FSDataInputStream dataInputStream = fs.open(path)) {
- // throw an EOF exception if there is no data available.
- if (dataInputStream.available() == 0) {
- throw new EOFException("No data in " + path);
- }
+ return load(fs, path, null);
+ }
+
+ /**
+ * Load from a Hadoop filesystem.
+ * If a file status is supplied, it's passed in to the openFile()
+ * call so that FS implementations can optimize their opening.
+ * @param fs filesystem
+ * @param path path
+ * @param status status of the file to open.
+ * @return a loaded object
+ * @throws PathIOException JSON parse problem
+ * @throws EOFException file status references an empty file
+ * @throws IOException IO problems
+ */
+ public T load(FileSystem fs, Path path, @Nullable FileStatus status)
+ throws IOException {
+
+ if (status != null && status.getLen() == 0) {
+ throw new EOFException("No data in " + path);
Review comment:
What happens when we don't pass the FileStatus? Will it still throw EOF
or just a normal IOE?
##########
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJsonSerialization.java
##########
@@ -181,5 +197,23 @@ public void testFileSystemEmptyPath() throws Throwable {
}
}
+ /**
+ * 0 byte file through the load(path, status) API will fail with an
Review comment:
Should we add a test with 0 byte file but not passing the fileStatus.
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java
##########
@@ -229,30 +235,44 @@ public T fromInstance(T instance) throws IOException {
/**
* Load from a Hadoop filesystem.
- * There's a check for data availability after the file is open, by
- * raising an EOFException if stream.available == 0.
- * This allows for a meaningful exception without the round trip overhead
- * of a getFileStatus call before opening the file. It may be brittle
- * against an FS stream which doesn't return a value here, but the
- * standard filesystems all do.
- * JSON parsing and mapping problems
- * are converted to IOEs.
* @param fs filesystem
* @param path path
* @return a loaded object
- * @throws IOException IO or JSON parse problems
+ * @throws PathIOException JSON parse problem
+ * @throws IOException IO problems
*/
public T load(FileSystem fs, Path path) throws IOException {
- try (FSDataInputStream dataInputStream = fs.open(path)) {
- // throw an EOF exception if there is no data available.
- if (dataInputStream.available() == 0) {
Review comment:
So we are delegating the clients to decide whether to throw EOF or not
based on dataInputStream.available() method rather than throwing EOF ourselves.
I think this is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 658351)
Time Spent: 1h (was: 50m)
> JsonSerialization raises EOFException reading JSON data stored on google GCS
> ----------------------------------------------------------------------------
>
> Key: HADOOP-17945
> URL: https://issues.apache.org/jira/browse/HADOOP-17945
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The JsonSerialization<> load code doesn't work on gcs as it uses
> "stream.available()" to fail with a meaningful message if the stream is empty.
> But that method is meant to say how much data is available without blocking,
> something we actually get wrong ourselves. Google GCS team didn't get it
> wrong, so on a read(), if there's no local buffer, an EOFException is raised
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
