[jira] [Created] (HADOOP-13870) Incorrect behavior of copyFromLocalFile on implementations of FilterFileSystem
Joe Ellis created HADOOP-13870:
--
Summary: Incorrect behavior of copyFromLocalFile on
implementations of FilterFileSystem
Key: HADOOP-13870
URL: https://issues.apache.org/jira/browse/HADOOP-13870
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Joe Ellis
This may be an incorrect assumption on my part, but it was my belief that
overriding the create method on a FilterFileSystem was sufficient to intercept
all calls that would write data through the FileSystem. This is apparently not
true because calling copyFromLocalFile on the FilterFileSystem eventually
invokes the create method on the wrapped FileSystem. I would expect open ->
create -> copy(opened, created) to be functionally equivalent to copyFromLocal
when using the same input and output paths, but this is not the case.
{code:java}
import java.io.IOException;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import org.apache.commons.io.IOUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FilterFileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.util.Progressable;
import org.junit.Before;
import org.junit.Test;
public final class CopyFromLocalFileTest {
private static final Path DATA_PATH = new Path("file:///tmp/test_in");
private static final Path OUT_PATH = new Path("file:///tmp/test_out");
private FileSystem localFs;
private FileSystem wrappedFs;
@Before
public void before() throws IOException {
localFs = FileSystem.get(URI.create("file:///"), new Configuration());
wrappedFs = new FailingFileSystem(localFs);
FSDataOutputStream tmpFile = localFs.create(DATA_PATH);
byte[] bytes = "data".getBytes(StandardCharsets.UTF_8);
tmpFile.write(bytes);
tmpFile.close();
}
@Test
public void test_correct() throws IOException {
FSDataInputStream in = wrappedFs.open(DATA_PATH);
FSDataOutputStream out = wrappedFs.create(OUT_PATH);
IOUtils.copy(in, out);
}
@Test
public void test_incorrect() throws IOException {
wrappedFs.copyFromLocalFile(DATA_PATH, OUT_PATH);
}
private static final class FailingFileSystem extends FilterFileSystem {
public FailingFileSystem(FileSystem fs) {
super(fs);
}
@Override
public FSDataOutputStream create(Path f, FsPermission permission,
boolean overwrite, int bufferSize,
short replication, long blockSize, Progressable progress)
throws IOException {
throw new IOException("fail");
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13869) using HADOOP_USER_CLASSPATH_FIRST inconsistently
Fei Hui created HADOOP-13869: Summary: using HADOOP_USER_CLASSPATH_FIRST inconsistently Key: HADOOP-13869 URL: https://issues.apache.org/jira/browse/HADOOP-13869 Project: Hadoop Common Issue Type: Bug Components: scripts Reporter: Fei Hui I find HADOOP_USER_CLASSPATH_FIRST is used inconsistently. I know it doesn't mattter because it affects classpath once HADOOP_USER_CLASSPATH_FIRST is not empty BUT Maybe it's better that using HADOOP_USER_CLASSPATH_FIRST uniformly -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/178/ [Dec 5, 2016 4:44:40 PM] (weichiu) HDFS-11181. Fuse wrapper has a typo. Contributed by Wei-Chiu Chuang. [Dec 5, 2016 5:35:17 PM] (xiao) HADOOP-13847. KMSWebApp should close KeyProviderCryptoExtension. [Dec 5, 2016 6:48:25 PM] (weichiu) HDFS-11201. Spelling errors in the logging, help, assertions and [Dec 5, 2016 6:54:43 PM] (weichiu) Revert "HDFS-11201. Spelling errors in the logging, help, assertions and [Dec 5, 2016 7:39:34 PM] (jianhe) YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. [Dec 5, 2016 7:48:58 PM] (liuml07) HDFS-11094. Send back HAState along with NamespaceInfo during a [Dec 5, 2016 8:04:07 PM] (liuml07) HADOOP-13675. Bug in return value for delete() calls in WASB. [Dec 5, 2016 8:44:20 PM] (xyao) Revert "HADOOP-10930. Refactor: Wrap Datanode IO related operations. [Dec 6, 2016 1:23:38 AM] (naganarasimha_gr) YARN-5921. Incorrect synchronization in [Dec 6, 2016 1:36:00 AM] (xiao) HADOOP-13864. KMS should not require truststore password. Contributed by [Dec 6, 2016 2:13:53 AM] (wang) HDFS-10581. Hide redundant table on NameNode WebUI when no nodes are [Dec 6, 2016 5:01:47 AM] (aajisaka) HADOOP-13835. Move Google Test Framework code from mapreduce to [Dec 6, 2016 7:09:35 AM] (wang) Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST [Dec 6, 2016 7:18:18 AM] (wang) HADOOP-13861. Spelling errors in logging and exceptions for code. -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestEncryptedTransfer hadoop.hdfs.server.datanode.TestDataNodeLifeline hadoop.hdfs.TestDFSShellGenericOptions hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/178/artifact/out/patch-compile-root.txt [172K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/178/artifact/out/patch-compile-root.txt [172K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/178/artifact/out/patch-compile-root.txt [172K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/178/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [204K]
[jira] [Created] (HADOOP-13868) Configure multi-part copies and uploads separately
Sean Mackrory created HADOOP-13868: -- Summary: Configure multi-part copies and uploads separately Key: HADOOP-13868 URL: https://issues.apache.org/jira/browse/HADOOP-13868 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory I've been looking at a big performance regression when writing to S3 from Spark that appears to have been introduced with HADOOP-12891. In the Amazon SDK, the default threshold for multi-part copies is 320x the threshold for multi-part uploads (and the block size is 20x bigger), so I don't think it's wise for us I did some quick tests and it seems to me the sweet spot when multi-part copies start being faster is around 512MB. It wasn't as significant, but using 104857600 (Amazon's default) for the blocksize was also slightly better. I propose we do the following, although they're independent. (1) Split the configuration. Ideally, I'd like to have fs.s3a.multipart.copy.threshold and fs.s3a.multipart.upload.threshold (and corresponding properties for the block size). But then there's the question of what to do with the existing fs.s3a.multipart.* properties. Deprecation? Leave it as a short-hand for configuring both (that's overridden by the more specific properties?). (2) Consider increasing the default values. In my tests, 256 MB seemed to be where multipart uploads came into their own, and 512 MB was where multipart copies started outperforming the alternative. Would be interested to hear what other people have seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12854) Move to netty 4.1.x release
[ https://issues.apache.org/jira/browse/HADOOP-12854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HADOOP-12854. - Resolution: Duplicate Dup of HADOOP-13866 > Move to netty 4.1.x release > --- > > Key: HADOOP-12854 > URL: https://issues.apache.org/jira/browse/HADOOP-12854 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > Netty is getting close to having a final release of a 4.1 netty-all artifact; > HDFS currently pulls in 4.1.0.Beta5 > Once a 4.1 release is out, switch to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[ https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HADOOP-13866: - > Upgrade netty-all to 4.1.1.Final > > > Key: HADOOP-13866 > URL: https://issues.apache.org/jira/browse/HADOOP-13866 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch > > > netty-all 4.1.1.Final is stable release which we should upgrade to. > See bottom of HADOOP-12927 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/ [Dec 5, 2016 4:44:40 PM] (weichiu) HDFS-11181. Fuse wrapper has a typo. Contributed by Wei-Chiu Chuang. [Dec 5, 2016 5:35:17 PM] (xiao) HADOOP-13847. KMSWebApp should close KeyProviderCryptoExtension. [Dec 5, 2016 6:48:25 PM] (weichiu) HDFS-11201. Spelling errors in the logging, help, assertions and [Dec 5, 2016 6:54:43 PM] (weichiu) Revert "HDFS-11201. Spelling errors in the logging, help, assertions and [Dec 5, 2016 7:39:34 PM] (jianhe) YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. [Dec 5, 2016 7:48:58 PM] (liuml07) HDFS-11094. Send back HAState along with NamespaceInfo during a [Dec 5, 2016 8:04:07 PM] (liuml07) HADOOP-13675. Bug in return value for delete() calls in WASB. [Dec 5, 2016 8:44:20 PM] (xyao) Revert "HADOOP-10930. Refactor: Wrap Datanode IO related operations. [Dec 6, 2016 1:23:38 AM] (naganarasimha_gr) YARN-5921. Incorrect synchronization in [Dec 6, 2016 1:36:00 AM] (xiao) HADOOP-13864. KMS should not require truststore password. Contributed by [Dec 6, 2016 2:13:53 AM] (wang) HDFS-10581. Hide redundant table on NameNode WebUI when no nodes are [Dec 6, 2016 5:01:47 AM] (aajisaka) HADOOP-13835. Move Google Test Framework code from mapreduce to -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestRollingUpgrade hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-patch-shellcheck.txt [28K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/whitespace-tabs.txt [1.3M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [152K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/247/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
