Repository: hadoop Updated Branches: refs/heads/branch-2.9 32a88442d -> b756beb67
http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md b/hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md deleted file mode 100644 index 62e6505..0000000 --- a/hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md +++ /dev/null @@ -1,294 +0,0 @@ -<!--- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -# Hadoop-Aliyun module: Integration with Aliyun Web Services - -<!-- MACRO{toc|fromDepth=0|toDepth=5} --> - -## Overview - -The `hadoop-aliyun` module provides support for Aliyun integration with -[Aliyun Object Storage Service (Aliyun OSS)](https://www.aliyun.com/product/oss). -The generated JAR file, `hadoop-aliyun.jar` also declares a transitive -dependency on all external artifacts which are needed for this support â enabling -downstream applications to easily use this support. - -To make it part of Apache Hadoop's default classpath, simply make sure -that HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-aliyun' in the list. - -### Features - -* Read and write data stored in Aliyun OSS. -* Present a hierarchical file system view by implementing the standard Hadoop -[`FileSystem`](../api/org/apache/hadoop/fs/FileSystem.html) interface. -* Can act as a source of data in a MapReduce job, or a sink. - -### Warning #1: Object Stores are not filesystems. - -Aliyun OSS is an example of "an object store". In order to achieve scalability -and especially high availability, Aliyun OSS has relaxed some of the constraints -which classic "POSIX" filesystems promise. - - - -Specifically - -1. Atomic operations: `delete()` and `rename()` are implemented by recursive -file-by-file operations. They take time at least proportional to the number of files, -during which time partial updates may be visible. `delete()` and `rename()` -can not guarantee atomicity. If the operations are interrupted, the filesystem -is left in an intermediate state. -2. File owner and group are persisted, but the permissions model is not enforced. -Authorization occurs at the level of the entire Aliyun account via -[Aliyun Resource Access Management (Aliyun RAM)](https://www.aliyun.com/product/ram). -3. Directory last access time is not tracked. -4. The append operation is not supported. - -### Warning #2: Directory last access time is not tracked, -features of Hadoop relying on this can have unexpected behaviour. E.g. the -AggregatedLogDeletionService of YARN will not remove the appropriate logfiles. - -### Warning #3: Your Aliyun credentials are valuable - -Your Aliyun credentials not only pay for services, they offer read and write -access to the data. Anyone with the account can not only read your datasets -âthey can delete them. - -Do not inadvertently share these credentials through means such as -1. Checking in to SCM any configuration files containing the secrets. -2. Logging them to a console, as they invariably end up being seen. -3. Defining filesystem URIs with the credentials in the URL, such as -`oss://accessKeyId:accessKeySecret@directory/file`. They will end up in -logs and error messages. -4. Including the secrets in bug reports. - -If you do any of these: change your credentials immediately! - -### Warning #4: The Aliyun OSS client provided by Aliyun E-MapReduce are different from this implementation - -Specifically: on Aliyun E-MapReduce, `oss://` is also supported but with -a different implementation. If you are using Aliyun E-MapReduce, -follow these instructions âand be aware that all issues related to Aliyun -OSS integration in E-MapReduce can only be addressed by Aliyun themselves: -please raise your issues with them. - -## OSS - -### Authentication properties - - <property> - <name>fs.oss.accessKeyId</name> - <description>Aliyun access key ID</description> - </property> - - <property> - <name>fs.oss.accessKeySecret</name> - <description>Aliyun access key secret</description> - </property> - - <property> - <name>fs.oss.credentials.provider</name> - <description> - Class name of a credentials provider that implements - com.aliyun.oss.common.auth.CredentialsProvider. Omit if using access/secret keys - or another authentication mechanism. The specified class must provide an - accessible constructor accepting java.net.URI and - org.apache.hadoop.conf.Configuration, or an accessible default constructor. - </description> - </property> - -### Other properties - - <property> - <name>fs.oss.endpoint</name> - <description>Aliyun OSS endpoint to connect to. An up-to-date list is - provided in the Aliyun OSS Documentation. - </description> - </property> - - <property> - <name>fs.oss.proxy.host</name> - <description>Hostname of the (optinal) proxy server for Aliyun OSS connection</description> - </property> - - <property> - <name>fs.oss.proxy.port</name> - <description>Proxy server port</description> - </property> - - <property> - <name>fs.oss.proxy.username</name> - <description>Username for authenticating with proxy server</description> - </property> - - <property> - <name>fs.oss.proxy.password</name> - <description>Password for authenticating with proxy server.</description> - </property> - - <property> - <name>fs.oss.proxy.domain</name> - <description>Domain for authenticating with proxy server.</description> - </property> - - <property> - <name>fs.oss.proxy.workstation</name> - <description>Workstation for authenticating with proxy server.</description> - </property> - - <property> - <name>fs.oss.attempts.maximum</name> - <value>20</value> - <description>How many times we should retry commands on transient errors.</description> - </property> - - <property> - <name>fs.oss.connection.establish.timeout</name> - <value>50000</value> - <description>Connection setup timeout in milliseconds.</description> - </property> - - <property> - <name>fs.oss.connection.timeout</name> - <value>200000</value> - <description>Socket connection timeout in milliseconds.</description> - </property> - - <property> - <name>fs.oss.paging.maximum</name> - <value>1000</value> - <description>How many keys to request from Aliyun OSS when doing directory listings at a time. - </description> - </property> - - <property> - <name>fs.oss.multipart.upload.size</name> - <value>10485760</value> - <description>Size of each of multipart pieces in bytes.</description> - </property> - - <property> - <name>fs.oss.multipart.upload.threshold</name> - <value>20971520</value> - <description>Minimum size in bytes before we start a multipart uploads or copy.</description> - </property> - - <property> - <name>fs.oss.multipart.download.size</name> - <value>102400/value> - <description>Size in bytes in each request from ALiyun OSS.</description> - </property> - - <property> - <name>fs.oss.buffer.dir</name> - <description>Comma separated list of directories to buffer OSS data before uploading to Aliyun OSS</description> - </property> - - <property> - <name>fs.oss.acl.default</name> - <value></vaule> - <description>Set a canned ACL for bucket. Value may be private, public-read, public-read-write. - </description> - </property> - - <property> - <name>fs.oss.server-side-encryption-algorithm</name> - <value></vaule> - <description>Specify a server-side encryption algorithm for oss: file system. - Unset by default, and the only other currently allowable value is AES256. - </description> - </property> - - <property> - <name>fs.oss.connection.maximum</name> - <value>32</value> - <description>Number of simultaneous connections to oss.</description> - </property> - - <property> - <name>fs.oss.connection.secure.enabled</name> - <value>true</value> - <description>Connect to oss over ssl or not, true by default.</description> - </property> - -## Testing the hadoop-aliyun Module - -To test `oss://` filesystem client, two files which pass in authentication -details to the test runner are needed. - -1. `auth-keys.xml` -2. `core-site.xml` - -Those two configuration files must be put into -`hadoop-tools/hadoop-aliyun/src/test/resources`. - -### `core-site.xml` - -This file pre-exists and sources the configurations created in `auth-keys.xml`. - -For most cases, no modification is needed, unless a specific, non-default property -needs to be set during the testing. - -### `auth-keys.xml` - -This file triggers the testing of Aliyun OSS module. Without this file, -*none of the tests in this module will be executed* - -It contains the access key Id/secret and proxy information that are needed to -connect to Aliyun OSS, and an OSS bucket URL should be also provided. - -1. `test.fs.oss.name` : the URL of the bucket for Aliyun OSS tests - -The contents of the bucket will be cleaned during the testing process, so -do not use the bucket for any purpose other than testing. - -### Run Hadoop contract tests -Create file `contract-test-options.xml` under `/test/resources`. If a -specific file `fs.contract.test.fs.oss` test path is not defined, those -tests will be skipped. Credentials are also needed to run any of those -tests, they can be copied from `auth-keys.xml` or through direct -XInclude inclusion. Here is an example of `contract-test-options.xml`: - - <?xml version="1.0"?> - <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> - <configuration> - - <include xmlns="http://www.w3.org/2001/XInclude"; - href="auth-keys.xml"/> - - <property> - <name>fs.contract.test.fs.oss</name> - <value>oss://spark-tests</value> - </property> - - <property> - <name>fs.oss.impl</name> - <value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value> - </property> - - <property> - <name>fs.oss.endpoint</name> - <value>oss-cn-hangzhou.aliyuncs.com</value> - </property> - - <property> - <name>fs.oss.buffer.dir</name> - <value>/tmp/oss</value> - </property> - - <property> - <name>fs.oss.multipart.download.size</name> - <value>102400</value> - </property> - </configuration> http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSTestUtils.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSTestUtils.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSTestUtils.java deleted file mode 100644 index 901cb2b..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSTestUtils.java +++ /dev/null @@ -1,77 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss; - -import org.apache.commons.lang.StringUtils; -import org.apache.hadoop.conf.Configuration; -import org.junit.internal.AssumptionViolatedException; - -import java.io.IOException; -import java.net.URI; - -/** - * Utility class for Aliyun OSS Tests. - */ -public final class AliyunOSSTestUtils { - - private AliyunOSSTestUtils() { - } - - /** - * Create the test filesystem. - * - * If the test.fs.oss.name property is not set, - * tests will fail. - * - * @param conf configuration - * @return the FS - * @throws IOException - */ - public static AliyunOSSFileSystem createTestFileSystem(Configuration conf) - throws IOException { - String fsname = conf.getTrimmed( - TestAliyunOSSFileSystemContract.TEST_FS_OSS_NAME, ""); - - boolean liveTest = StringUtils.isNotEmpty(fsname); - URI testURI = null; - if (liveTest) { - testURI = URI.create(fsname); - liveTest = testURI.getScheme().equals(Constants.FS_OSS); - } - - if (!liveTest) { - throw new AssumptionViolatedException("No test filesystem in " - + TestAliyunOSSFileSystemContract.TEST_FS_OSS_NAME); - } - AliyunOSSFileSystem ossfs = new AliyunOSSFileSystem(); - ossfs.initialize(testURI, conf); - return ossfs; - } - - /** - * Generate unique test path for multiple user tests. - * - * @return root test path - */ - public static String generateUniqueTestPath() { - String testUniqueForkId = System.getProperty("test.unique.fork.id"); - return testUniqueForkId == null ? "/test" : - "/" + testUniqueForkId + "/test"; - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunCredentials.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunCredentials.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunCredentials.java deleted file mode 100644 index e08a4dc..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunCredentials.java +++ /dev/null @@ -1,78 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss; - -import com.aliyun.oss.common.auth.Credentials; -import com.aliyun.oss.common.auth.InvalidCredentialsException; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.aliyun.oss.contract.AliyunOSSContract; -import org.apache.hadoop.fs.contract.AbstractFSContract; -import org.apache.hadoop.fs.contract.AbstractFSContractTestBase; -import org.junit.Test; - -import java.io.IOException; - -import static org.apache.hadoop.fs.aliyun.oss.Constants.ACCESS_KEY_ID; -import static org.apache.hadoop.fs.aliyun.oss.Constants.ACCESS_KEY_SECRET; -import static org.apache.hadoop.fs.aliyun.oss.Constants.SECURITY_TOKEN; - -/** - * Tests use of temporary credentials (for example, Aliyun STS & Aliyun OSS). - * This test extends a class that "does things to the root directory", and - * should only be used against transient filesystems where you don't care about - * the data. - */ -public class TestAliyunCredentials extends AbstractFSContractTestBase { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } - - @Test - public void testCredentialMissingAccessKeyId() throws Throwable { - Configuration conf = new Configuration(); - conf.set(ACCESS_KEY_ID, ""); - conf.set(ACCESS_KEY_SECRET, "accessKeySecret"); - conf.set(SECURITY_TOKEN, "token"); - validateCredential(conf); - } - - @Test - public void testCredentialMissingAccessKeySecret() throws Throwable { - Configuration conf = new Configuration(); - conf.set(ACCESS_KEY_ID, "accessKeyId"); - conf.set(ACCESS_KEY_SECRET, ""); - conf.set(SECURITY_TOKEN, "token"); - validateCredential(conf); - } - - private void validateCredential(Configuration conf) { - try { - AliyunCredentialsProvider provider - = new AliyunCredentialsProvider(conf); - Credentials credentials = provider.getCredentials(); - fail("Expected a CredentialInitializationException, got " + credentials); - } catch (InvalidCredentialsException expected) { - // expected - } catch (IOException e) { - fail("Unexpected exception."); - } - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java deleted file mode 100644 index 19120a6..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java +++ /dev/null @@ -1,218 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileAlreadyExistsException; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.FileSystemContractBaseTest; -import org.apache.hadoop.fs.Path; - -import org.junit.Before; -import org.junit.Test; - -import java.io.FileNotFoundException; -import java.io.IOException; - -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; -import static org.junit.Assert.fail; -import static org.junit.Assume.assumeNotNull; -import static org.junit.Assume.assumeTrue; - -/** - * Tests a live Aliyun OSS system. - */ -public class TestAliyunOSSFileSystemContract - extends FileSystemContractBaseTest { - public static final String TEST_FS_OSS_NAME = "test.fs.oss.name"; - private static Path testRootPath = - new Path(AliyunOSSTestUtils.generateUniqueTestPath()); - - @Before - public void setUp() throws Exception { - Configuration conf = new Configuration(); - fs = AliyunOSSTestUtils.createTestFileSystem(conf); - assumeNotNull(fs); - } - - public Path getTestBaseDir() { - return testRootPath; - } - - @Test - public void testMkdirsWithUmask() throws Exception { - // not supported - } - - @Test - public void testRootDirAlwaysExists() throws Exception { - //this will throw an exception if the path is not found - fs.getFileStatus(super.path("/")); - //this catches overrides of the base exists() method that don't - //use getFileStatus() as an existence probe - assertTrue("FileSystem.exists() fails for root", - fs.exists(super.path("/"))); - } - - @Test - public void testRenameRootDirForbidden() throws Exception { - assumeTrue(renameSupported()); - rename(super.path("/"), - super.path("/test/newRootDir"), - false, true, false); - } - - @Test - public void testDeleteSubdir() throws IOException { - Path parentDir = this.path("/test/hadoop"); - Path file = this.path("/test/hadoop/file"); - Path subdir = this.path("/test/hadoop/subdir"); - this.createFile(file); - - assertTrue("Created subdir", this.fs.mkdirs(subdir)); - assertTrue("File exists", this.fs.exists(file)); - assertTrue("Parent dir exists", this.fs.exists(parentDir)); - assertTrue("Subdir exists", this.fs.exists(subdir)); - - assertTrue("Deleted subdir", this.fs.delete(subdir, true)); - assertTrue("Parent should exist", this.fs.exists(parentDir)); - - assertTrue("Deleted file", this.fs.delete(file, false)); - assertTrue("Parent should exist", this.fs.exists(parentDir)); - } - - - @Override - protected boolean renameSupported() { - return true; - } - - @Test - public void testRenameNonExistentPath() throws Exception { - assumeTrue(renameSupported()); - Path src = this.path("/test/hadoop/path"); - Path dst = this.path("/test/new/newpath"); - try { - super.rename(src, dst, false, false, false); - fail("Should throw FileNotFoundException!"); - } catch (FileNotFoundException e) { - // expected - } - } - - @Test - public void testRenameFileMoveToNonExistentDirectory() throws Exception { - assumeTrue(renameSupported()); - Path src = this.path("/test/hadoop/file"); - this.createFile(src); - Path dst = this.path("/test/new/newfile"); - try { - super.rename(src, dst, false, true, false); - fail("Should throw FileNotFoundException!"); - } catch (FileNotFoundException e) { - // expected - } - } - - @Test - public void testRenameDirectoryMoveToNonExistentDirectory() throws Exception { - assumeTrue(renameSupported()); - Path src = this.path("/test/hadoop/dir"); - this.fs.mkdirs(src); - Path dst = this.path("/test/new/newdir"); - try { - super.rename(src, dst, false, true, false); - fail("Should throw FileNotFoundException!"); - } catch (FileNotFoundException e) { - // expected - } - } - - @Test - public void testRenameFileMoveToExistingDirectory() throws Exception { - super.testRenameFileMoveToExistingDirectory(); - } - - @Test - public void testRenameFileAsExistingFile() throws Exception { - assumeTrue(renameSupported()); - Path src = this.path("/test/hadoop/file"); - this.createFile(src); - Path dst = this.path("/test/new/newfile"); - this.createFile(dst); - try { - super.rename(src, dst, false, true, true); - fail("Should throw FileAlreadyExistsException"); - } catch (FileAlreadyExistsException e) { - // expected - } - } - - @Test - public void testRenameDirectoryAsExistingFile() throws Exception { - assumeTrue(renameSupported()); - Path src = this.path("/test/hadoop/dir"); - this.fs.mkdirs(src); - Path dst = this.path("/test/new/newfile"); - this.createFile(dst); - try { - super.rename(src, dst, false, true, true); - fail("Should throw FileAlreadyExistsException"); - } catch (FileAlreadyExistsException e) { - // expected - } - } - - @Test - public void testGetFileStatusFileAndDirectory() throws Exception { - Path filePath = this.path("/test/oss/file1"); - this.createFile(filePath); - assertTrue("Should be file", this.fs.getFileStatus(filePath).isFile()); - assertFalse("Should not be directory", - this.fs.getFileStatus(filePath).isDirectory()); - - Path dirPath = this.path("/test/oss/dir"); - this.fs.mkdirs(dirPath); - assertTrue("Should be directory", - this.fs.getFileStatus(dirPath).isDirectory()); - assertFalse("Should not be file", this.fs.getFileStatus(dirPath).isFile()); - - Path parentPath = this.path("/test/oss"); - for (FileStatus fileStatus: fs.listStatus(parentPath)) { - assertTrue("file and directory should be new", - fileStatus.getModificationTime() > 0L); - } - } - - @Test - public void testMkdirsForExistingFile() throws Exception { - Path testFile = this.path("/test/hadoop/file"); - assertFalse(this.fs.exists(testFile)); - this.createFile(testFile); - assertTrue(this.fs.exists(testFile)); - try { - this.fs.mkdirs(testFile); - fail("Should throw FileAlreadyExistsException!"); - } catch (FileAlreadyExistsException e) { - // expected - } - } - -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemStore.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemStore.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemStore.java deleted file mode 100644 index 7f4bac2..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemStore.java +++ /dev/null @@ -1,125 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.Path; -import org.junit.After; -import org.junit.Before; -import org.junit.BeforeClass; -import org.junit.Test; - -import java.io.BufferedInputStream; -import java.io.BufferedOutputStream; -import java.io.IOException; -import java.io.InputStream; -import java.io.OutputStream; -import java.net.URI; -import java.security.DigestInputStream; -import java.security.DigestOutputStream; -import java.security.MessageDigest; -import java.security.NoSuchAlgorithmException; - -import static org.junit.Assert.assertArrayEquals; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; -import static org.junit.Assume.assumeNotNull; - -/** - * Test the bridging logic between Hadoop's abstract filesystem and - * Aliyun OSS. - */ -public class TestAliyunOSSFileSystemStore { - private Configuration conf; - private AliyunOSSFileSystemStore store; - private AliyunOSSFileSystem fs; - - @Before - public void setUp() throws Exception { - conf = new Configuration(); - fs = new AliyunOSSFileSystem(); - fs.initialize(URI.create(conf.get("test.fs.oss.name")), conf); - store = fs.getStore(); - } - - @After - public void tearDown() throws Exception { - try { - store.purge("test"); - } catch (Exception e) { - e.printStackTrace(); - throw e; - } - } - - @BeforeClass - public static void checkSettings() throws Exception { - Configuration conf = new Configuration(); - assumeNotNull(conf.get(Constants.ACCESS_KEY_ID)); - assumeNotNull(conf.get(Constants.ACCESS_KEY_SECRET)); - assumeNotNull(conf.get("test.fs.oss.name")); - } - - protected void writeRenameReadCompare(Path path, long len) - throws IOException, NoSuchAlgorithmException { - // If len > fs.oss.multipart.upload.threshold, - // we'll use a multipart upload copy - MessageDigest digest = MessageDigest.getInstance("MD5"); - OutputStream out = new BufferedOutputStream( - new DigestOutputStream(fs.create(path, false), digest)); - for (long i = 0; i < len; i++) { - out.write('Q'); - } - out.flush(); - out.close(); - - assertTrue("Exists", fs.exists(path)); - - Path copyPath = path.suffix(".copy"); - fs.rename(path, copyPath); - - assertTrue("Copy exists", fs.exists(copyPath)); - - // Download file from Aliyun OSS and compare the digest against the original - MessageDigest digest2 = MessageDigest.getInstance("MD5"); - InputStream in = new BufferedInputStream( - new DigestInputStream(fs.open(copyPath), digest2)); - long copyLen = 0; - while (in.read() != -1) { - copyLen++; - } - in.close(); - - assertEquals("Copy length matches original", len, copyLen); - assertArrayEquals("Digests match", digest.digest(), digest2.digest()); - } - - @Test - public void testSmallUpload() throws IOException, NoSuchAlgorithmException { - // Regular upload, regular copy - writeRenameReadCompare(new Path("/test/small"), 16384); - } - - @Test - public void testLargeUpload() - throws IOException, NoSuchAlgorithmException { - // Multipart upload, multipart copy - writeRenameReadCompare(new Path("/test/xlarge"), 52428800L); // 50MB byte - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSInputStream.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSInputStream.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSInputStream.java deleted file mode 100644 index d798caf..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSInputStream.java +++ /dev/null @@ -1,155 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FSDataInputStream; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.contract.ContractTestUtils; -import org.apache.hadoop.io.IOUtils; -import org.apache.hadoop.fs.FileStatus; -import org.junit.After; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.Timeout; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.Random; - -import static org.junit.Assert.assertTrue; - -/** - * Tests basic functionality for AliyunOSSInputStream, including seeking and - * reading files. - */ -public class TestAliyunOSSInputStream { - - private FileSystem fs; - - private static final Logger LOG = - LoggerFactory.getLogger(TestAliyunOSSInputStream.class); - - private static String testRootPath = - AliyunOSSTestUtils.generateUniqueTestPath(); - - @Rule - public Timeout testTimeout = new Timeout(30 * 60 * 1000); - - @Before - public void setUp() throws Exception { - Configuration conf = new Configuration(); - fs = AliyunOSSTestUtils.createTestFileSystem(conf); - } - - @After - public void tearDown() throws Exception { - if (fs != null) { - fs.delete(new Path(testRootPath), true); - } - } - - private Path setPath(String path) { - if (path.startsWith("/")) { - return new Path(testRootPath + path); - } else { - return new Path(testRootPath + "/" + path); - } - } - - @Test - public void testSeekFile() throws Exception { - Path smallSeekFile = setPath("/test/smallSeekFile.txt"); - long size = 5 * 1024 * 1024; - - ContractTestUtils.generateTestFile(this.fs, smallSeekFile, size, 256, 255); - LOG.info("5MB file created: smallSeekFile.txt"); - - FSDataInputStream instream = this.fs.open(smallSeekFile); - int seekTimes = 5; - LOG.info("multiple fold position seeking test...:"); - for (int i = 0; i < seekTimes; i++) { - long pos = size / (seekTimes - i) - 1; - LOG.info("begin seeking for pos: " + pos); - instream.seek(pos); - assertTrue("expected position at:" + pos + ", but got:" - + instream.getPos(), instream.getPos() == pos); - LOG.info("completed seeking at pos: " + instream.getPos()); - } - LOG.info("random position seeking test...:"); - Random rand = new Random(); - for (int i = 0; i < seekTimes; i++) { - long pos = Math.abs(rand.nextLong()) % size; - LOG.info("begin seeking for pos: " + pos); - instream.seek(pos); - assertTrue("expected position at:" + pos + ", but got:" - + instream.getPos(), instream.getPos() == pos); - LOG.info("completed seeking at pos: " + instream.getPos()); - } - IOUtils.closeStream(instream); - } - - @Test - public void testReadFile() throws Exception { - final int bufLen = 256; - final int sizeFlag = 5; - String filename = "readTestFile_" + sizeFlag + ".txt"; - Path readTestFile = setPath("/test/" + filename); - long size = sizeFlag * 1024 * 1024; - - ContractTestUtils.generateTestFile(this.fs, readTestFile, size, 256, 255); - LOG.info(sizeFlag + "MB file created: /test/" + filename); - - FSDataInputStream instream = this.fs.open(readTestFile); - byte[] buf = new byte[bufLen]; - long bytesRead = 0; - while (bytesRead < size) { - int bytes; - if (size - bytesRead < bufLen) { - int remaining = (int)(size - bytesRead); - bytes = instream.read(buf, 0, remaining); - } else { - bytes = instream.read(buf, 0, bufLen); - } - bytesRead += bytes; - - if (bytesRead % (1024 * 1024) == 0) { - int available = instream.available(); - int remaining = (int)(size - bytesRead); - assertTrue("expected remaining:" + remaining + ", but got:" + available, - remaining == available); - LOG.info("Bytes read: " + Math.round((double)bytesRead / (1024 * 1024)) - + " MB"); - } - } - assertTrue(instream.available() == 0); - IOUtils.closeStream(instream); - } - - @Test - public void testDirectoryModifiedTime() throws Exception { - Path emptyDirPath = setPath("/test/emptyDirectory"); - fs.mkdirs(emptyDirPath); - FileStatus dirFileStatus = fs.getFileStatus(emptyDirPath); - assertTrue("expected the empty dir is new", - dirFileStatus.getModificationTime() > 0L); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSOutputStream.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSOutputStream.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSOutputStream.java deleted file mode 100644 index 6b87d9c..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSOutputStream.java +++ /dev/null @@ -1,91 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.contract.ContractTestUtils; -import org.junit.After; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.Timeout; - -import java.io.IOException; - -/** - * Tests regular and multi-part upload functionality for AliyunOSSOutputStream. - */ -public class TestAliyunOSSOutputStream { - private FileSystem fs; - private static String testRootPath = - AliyunOSSTestUtils.generateUniqueTestPath(); - - @Rule - public Timeout testTimeout = new Timeout(30 * 60 * 1000); - - @Before - public void setUp() throws Exception { - Configuration conf = new Configuration(); - conf.setLong(Constants.MIN_MULTIPART_UPLOAD_THRESHOLD_KEY, 5 * 1024 * 1024); - conf.setInt(Constants.MULTIPART_UPLOAD_SIZE_KEY, 5 * 1024 * 1024); - fs = AliyunOSSTestUtils.createTestFileSystem(conf); - } - - @After - public void tearDown() throws Exception { - if (fs != null) { - fs.delete(new Path(testRootPath), true); - } - } - - protected Path getTestPath() { - return new Path(testRootPath + "/test-aliyun-oss"); - } - - @Test - public void testRegularUpload() throws IOException { - ContractTestUtils.createAndVerifyFile(fs, getTestPath(), 1024 * 1024); - } - - @Test - public void testMultiPartUpload() throws IOException { - ContractTestUtils.createAndVerifyFile(fs, getTestPath(), 6 * 1024 * 1024); - } - - @Test - public void testMultiPartUploadLimit() throws IOException { - long partSize1 = AliyunOSSUtils.calculatePartSize(10 * 1024, 100 * 1024); - assert(10 * 1024 / partSize1 < Constants.MULTIPART_UPLOAD_PART_NUM_LIMIT); - - long partSize2 = AliyunOSSUtils.calculatePartSize(200 * 1024, 100 * 1024); - assert(200 * 1024 / partSize2 < Constants.MULTIPART_UPLOAD_PART_NUM_LIMIT); - - long partSize3 = AliyunOSSUtils.calculatePartSize(10000 * 100 * 1024, - 100 * 1024); - assert(10000 * 100 * 1024 / partSize3 - < Constants.MULTIPART_UPLOAD_PART_NUM_LIMIT); - - long partSize4 = AliyunOSSUtils.calculatePartSize(10001 * 100 * 1024, - 100 * 1024); - assert(10001 * 100 * 1024 / partSize4 - < Constants.MULTIPART_UPLOAD_PART_NUM_LIMIT); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/AliyunOSSContract.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/AliyunOSSContract.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/AliyunOSSContract.java deleted file mode 100644 index 624c606..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/AliyunOSSContract.java +++ /dev/null @@ -1,49 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.contract.AbstractBondedFSContract; - -/** - * The contract of Aliyun OSS: only enabled if the test bucket is provided. - */ -public class AliyunOSSContract extends AbstractBondedFSContract { - - public static final String CONTRACT_XML = "contract/aliyun-oss.xml"; - - public AliyunOSSContract(Configuration conf) { - super(conf); - //insert the base features - addConfResource(CONTRACT_XML); - } - - @Override - public String getScheme() { - return "oss"; - } - - @Override - public Path getTestPath() { - String testUniqueForkId = System.getProperty("test.unique.fork.id"); - return testUniqueForkId == null ? super.getTestPath() : - new Path("/" + testUniqueForkId, "test"); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractCreate.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractCreate.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractCreate.java deleted file mode 100644 index 88dd8cd..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractCreate.java +++ /dev/null @@ -1,35 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractCreateTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; - -/** - * Aliyun OSS contract creating tests. - */ -public class TestAliyunOSSContractCreate extends AbstractContractCreateTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } - -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDelete.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDelete.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDelete.java deleted file mode 100644 index 1658d80..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDelete.java +++ /dev/null @@ -1,34 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractDeleteTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; - -/** - * Aliyun OSS contract deleting tests. - */ -public class TestAliyunOSSContractDelete extends AbstractContractDeleteTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDistCp.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDistCp.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDistCp.java deleted file mode 100644 index 18d09d5..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractDistCp.java +++ /dev/null @@ -1,44 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.tools.contract.AbstractContractDistCpTest; - -import static org.apache.hadoop.fs.aliyun.oss.Constants.*; - -/** - * Contract test suite covering Aliyun OSS integration with DistCp. - */ -public class TestAliyunOSSContractDistCp extends AbstractContractDistCpTest { - - private static final long MULTIPART_SETTING = 8 * 1024 * 1024; // 8 MB - - @Override - protected Configuration createConfiguration() { - Configuration newConf = super.createConfiguration(); - newConf.setLong(MIN_MULTIPART_UPLOAD_THRESHOLD_KEY, MULTIPART_SETTING); - newConf.setLong(MULTIPART_UPLOAD_SIZE_KEY, MULTIPART_SETTING); - return newConf; - } - - @Override - protected AliyunOSSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractGetFileStatus.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractGetFileStatus.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractGetFileStatus.java deleted file mode 100644 index c69124d..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractGetFileStatus.java +++ /dev/null @@ -1,35 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; - -/** - * Test getFileStatus and related listing operations. - */ -public class TestAliyunOSSContractGetFileStatus - extends AbstractContractGetFileStatusTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } - -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractMkdir.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractMkdir.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractMkdir.java deleted file mode 100644 index 6cb7549..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractMkdir.java +++ /dev/null @@ -1,34 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractMkdirTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; - -/** - * Aliyun OSS contract directory tests. - */ -public class TestAliyunOSSContractMkdir extends AbstractContractMkdirTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractOpen.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractOpen.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractOpen.java deleted file mode 100644 index 099aba6..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractOpen.java +++ /dev/null @@ -1,34 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractOpenTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; - -/** - * Aliyun OSS contract opening file tests. - */ -public class TestAliyunOSSContractOpen extends AbstractContractOpenTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRename.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRename.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRename.java deleted file mode 100644 index e15b3ba..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRename.java +++ /dev/null @@ -1,35 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractRenameTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; - -/** - * Aliyun OSS contract renaming tests. - */ -public class TestAliyunOSSContractRename extends AbstractContractRenameTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } - -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRootDir.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRootDir.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRootDir.java deleted file mode 100644 index 9faae37..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractRootDir.java +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.FileNotFoundException; -import java.io.IOException; - -/** - * Root dir operations against an Aliyun OSS bucket. - */ -public class TestAliyunOSSContractRootDir extends - AbstractContractRootDirectoryTest { - - private static final Logger LOG = - LoggerFactory.getLogger(TestAliyunOSSContractRootDir.class); - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } - - @Override - public void testListEmptyRootDirectory() throws IOException { - for (int attempt = 1, maxAttempts = 10; attempt <= maxAttempts; ++attempt) { - try { - super.testListEmptyRootDirectory(); - break; - } catch (AssertionError | FileNotFoundException e) { - if (attempt < maxAttempts) { - LOG.info("Attempt {} of {} for empty root directory test failed. " - + "Attempting retry.", attempt, maxAttempts); - try { - Thread.sleep(1000); - } catch (InterruptedException e2) { - Thread.currentThread().interrupt(); - fail("Test interrupted."); - break; - } - } else { - LOG.error( - "Empty root directory test failed {} attempts. Failing test.", - maxAttempts); - throw e; - } - } - } - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractSeek.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractSeek.java b/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractSeek.java deleted file mode 100644 index d9b3674..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/contract/TestAliyunOSSContractSeek.java +++ /dev/null @@ -1,60 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.fs.aliyun.oss.contract; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FSDataInputStream; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.contract.AbstractContractSeekTest; -import org.apache.hadoop.fs.contract.AbstractFSContract; -import org.junit.Test; - -import static org.apache.hadoop.fs.contract.ContractTestUtils.createFile; -import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; - -/** - * Aliyun OSS contract seeking tests. - */ -public class TestAliyunOSSContractSeek extends AbstractContractSeekTest { - - @Override - protected AbstractFSContract createContract(Configuration conf) { - return new AliyunOSSContract(conf); - } - - @Test - public void testSeekBeyondDownloadSize() throws Throwable { - describe("seek and read beyond download size."); - - Path byteFile = path("byte_file.txt"); - // 'fs.oss.multipart.download.size' = 100 * 1024 - byte[] block = dataset(100 * 1024 + 10, 0, 255); - FileSystem fs = getFileSystem(); - createFile(fs, byteFile, true, block); - - FSDataInputStream instream = getFileSystem().open(byteFile); - instream.seek(100 * 1024 - 1); - assertEquals(100 * 1024 - 1, instream.getPos()); - assertEquals(144, instream.read()); - instream.seek(100 * 1024 + 1); - assertEquals(100 * 1024 + 1, instream.getPos()); - assertEquals(146, instream.read()); - } -} http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/resources/contract/aliyun-oss.xml ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/resources/contract/aliyun-oss.xml b/hadoop-tools/hadoop-aliyun/src/test/resources/contract/aliyun-oss.xml deleted file mode 100644 index 9ec4be6..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/resources/contract/aliyun-oss.xml +++ /dev/null @@ -1,120 +0,0 @@ -<?xml version="1.0"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - ~ Licensed to the Apache Software Foundation (ASF) under one - ~ or more contributor license agreements. See the NOTICE file - ~ distributed with this work for additional information - ~ regarding copyright ownership. The ASF licenses this file - ~ to you under the Apache License, Version 2.0 (the - ~ "License"); you may not use this file except in compliance - ~ with the License. You may obtain a copy of the License at - ~ - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> -<configuration> - <property> - <name>fs.contract.test.random-seek-count</name> - <value>10</value> - </property> - - <property> - <name>fs.contract.is-blobstore</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.is-case-sensitive</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.rename-returns-false-if-source-missing</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.rename-remove-dest-if-empty-dir</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.supports-append</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.supports-atomic-directory-delete</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.supports-atomic-rename</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.supports-block-locality</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.supports-concat</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.supports-seek</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.supports-seek-on-closed-file</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.rejects-seek-past-eof</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.supports-strict-exceptions</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.supports-unix-permissions</name> - <value>false</value> - </property> - - <property> - <name>fs.contract.rename-overwrites-dest</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.test.root-tests-enabled</name> - <value>true</value> - </property> - - <property> - <name>fs.contract.supports-getfilestatus</name> - <value>true</value> - </property> - - <property> - <name>fs.oss.multipart.download.size</name> - <value>102400</value> - </property> - - <property> - <name>fs.contract.create-visibility-delayed</name> - <value>true</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/resources/core-site.xml ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/resources/core-site.xml b/hadoop-tools/hadoop-aliyun/src/test/resources/core-site.xml deleted file mode 100644 index fa4118c..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/resources/core-site.xml +++ /dev/null @@ -1,46 +0,0 @@ -<?xml version="1.0"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - ~ Licensed to the Apache Software Foundation (ASF) under one - ~ or more contributor license agreements. See the NOTICE file - ~ distributed with this work for additional information - ~ regarding copyright ownership. The ASF licenses this file - ~ to you under the Apache License, Version 2.0 (the - ~ "License"); you may not use this file except in compliance - ~ with the License. You may obtain a copy of the License at - ~ - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> -<configuration> - - <property> - <name>hadoop.tmp.dir</name> - <value>target/build/test</value> - <description>A base for other temporary directories.</description> - <final>true</final> - </property> - - <!-- Turn security off for tests by default --> - <property> - <name>hadoop.security.authentication</name> - <value>simple</value> - </property> - - <!-- - To run these tests. - - # Create a file auth-keys.xml - DO NOT ADD TO REVISION CONTROL - # add the property test.fs.oss.name to point to an OSS filesystem URL - # Add the credentials for the service you are testing against - --> - <include xmlns="http://www.w3.org/2001/XInclude"; href="auth-keys.xml"> - <fallback/> - </include> - -</configuration> http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-aliyun/src/test/resources/log4j.properties ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aliyun/src/test/resources/log4j.properties b/hadoop-tools/hadoop-aliyun/src/test/resources/log4j.properties deleted file mode 100644 index bb5cbe5..0000000 --- a/hadoop-tools/hadoop-aliyun/src/test/resources/log4j.properties +++ /dev/null @@ -1,23 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# log4j configuration used during build and unit tests - -log4j.rootLogger=INFO,stdout -log4j.threshold=ALL -log4j.appender.stdout=org.apache.log4j.ConsoleAppender -log4j.appender.stdout.layout=org.apache.log4j.PatternLayout -log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/hadoop-tools-dist/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-tools-dist/pom.xml b/hadoop-tools/hadoop-tools-dist/pom.xml index 3af7aee..3ecc51b 100644 --- a/hadoop-tools/hadoop-tools-dist/pom.xml +++ b/hadoop-tools/hadoop-tools-dist/pom.xml @@ -102,12 +102,6 @@ </dependency> <dependency> <groupId>org.apache.hadoop</groupId> - <artifactId>hadoop-aliyun</artifactId> - <scope>compile</scope> - <version>${project.version}</version> - </dependency> - <dependency> - <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-sls</artifactId> <scope>compile</scope> </dependency> http://git-wip-us.apache.org/repos/asf/hadoop/blob/12901cdc/hadoop-tools/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-tools/pom.xml b/hadoop-tools/pom.xml index 117d12c..e307a40 100644 --- a/hadoop-tools/pom.xml +++ b/hadoop-tools/pom.xml @@ -48,7 +48,6 @@ <module>hadoop-aws</module> <module>hadoop-azure</module> <module>hadoop-azure-datalake</module> - <module>hadoop-aliyun</module> </modules> <build> --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
