[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573672#comment-16573672 ]
Thomas Marquardt edited comment on HADOOP-15546 at 8/8/18 6:30 PM: ------------------------------------------------------------------- I will commit this shortly. +1 from me. All tests pass with fs.azure.scale.test.enabled set to true for both WASB and ABFS. *Tests run: 257, Failures: 0, Errors: 0, Skipped: 11* *Tests run: 1, Failures: 0, Errors: 0, Skipped: 0* *Tests run: 769, Failures: 0, Errors: 0, Skipped: 183* *Tests run: 186, Failures: 0, Errors: 0, Skipped: 10* We will address the Yetus issue later. We will address the issues around config in a separate JIRA. For now, the recommendation is to use separate files (azure-auth-keys.xml and azure-bfs-auth-keys.xml) for WASB and ABFS. The contents are as shown in the example below: *azure-auth-keys.xml:* <property> <name>fs.contract.test.fs.wasb</name> <value>wasb://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value> <description>The name of the azure file system for testing.</description> </property> <property> <name>fs.azure.test.account.name</name> <value>*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value> </property> <property> <name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.scale.test.enabled</name> <value>true</value> </property> <property> <name>fs.azure.secure.mode</name> <value>false</value> </property> *azure-bfs-auth-keys.xml*: <property> <name>fs.contract.test.fs.abfs</name> <value>abfs://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value> <description>The name of the azure file system for testing.</description> </property> <property> <name>fs.azure.test.account.name</name> <value>*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value> </property> <property> <name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.scale.test.enabled</name> <value>true</value> </property> was (Author: tmarquardt): I will commit this shortly. +1 from me. All tests pass with fs.azure.scale.test.enabled set to true for both WASB and ABFS. *Tests run: 257, Failures: 0, Errors: 0, Skipped: 11* *Tests run: 1, Failures: 0, Errors: 0, Skipped: 0* *Tests run: 769, Failures: 0, Errors: 0, Skipped: 183* *Tests run: 186, Failures: 0, Errors: 0, Skipped: 10* We will address the Yetus issue later. We will address the issues around config in a separate JIRA. For now, the recommendation is to use separate files (azure-auth-keys.xml and azure-bfs-auth-keys.xml) for WASB and ABFS. The contents are as shown in the example below: *azure-auth-keys.xml:* <property> <name>fs.contract.test.fs.wasb</name> <value>wasb://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value> <description>The name of the azure file system for testing.</description> </property> <property> <name>fs.azure.test.account.name</name> <value>*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</value> </property> <property> <name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.scale.test.enabled</name> <value>true</value> </property> <property> <name>fs.azure.secure.mode</name> <value>false</value> </property> *azure-bfs-auth-keys.xml*: <property> <name>fs.contract.test.fs.abfs</name> <value>abfs://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value> <description>The name of the azure file system for testing.</description> </property> <property> <name>fs.azure.test.account.name</name> <value>*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</value> </property> <property> <name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net</name> <value>*{YOUR_ACCOUNT_KEY}*</value> </property> <property> <name>fs.azure.scale.test.enabled</name> <value>true</value> </property> > ABFS: tune imports & javadocs; stabilise tests > ---------------------------------------------- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Affects Versions: HADOOP-15407 > Reporter: Steve Loughran > Assignee: Thomas Marquardt > Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch, > HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org