[jira] [Created] (HDDS-1931) Recon cannot download OM DB snapshot in ozonesecure
Doroszlai, Attila created HDDS-1931:
---
Summary: Recon cannot download OM DB snapshot in ozonesecure
Key: HDDS-1931
URL: https://issues.apache.org/jira/browse/HDDS-1931
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: docker, Ozone Recon
Affects Versions: 0.5.0
Reporter: Doroszlai, Attila
{code}
recon_1 | 2019-08-07 22:09:40 ERROR OzoneManagerServiceProviderImpl:186 -
Unable to obtain Ozone Manager DB Snapshot.
recon_1 | java.io.IOException: Unexpected exception when trying to reach
Ozone Manager,
recon_1 |
recon_1 |
recon_1 | Error 401 Authentication required
recon_1 |
recon_1 | HTTP ERROR 401
recon_1 | Problem accessing /dbCheckpoint. Reason:
recon_1 | Authentication required
recon_1 |
recon_1 |
recon_1 |
recon_1 | at
org.apache.hadoop.ozone.recon.ReconUtils.makeHttpCall(ReconUtils.java:171)
recon_1 | at
org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.getOzoneManagerDBSnapshot(OzoneManagerServiceProviderImpl.java:170)
recon_1 | at
org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.updateReconOmDBWithNewSnapshot(OzoneManagerServiceProviderImpl.java:141)
recon_1 | at
org.apache.hadoop.ozone.recon.ReconServer.lambda$scheduleReconTasks$1(ReconServer.java:138)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-14034) Support getQuotaUsage API in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun reopened HDFS-14034:
-
Re-opening this for backporting to branch-2.
> Support getQuotaUsage API in WebHDFS
>
>
> Key: HDFS-14034
> URL: https://issues.apache.org/jira/browse/HDFS-14034
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, webhdfs
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14034-branch-2.000.patch,
> HDFS-14034-branch-2.001.patch, HDFS-14034.000.patch, HDFS-14034.001.patch,
> HDFS-14034.002.patch, HDFS-14034.004.patch
>
>
> HDFS-8898 added support for a new API, {{getQuotaUsage}} which can fetch
> quota usage on a directory with significantly lower impact than the similar
> {{getContentSummary}}. This JIRA is to track adding support for this API to
> WebHDFS.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1740) Handle Failure to Update Ozone Container YAML
[ https://issues.apache.org/jira/browse/HDDS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Supratim Deka resolved HDDS-1740. - Resolution: Not A Problem On the Datanode, Container state changes are driven through KeyValueContainer.updateContainerData() this always resets the in-memory state of the container to the previous state in case the update to the container YAML hits any exception. Also, the container YAML is sync flushed to persistent storage as implemented in: ContainerDataYAML.createContainerFile() So I am marking this as not a problem. > Handle Failure to Update Ozone Container YAML > - > > Key: HDDS-1740 > URL: https://issues.apache.org/jira/browse/HDDS-1740 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Ensure consistent state in-memory and in the persistent YAML file for the > Container. > If an update to the YAML fails, then the in-memory state also does not change. > This ensures that in every container report, the SCM continues to see the > specific container is still in the old state. And this triggers a retry of > the state change operation from the SCM. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
Xiaoyu Yao created HDDS-1930: Summary: Test Topology Aware Job scheduling with Ozone Topology Key: HDDS-1930 URL: https://issues.apache.org/jira/browse/HDDS-1930 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao My initial results with Terasort does not seem to report the counter properly. Most of the requests are handled by rack locl but no node local. This ticket is opened to add more system testing to validate the feature. Total Allocated Containers: 3778 Each table cell represents the number of NodeLocal/RackLocal/OffSwitch containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. Node Local Request Rack Local Request Off Switch Request Num Node Local Containers (satisfied by)0 Num Rack Local Containers (satisfied by)0 3648 Num Off Switch Containers (satisfied by)0 96 34 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1865) Use "ozone.network.topology.aware.read" to control both RPC client and server side logic
[ https://issues.apache.org/jira/browse/HDDS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1865. -- Resolution: Fixed Fix Version/s: 0.5.0 Thanks [~Sammi] for the contribution. I've merged the patch to trunk. > Use "ozone.network.topology.aware.read" to control both RPC client and server > side logic > - > > Key: HDDS-1865 > URL: https://issues.apache.org/jira/browse/HDDS-1865 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1929) OM started on recon host in ozonesecure compose
Doroszlai, Attila created HDDS-1929:
---
Summary: OM started on recon host in ozonesecure compose
Key: HDDS-1929
URL: https://issues.apache.org/jira/browse/HDDS-1929
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: docker
Affects Versions: 0.5.0
Reporter: Doroszlai, Attila
Assignee: Doroszlai, Attila
OM is started temporarily on {{recon}} host in {{ozonesecure}} compose:
{noformat}
recon_1 | 2019-08-07 19:41:46 INFO OzoneManagerStarter:51 - STARTUP_MSG:
recon_1 | /
recon_1 | STARTUP_MSG: Starting OzoneManager
recon_1 | STARTUP_MSG: host = recon/192.168.16.4
recon_1 | STARTUP_MSG: args = [--init]
...
recon_1 | SHUTDOWN_MSG: Shutting down OzoneManager at recon/192.168.16.4
...
recon_1 | 2019-08-07 19:41:52 INFO ReconServer:81 - Initializing Recon
server...
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1928) Cannot run ozone-recon compose due to syntax error
Doroszlai, Attila created HDDS-1928:
---
Summary: Cannot run ozone-recon compose due to syntax error
Key: HDDS-1928
URL: https://issues.apache.org/jira/browse/HDDS-1928
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: docker
Affects Versions: 0.4.1
Reporter: Doroszlai, Attila
Assignee: Doroszlai, Attila
{noformat}
$ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone-recon
$ docker-compose up -d --scale datanode=3
ERROR: yaml.scanner.ScannerError: mapping values are not allowed here
in "./docker-compose.yaml", line 20, column 33
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1927) Create AclUtil class with helpers for add/remove Acl.
Bharat Viswanadham created HDDS-1927: Summary: Create AclUtil class with helpers for add/remove Acl. Key: HDDS-1927 URL: https://issues.apache.org/jira/browse/HDDS-1927 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham This Jira is created based on @xiaoyu comment on HDDS-1884 Can we abstract these add/remove logic into common AclUtil class as we can see similar logic in both bucket manager and key manager? For example, public static boolean addAcl(List existingAcls, OzoneAcl newAcl) public static boolean removeAcl(List existingAcls, OzoneAcl newAcl) But to do this, we need both OmKeyInfo and OMBucketInfo to use list of OzoneAcl/OzoneAclInfo. This Jira is to do that refactor, and also address above comment to move common logic to AclUtils. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Hadoop Storage Online Sync Notes 8/5/2019
Very happy to have CR from Uber leading today's discussion. Here's today's sync meeting notes. https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit 8/5/2019 CR Hota (Uber) gave an update on Router Based Federation Attendee: Cloudera (Weichiu, Adam), Uber (CR Hota) and Target (Craig) Rename: There is a change in Hive that upon exception, do a copy instead. How/where can the community help:it already support all NN APIs, running in production, mostly now is the efficiency improvement. How to migrate from non-federation to RBF —> easy. still use hfs:// scheme. Will have to update metadata (HMS) How to migrate from ViewFS based federation to RBF —> ViewFS use view’s:// so it’ll be harder to migrate. View FS based is limited to 4 namespaces. There is no such limit in RBF. Uber is already at 5 namespaces. Cluster utilization rebalancer. Not a priority at Uber because of UDestinty. Router HA supported. All routers’ state is synchronized. (Uber: has 10 routers in one cluster) Latency compare to single Namenode which is bottlenecked in 1 NN lock. Read-only name node help solve this problem too. Presto is more latency sensitive. So Uber made a change to support “read-only router” In general, very negligible latency. If there is, just add more routers. Uber doesn’t want to manage 4-5 thousand clusters. They want to manage some set of 1000 thousand clusters in the future. Isolation There is a current problem. Very important for production deployment. See HDFS-14090: fairness in router. Let me know your feedback. Is this the right topic you are looking for? Do you want to present other topics? Development discussion, demos, best practices are welcomed. Best, Weichiu
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/ [Aug 6, 2019 3:12:15 AM] (ayushsaxena) HDFS-14679. Failed to add erasure code policies with example template. [Aug 6, 2019 9:06:04 AM] (github) HDDS-1905. PipelineActionHandler is not closing the pipeline when close [Aug 6, 2019 1:29:58 PM] (weichiu) YARN-9410. Typo in documentation: Using FPGA On YARN (#1220) Contributed [Aug 6, 2019 2:57:46 PM] (weichiu) HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. [Aug 6, 2019 4:05:53 PM] (weichiu) YARN-9678. TestGpuResourceHandler / TestFpgaResourceHandler should be [Aug 6, 2019 5:21:55 PM] (weichiu) YARN-9678. Addendum: TestGpuResourceHandler / TestFpgaResourceHandler [Aug 6, 2019 7:07:53 PM] (github) HDDS-1901. Fix Ozone HTTP WebConsole Authentication. Contributed by [Aug 6, 2019 7:18:24 PM] (weichiu) HADOOP-16275. Upgrade Mockito to the latest version. Contributed by [Aug 6, 2019 8:52:30 PM] (haibochen) YARN-9559. Create AbstractContainersLauncher for pluggable [Aug 6, 2019 9:04:17 PM] (eyang) HADOOP-16457. Fixed Kerberos activation in ServiceAuthorizationManager. [Aug 6, 2019 10:24:11 PM] (weichiu) HDFS-14652. Addendum: HealthMonitor connection retry times should be [Aug 6, 2019 10:30:50 PM] (weichiu) HDFS-14692. Upload button should not encode complete url. Contributed by [Aug 7, 2019 12:00:54 AM] (bharat) HDDS-1918. hadoop-ozone-tools has integration tests run as unit (#1236) [Aug 7, 2019 12:29:18 AM] (aajisaka) HADOOP-15908. hadoop-build-tools jar is downloaded from remote [Aug 7, 2019 1:00:34 AM] (bharat) HDDS-1919. Fix Javadoc in TestAuditParser (#1240) -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core Class org.apache.hadoop.applications.mawo.server.common.TaskStatus implements Cloneable but does not define or use clone method At TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 39-346] Equals method for org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument is of type WorkerId At WorkerId.java:the argument is of type WorkerId At WorkerId.java:[line 114] org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does not check for null argument At WorkerId.java:null argument At WorkerId.java:[lines 114-115] Failed junit tests : hadoop.hdfs.server.datanode.TestLargeBlockReport hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupElasticMemoryController hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-compile-javac-root.txt [332K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-patch-pylint.txt [220K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1221/artifact/out/diff-patch-shelldocs.txt [44K] whitespace:
[jira] [Reopened] (HDFS-14631) The DirectoryScanner doesn't fix the wrongly placed replica.
[
https://issues.apache.org/jira/browse/HDFS-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun reopened HDFS-14631:
upload patch for branch-2.x
> The DirectoryScanner doesn't fix the wrongly placed replica.
>
>
> Key: HDFS-14631
> URL: https://issues.apache.org/jira/browse/HDFS-14631
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14631-branch-2.9.001.patch, HDFS-14631.001.patch,
> HDFS-14631.002.patch, HDFS-14631.003.patch, HDFS-14631.004.patch
>
>
> When DirectoryScanner scans block files, if the block refers to the block
> file does not exist the DirectoryScanner will update the block based on the
> replica file found on the disk. See FsDatasetImpl#checkAndUpdate.
>
> {code:java}
> /*
> * Block exists in volumeMap and the block file exists on the disk
> */
> // Compare block files
> if (memBlockInfo.blockDataExists()) {
> ...
> } else {
> // Block refers to a block file that does not exist.
> // Update the block with the file found on the disk. Since the block
> // file and metadata file are found as a pair on the disk, update
> // the block based on the metadata file found on the disk
> LOG.warn("Block file in replica "
> + memBlockInfo.getBlockURI()
> + " does not exist. Updating it to the file found during scan "
> + diskFile.getAbsolutePath());
> memBlockInfo.updateWithReplica(
> StorageLocation.parse(diskFile.toString()));
> LOG.warn("Updating generation stamp for block " + blockId
> + " from " + memBlockInfo.getGenerationStamp() + " to " + diskGS);
> memBlockInfo.setGenerationStamp(diskGS);
> }
> {code}
> But the DirectoryScanner doesn't really fix it because in
> LocalReplica#parseBaseDir() the 'subdir' are ignored.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1926) The new caching layer is used for old OM requests but not updated
Elek, Marton created HDDS-1926:
--
Summary: The new caching layer is used for old OM requests but not
updated
Key: HDDS-1926
URL: https://issues.apache.org/jira/browse/HDDS-1926
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: om
Reporter: Elek, Marton
HDDS-1499 introduced a new caching layer together with a double-buffer based db
writer to support OM HA.
TLDR: I think the caching layer is not updated for new volume creation. And
(slightly related to this problem) I suggest to separated the TypedTable and
the caching layer.
## How to reproduce the problem?
1. Start a docker compose cluster
2. Create one volume (let's say `/vol1`)
3. Restart the om (!)
4. Try to create an _other_ volume twice!
```
bash-4.2$ ozone sh volume create /vol2
2019-08-07 12:29:47 INFO RpcClient:288 - Creating Volume: vol2, with hadoop as
owner.
bash-4.2$ ozone sh volume create /vol2
2019-08-07 12:29:50 INFO RpcClient:288 - Creating Volume: vol2, with hadoop as
owner.
```
Expected behavior is an error:
{code}
bash-4.2$ ozone sh volume create /vol1
2019-08-07 09:48:39 INFO RpcClient:288 - Creating Volume: vol1, with hadoop as
owner.
bash-4.2$ ozone sh volume create /vol1
2019-08-07 09:48:42 INFO RpcClient:288 - Creating Volume: vol1, with hadoop as
owner.
VOLUME_ALREADY_EXISTS
{code}
The problem is that the new cache is used even for the old code path
(TypedTable):
{code}
@Override
public VALUE get(KEY key) throws IOException {
// Here the metadata lock will guarantee that cache is not updated for same
// key during get key.
CacheResult> cacheResult =
cache.lookup(new CacheKey<>(key));
if (cacheResult.getCacheStatus() == EXISTS) {
return cacheResult.getValue().getCacheValue();
} else if (cacheResult.getCacheStatus() == NOT_EXIST) {
return null;
} else {
return getFromTable(key);
}
}
{code}
For volume table after the FIRST start it always returns with
`getFromTable(key)` due to the condition in the `TableCacheImpl.lookup`:
{code}
public CacheResult lookup(CACHEKEY cachekey) {
if (cache.size() == 0) {
return new CacheResult<>(CacheResult.CacheStatus.MAY_EXIST,
null);
}
{code}
But after a restart the cache is pre-loaded by the TypedTable.constructor.
After the restart, the real caching logic will be used (as cache.size()>0),
which cause a problem as the cache is NOT updated from the old code path.
An additional problem is that the cache is turned on for all the metadata table
even if the cache is not required...
## Proposed solution
As I commented at HDDS-1499 this caching layer is not a "traditional cache".
It's not updated during the typedTable.put() call but updated by a separated
component during double-buffer flash.
I would suggest to remove the cache related methods from TypedTable (move to a
separated implementation). I think this kind of caching can be independent from
the TypedTable implementation. We can continue to use the simple TypedTable
everywhere where we don't need to use any kind of caching.
For caching we can use a separated object. It would make it more visible that
the cache should always be updated manually all the time. This separated
caching utility may include a reference to the original TypedTable/Table. With
this approach we can separate the different responsibilities but provide the
same functionality.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/ [Aug 6, 2019 4:15:24 PM] (ekrogen) HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be [Aug 6, 2019 9:30:17 PM] (weichiu) HDFS-12914. Block report leases cause missing blocks until next report. [Aug 6, 2019 10:14:45 PM] (jhung) YARN-9559. Create AbstractContainersLauncher for pluggable -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestSafeMode hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.api.impl.TestAMRMProxy hadoop.mapred.gridmix.TestDistCacheEmulation cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [432K]
[jira] [Created] (HDDS-1925) ozonesecure acceptance test broken by HTTP auth requirement
Doroszlai, Attila created HDDS-1925:
---
Summary: ozonesecure acceptance test broken by HTTP auth
requirement
Key: HDDS-1925
URL: https://issues.apache.org/jira/browse/HDDS-1925
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: docker, test
Affects Versions: 0.4.1
Reporter: Doroszlai, Attila
Acceptance test is failing at {{ozonesecure}} with the following error from
{{jq}}:
{noformat:title=https://github.com/elek/ozone-ci/blob/325779d34623061e27b80ade3b749210648086d1/byscane/byscane-nightly-ds7lx/acceptance/output.log#L2779}
parse error: Invalid numeric literal at line 2, column 0
{noformat}
Example compose environments wait for datanodes to be up:
{code:title=https://github.com/apache/hadoop/blob/9cd211ac86bb1124bdee572fddb6f86655b19b73/hadoop-ozone/dist/src/main/compose/testlib.sh#L71-L72}
docker-compose -f "$COMPOSE_FILE" up -d --scale datanode="${datanode_count}"
wait_for_datanodes "$COMPOSE_FILE" "${datanode_count}"
{code}
The number of datanodes up is determined via HTTP query of JMX endpoint:
{code:title=https://github.com/apache/hadoop/blob/9cd211ac86bb1124bdee572fddb6f86655b19b73/hadoop-ozone/dist/src/main/compose/testlib.sh#L44-L46}
#This line checks the number of HEALTHY datanodes registered in scm over
the
# jmx HTTP servlet
datanodes=$(docker-compose -f "${compose_file}" exec -T scm curl -s
'http://localhost:9876/jmx?qry=Hadoop:service=SCMNodeManager,name=SCMNodeManagerInfo'
| jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") | .value')
{code}
The problem is that no authentication is performed before or during the
request, which is no longer allowed since HDDS-1901:
{code}
$ docker-compose exec -T scm curl -s
'http://localhost:9876/jmx?qry=Hadoop:service=SCMNodeManager,name=SCMNodeManagerInfo'
Error 401 Authentication required
HTTP ERROR 401
Problem accessing /jmx. Reason:
Authentication required
{code}
{code}
$ docker-compose exec -T scm curl -s
'http://localhost:9876/jmx?qry=Hadoop:service=SCMNodeManager,name=SCMNodeManagerInfo'
| jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") | .value'
parse error: Invalid numeric literal at line 2, column 0
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1924) ozone sh bucket path command does not exist
Mukul Kumar Singh created HDDS-1924: --- Summary: ozone sh bucket path command does not exist Key: HDDS-1924 URL: https://issues.apache.org/jira/browse/HDDS-1924 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.4.0 Reporter: Mukul Kumar Singh ozone sh bucket path command does not exist but it is mentioned in the static/docs/interface/s3.html. The command should either be added back or a the documentation should be improved. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox
Mukul Kumar Singh created HDDS-1923: --- Summary: static/docs/start.html page doesn't render correctly on Firefox Key: HDDS-1923 URL: https://issues.apache.org/jira/browse/HDDS-1923 Project: Hadoop Distributed Data Store Issue Type: Bug Components: documentation Affects Versions: 0.4.0 Reporter: Mukul Kumar Singh static/docs/start.html page doesn't render correctly on Firefox -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1922) Next button on the bottom of "static/docs/index.html" landing page does not work
Mukul Kumar Singh created HDDS-1922: --- Summary: Next button on the bottom of "static/docs/index.html" landing page does not work Key: HDDS-1922 URL: https://issues.apache.org/jira/browse/HDDS-1922 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.4.0 Reporter: Mukul Kumar Singh On Ozone landing doc page, the next link doesn't work . -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
