[jira] [Resolved] (HDFS-13510) Ozone: Fix precommit hook for Ozone/Hdds on trunk

2020-06-23 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDFS-13510.

Resolution: Won't Fix

> Ozone: Fix precommit hook for Ozone/Hdds on trunk
> -
>
> Key: HDFS-13510
> URL: https://issues.apache.org/jira/browse/HDFS-13510
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>
> Current precommit doesn't work with the ozone projects as they are in an 
> optional profile.
> This jira may not have any code change but I opened it to track the required 
> changes on builds.apache.org and make the changes more transparent.
> I think we need the following changes:
> 1. Separated jira subproject, as planned
> 2. After that we can create new Precommit-OZONE-Build job which will be 
> triggered by the PreCommit-Admin (jira filter should be modified)
> 3. In the Precommit-OZONE-Build we need to enable the hdds profile. It could 
> be done by modifying the yetus personality or the create a .mvn/mvn.config
> 4. We need the ozone/hdds snapshot artifacts in apache nexus:
>   a.) One option is adding -P hdds to the Hadoop-trunk-Commit. This is the 
> simplified but Hdds/Ozone build failure will cause missing artifacts on nexus 
> (low chance as the merge will be guarded by PreCommit hook)
>   b.) Other options is to create a Hadoop-Ozone-trunk-Commit which do a full 
> compilation but only hdds and ozone artifacts will be deployed (some sync 
> problem maybe here if different core artifacts are uploaded...)
> 5. And we also need a daily unit test run. (qbt) 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2606) Acceptance tests are flaky due to async pipeline creation

2019-11-21 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2606:
-

 Summary: Acceptance tests are flaky due to async pipeline creation
 Key: HDDS-2606
 URL: https://issues.apache.org/jira/browse/HDDS-2606
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Reporter: Marton Elek


Some of the acceptance tests are failing because the first Ratis.THREE pipeline 
is not created on time:

For example in a HA proxy acceptance test the first block allocation is tarted 
before moving the Ratis.THREE pipeline to OPEN state.
{code:java}
scm_1   | 2019-11-20 14:45:38,359 INFO pipeline.PipelineStateManager: 
Pipeline Pipeline[ Id: 4d27f405-d257-4b1b-a7b3-51bbd57356db, Nodes: 
2c010ef7-8c0e-45e3-b230-326bf759773b{ip: 172.25.0.7, host: 
ozones3-haproxy_datanode_2.ozones3-haproxy_default, networkLocation: 
/default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] moved 
to OPEN state
scm_1   | 2019-11-20 14:45:46,944 WARN block.BlockManagerImpl: Pipeline 
creation failed for type:RATIS factor:THREE. Retrying get pipelines call once.
scm_1   | 
org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot 
create pipeline of factor 3 using 0 nodes.
scm_1   |   at 
org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:153)
scm_1   |   at 
org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:58)
scm_1   |   at 
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:155)
scm_1   |   at 
org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:198)
scm_1   |   at 
org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:187)
scm_1   |   at 
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:159)
scm_1   |   at 
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
scm_1   |   at 
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
scm_1   |   at 
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:98)
scm_1   |   at 
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:13157)
scm_1   |   at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
scm_1   |   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
scm_1   |   at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
scm_1   |   at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
scm_1   |   at java.base/java.security.AccessController.doPrivileged(Native 
Method)
scm_1   |   at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
scm_1   |   at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
scm_1   |   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
scm_1   | 2019-11-20 14:45:46,944 INFO block.BlockManagerImpl: Could not 
find available pipeline of type:RATIS and factor:THREE even after retrying
scm_1   | 2019-11-20 14:45:46,945 ERROR block.BlockManagerImpl: Unable to 
allocate a block for the size: 268435456, type: RATIS, factor: THREE
scm_1   | 2019-11-20 14:45:49,274 INFO pipeline.PipelineStateManager: 
Pipeline Pipeline[ Id: e208588c-9a16-4519-89dc-7bad5bae4331, Nodes: 
1da10d32-12be-4328-bc0a-f3d8de21b056{ip: 172.25.0.8, host: 
ozones3-haproxy_datanode_3.ozones3-haproxy_default, networkLocation: 
/default-rack, certSerialId: null}d9edb776-ee02-48a1-8c73-40f33bc0d128{ip: 
172.25.0.5, host: ozones3-haproxy_datanode_1.ozones3-haproxy_default, 
networkLocation: /default-rack, certSerialId: 
null}2c010ef7-8c0e-45e3-b230-326bf759773b{ip: 172.25.0.7, host: 
ozones3-haproxy_datanode_2.ozones3-haproxy_default, networkLocation: 
/default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] moved 
to OPEN state {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2587) Enable sonarcloud measurement as part of CI builds

2019-11-20 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2587:
-

 Summary: Enable sonarcloud measurement as part of CI builds
 Key: HDDS-2587
 URL: https://issues.apache.org/jira/browse/HDDS-2587
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


As sonarcloud has been started to use, it would be great to upload measurement 
data from github actions steps...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2533) Disable failing acceptance and unit tests

2019-11-19 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2533.
---
Fix Version/s: 0.5.0
 Assignee: Marton Elek
   Resolution: Fixed

> Disable failing acceptance and unit tests
> -
>
> Key: HDDS-2533
> URL: https://issues.apache.org/jira/browse/HDDS-2533
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> As of now we have 40 pull requests in open change but the CI gate are broken 
> by earlier commit.
>  
> I propose the following solution to make it possible to safely merge pull 
> requests:
>  # Disable the failing tests (we identified them at 
> [https://github.com/apache/hadoop-ozone/pull/203)]
>  # Commit the immediate fixes (like HDDS-2521)
>  # Start a discussion on ozone-dev to use a more strict revert policy (revert 
> immediately)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2542) Race condition between read and write stateMachineData

2019-11-19 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2542:
-

 Summary: Race condition between read and write stateMachineData
 Key: HDDS-2542
 URL: https://issues.apache.org/jira/browse/HDDS-2542
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Reporter: Marton Elek


The write payload (the chunk itself) is sent to the Ratis as an external, 
binary byte array. It's not part of the LogEntry and saved from an async thread 
with calling ContainerStateMachine.writeStateMachineData

 

As it's an async thread it's possible that the stateMachineData is not yet 
written when the data should be sent to the followers in the next heartbeat.

By design a cache is used to avoid this issue but there are multiple problems 
with the cache.

First, the current cache size is chunkExecutor.getCorePoolSize() which is not 
enough. By default it means 10 executor thread and a cache with size 10. But in 
case of one very slow and nine very fast writer the cache entries can be 
invalidated before the write.

In my tests (freon datanode-chunk-writer-generator) I have seen missed cache 
hits even with cache size 5000.

Second: as the readStateMachineData and writeStateMachien data are called from 
two different thread there is a race condition independent from the the cache 
size. It's possible that the write thread has not yet added the data to the 
cache but the read thread needs it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2541) CI builds should use merged code state instead of the forked branch

2019-11-19 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2541:
-

 Summary: CI builds should use merged code state instead of the 
forked branch
 Key: HDDS-2541
 URL: https://issues.apache.org/jira/browse/HDDS-2541
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: build
Reporter: Marton Elek


As of now the github actions based CI runs uses the branch of the PR which is 
the forked repo most of the time.

It would be better to force a rebase/merge (without push) before the builds to 
test the possible state after the merge not before.

For example if a PR branch uses elek/hadoop-ozone:HDDS-1234 and request a merge 
to apache/hadoop-ozone:master then the build should download the HDDS-1234 from 
elek/hadoop-ozone AND *rebase/merge* to the apache/hadoop-ozone *before* the 
build.

This merge is temporary just for the build/checks (no push at all).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky

2019-11-18 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2535:
-

 Summary: TestOzoneManagerDoubleBufferWithOMResponse is flaky
 Key: HDDS-2535
 URL: https://issues.apache.org/jira/browse/HDDS-2535
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Manager
Reporter: Marton Elek


Flakiness can be reproduced locally. Usually it passes, but when I started to 
run it 100 times parallel with high cpu load it failed with the 3rd attempt 
(timed out)
{code:java}
---
Test set: 
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
---
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<< 
FAILURE! - in 
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
  Time elapsed: 500.122 s  <<< ERROR!
java.lang.Exception: test timed out after 50 milliseconds
at java.lang.Thread.sleep(Native Method)
at 
org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
at 
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
at 
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 {code}
Independent from the flakiness I think a test where the timeout is 8 minutes 
and starts 1000 threads to insert 500 buckets (500_000 buckets all together) 
it's more like an integration test and would be better to move the slowest part 
to the integration-test project.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2533) Disable failing acceptance and unit tests

2019-11-18 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2533:
-

 Summary: Disable failing acceptance and unit tests
 Key: HDDS-2533
 URL: https://issues.apache.org/jira/browse/HDDS-2533
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


As of now we have 40 pull requests in open change but the CI gate are broken by 
earlier commit.

 

I propose the following solution to make it possible to safely merge pull 
requests:
 # Disable the failing tests (we identified them at 
[https://github.com/apache/hadoop-ozone/pull/203)]
 # Commit the immediate fixes (like HDDS-2521)
 # Start a discussion on ozone-dev to use a more strict revert policy (revert 
immediately)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2495) Sonar - "notify" may not wake up the appropriate thread

2019-11-15 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2495.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> Sonar - "notify" may not wake up the appropriate thread
> ---
>
> Key: HDDS-2495
> URL: https://issues.apache.org/jira/browse/HDDS-2495
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Matthew Sharp
>Assignee: Matthew Sharp
>Priority: Minor
>  Labels: pull-request-available, sonar
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Addresses same issue within ReplicationManager:
> [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-sVKcVY8lQ4ZsDi=AW5md-sVKcVY8lQ4ZsDi]
> [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-sVKcVY8lQ4ZsDh=AW5md-sVKcVY8lQ4ZsDh]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2507) Remove the hard-coded exclusion of TestMiniChaosOzoneCluster

2019-11-15 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2507:
-

 Summary: Remove the hard-coded exclusion of 
TestMiniChaosOzoneCluster
 Key: HDDS-2507
 URL: https://issues.apache.org/jira/browse/HDDS-2507
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


We excluded the execution of TestMiniChaosOzoneCluster from the 
hadoop-ozone/dev-support/checks/integration.sh because it was not stable enough.

Unfortunately this exclusion makes it impossible to use custom exclusion lists 
(-Dsurefire.excludesFile=) as excludesFile can't be used if -Dtest=!... is 
already used.

I propose to remove this exclusion to make it possible to use different 
exclusion for different runs (pr check, daily, etc.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2506) Remove keyAllocationInfo and replication info from the auditLog

2019-11-15 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2506:
-

 Summary: Remove keyAllocationInfo and replication info from the 
auditLog
 Key: HDDS-2506
 URL: https://issues.apache.org/jira/browse/HDDS-2506
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Manager
Reporter: Marton Elek


During the review of HDDS-2470 I found that the full keyLocationInfo is added 
to the audit log for s3 operations:

 
{code:java}

2019-11-15 12:34:18,538 | INFO  | OMAudit | user=hadoop | ip=192.168.16.2 | 
op=ALLOCATE_KEY {volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, 
key=key1, dataSize=3813, replicationType=RATIS, replicationFactor=ONE, 
keyLocationInfo=[]} | ret=SUCCESS |  2019-11-15 12:34:20,576 | INFO  | OMAudit 
| user=hadoop | ip=192.168.16.2 | op=ALLOCATE_KEY 
{volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, key=key1, 
dataSize=3813, replicationType=RATIS, replicationFactor=ONE, 
keyLocationInfo=[]} | ret=SUCCESS |  2019-11-15 12:34:20,626 | INFO  | OMAudit 
| user=hadoop | ip=192.168.16.2 | op=ALLOCATE_BLOCK 
{volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, key=key1, 
dataSize=3813, replicationType=RATIS, replicationFactor=THREE, 
keyLocationInfo=[], clientID=103141950132977668} | ret=SUCCESS |  2019-11-15 
12:34:51,705 | INFO  | OMAudit | user=hadoop | ip=192.168.16.2 | 
op=COMMIT_MULTIPART_UPLOAD_PARTKEY {volume=s3b607288814a5da737a92fb067500396e, 
bucket=bucket1, key=key1, dataSize=3813, replicationType=RATIS, 
replicationFactor=ONE, keyLocationInfo=[blockID {  containerBlockID {
containerID: 1localID: 103141950135009280  }  blockCommitSequenceId: 
2}offset: 0length: 3813createVersion: 0pipeline {  members {uuid: 
"eefe54e8-5723-458e-9204-207c6b97c9b3"ipAddress: "192.168.16.3"
hostName: "ozones3_datanode_1.ozones3_default"ports {  name: "RATIS"
  value: 9858}ports {  name: "STANDALONE"  value: 9859}
networkName: "eefe54e8-5723-458e-9204-207c6b97c9b3"networkLocation: 
"/default-rack"  }  members {uuid: "ebf127d7-90a9-4f06-8fe5-a0c9c9adb743"   
 ipAddress: "192.168.16.7"hostName: "ozones3_datanode_2.ozones3_default"
ports {  name: "RATIS"  value: 9858}ports {  name: 
"STANDALONE"  value: 9859}networkName: 
"ebf127d7-90a9-4f06-8fe5-a0c9c9adb743"networkLocation: "/default-rack"  }  
members {uuid: "9979c326-4982-4a4c-b34e-e70c1d825f5f"ipAddress: 
"192.168.16.6"hostName: "ozones3_datanode_3.ozones3_default"ports { 
 name: "RATIS"  value: 9858}ports {  name: "STANDALONE"  
value: 9859}networkName: "9979c326-4982-4a4c-b34e-e70c1d825f5f"
networkLocation: "/default-rack"  }  state: PIPELINE_OPEN  type: RATIS  factor: 
THREE  id {id: "69ba305b-fe89-4f5c-97cd-b894d5ee8f2b"  }  leaderID: ""}], 
partNumber=1, 
partName=/s3b607288814a5da737a92fb067500396e/bucket1/key1103141950132977668} | 
ret=SUCCESS |  2019-11-15 12:42:10,883 | INFO  | OMAudit | user=hadoop | 
ip=192.168.16.2 | op=COMPLETE_MULTIPART_UPLOAD 
{volume=s3b607288814a5da737a92fb067500396e, bucket=bucket1, key=key1, 
dataSize=0, replicationType=RATIS, replicationFactor=ONE, keyLocationInfo=[], 
multipartList=[partNumber: 1partName: 
"/s3b607288814a5da737a92fb067500396e/bucket1/key1103141950132977668"]} | 
ret=SUCCESS |  
 {code}
Including the full keyLocation info in the audit log may cause some problems:
 * It makes the the audit log slower
 * It makes harder to parse the audit log

I think it's better to separate the debug log (which can be provided easily 
with ozone insight tool) from the audit log. Therefore I suggest to remove the 
keyLocationInfo, replicationType, replicationFactor from the aduit log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2482) Enable github actions for pull requests

2019-11-15 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2482.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> Enable github actions for pull requests
> ---
>
> Key: HDDS-2482
> URL: https://issues.apache.org/jira/browse/HDDS-2482
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HDDS-2400 introduced a github actions workflow for each "push" event. It 
> turned out that pushing to a forked repository doesn't trigger this event 
> even if it's part of a PR.
>  
> We need to enable the execution for pull_request events:
> References:
>  
> [https://github.community/t5/GitHub-Actions/Run-a-GitHub-action-on-pull-request-for-PR-opened-from-a-forked/m-p/31147#M690]
> [https://help.github.com/en/actions/automating-your-workflow-with-github-actions/events-that-trigger-workflows#pull-request-events-for-forked-repositories]
> {noformat}
> Note: By default, a workflow only runs when a pull_request's activity type is 
> opened, synchronize, or reopened. To trigger workflows for more activity 
> types, use the types keyword.{noformat}
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2482) Enable github actions for pull requests

2019-11-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2482:
-

 Summary: Enable github actions for pull requests
 Key: HDDS-2482
 URL: https://issues.apache.org/jira/browse/HDDS-2482
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


HDDS-2400 introduced a github actions workflow for each "push" event. It turned 
out that pushing to a forked repository doesn't trigger this event even if it's 
part of a PR.

 

We need to enable the execution for pull_request events:

References:

 
[https://github.community/t5/GitHub-Actions/Run-a-GitHub-action-on-pull-request-for-PR-opened-from-a-forked/m-p/31147#M690]

[https://help.github.com/en/actions/automating-your-workflow-with-github-actions/events-that-trigger-workflows#pull-request-events-for-forked-repositories]
{noformat}
Note: By default, a workflow only runs when a pull_request's activity type is 
opened, synchronize, or reopened. To trigger workflows for more activity types, 
use the types keyword.{noformat}
 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2461) Logging by ChunkUtils is misleading

2019-11-12 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2461:
-

 Summary: Logging by ChunkUtils is misleading
 Key: HDDS-2461
 URL: https://issues.apache.org/jira/browse/HDDS-2461
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Reporter: Marton Elek


During a k8s based test I found a lot of log message like:
{code:java}
2019-11-12 14:27:13 WARN  ChunkManagerImpl:209 - Duplicate write chunk request. 
Chunk overwrite without explicit request. 
ChunkInfo{chunkName='A9UrLxiEUN_testdata_chunk_4465025, offset=0, len=1024} 
{code}
I was very surprised as at ChunkManagerImpl:209 there was no similar lines.

It turned out that it's logged by ChunkUtils but it's used the logger of 
ChunkManagerImpl.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2325) BenchMarkDatanodeDispatcher genesis test is failing with NPE

2019-11-11 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2325.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> BenchMarkDatanodeDispatcher genesis test is failing with NPE
> 
>
> Key: HDDS-2325
> URL: https://issues.apache.org/jira/browse/HDDS-2325
> Project: Hadoop Distributed Data Store
>  Issue Type: New Feature
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> ## What changes were proposed in this pull request?
> Genesis is a microbenchmark tool for Ozone based on JMH 
> ([https://openjdk.java.net/projects/code-tools/jmh/).]
>  
> Due to the recent Datanode changes the BenchMarkDatanodeDispatcher is failing 
> with NPE:
>  
> {code:java}
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.ozone.container.common.interfaces.Handler.(Handler.java:69)
>   at 
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.(KeyValueHandler.java:114)
>   at 
> org.apache.hadoop.ozone.container.common.interfaces.Handler.getHandlerForContainerType(Handler.java:78)
>   at 
> org.apache.hadoop.ozone.genesis.BenchMarkDatanodeDispatcher.initialize(BenchMarkDatanodeDispatcher.java:115)
>   at 
> org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest._jmh_tryInit_f_benchmarkdatanodedispatcher0_G(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:438)
>   at 
> org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest.createContainer_Throughput(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:71)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
>   at 
> org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
>  {code}
> And this is the just the biggest problem there are a few other problems. I 
> propose the following fixes:
> *fix 1*: NPE is thrown because the 'context' object is required by 
> KeyValueHandler/Handler classes.
> In fact the context is not required, we need two functionalities/info from 
> the context: the ability to send icr (IncrementalContainerReport) and the ID 
> of the datanode.
> Law of Demeter principle suggests to have only the minimum required 
> information from other classes.
> For example instead of having context but using only 
> context.getParent().getDatanodeDetails().getUuidString() we can have only the 
> UUID string which makes more easy to test (unit and benchmark) the 
> Handler/KeyValueHandler.
> This is the biggest (but still small change) in this patch: I started to use 
> the datanodeId and an icrSender instead of having the full context.
> *fix 2,3:* There were a few other problems. The scmId was missing if the 
> writeChunk was called from Benchmark and and the Checksum was also missing.
> *fix 4:* I also had a few other problems: very huge containers are used 
> (default 5G) and as the benchmark starts with creating 100 containers it 
> requires 500G space by default. I adjusted the container size to make it 
> possible to run on local machine.
>  
> ## How this patch can be tested?
> {code:java}
> ./ozone genesis -benchmark=BenchMarkDatanodeDispatcher.writeChunk{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1701) Move dockerbin script to libexec

2019-11-08 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-1701.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> Move dockerbin script to libexec
> 
>
> Key: HDDS-1701
> URL: https://issues.apache.org/jira/browse/HDDS-1701
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Eric Yang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Ozone tarball structure contains a new bin script directory called dockerbin. 
>  These utility script can be relocated to OZONE_HOME/libexec because they are 
> internal binaries that are not intended to be executed directly by users or 
> shell scripts.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2444) Remove server side dependencies from ozonefs jar files

2019-11-08 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2444:
-

 Summary: Remove server side dependencies from ozonefs jar files
 Key: HDDS-2444
 URL: https://issues.apache.org/jira/browse/HDDS-2444
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: Ozone Filesystem
Reporter: Marton Elek


During the review of HDDS-2427 we found that some of the server side 
dependencies (container-service, framework) are added to the ozonefs library 
jars. Server side dependencies should be excluded from the client side to make 
the client safer and the build faster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2414) Simplify robot tests with removing output greps

2019-11-07 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2414:
-

 Summary: Simplify robot tests with removing output greps
 Key: HDDS-2414
 URL: https://issues.apache.org/jira/browse/HDDS-2414
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek


The robot tests under hadoop-ozone/dist/src/main/smoketest contain a lot of 
grep fragments to filter out the output of the CLI commands:
{code:java}
ozone sh key list o3://om/fstest/bucket1 | grep -v WARN | jq -r '.name' {code}
It was introduced because we had some unrelated logs on the console.

Fortunately the ozone log4j.properties has been adjusted to hide the unrelated 
lines.

It would be great to remove all of these "grep -v " fragments to make sure 
that all the CLI work well with annoying logging.

For example the previous line should work as
{code:java}
ozone sh key list o3://om/fstest/bucket1 | jq -r '.name'  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2413) Set configuration variables from annotated java objects

2019-11-07 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2413:
-

 Summary: Set configuration variables from annotated java objects
 Key: HDDS-2413
 URL: https://issues.apache.org/jira/browse/HDDS-2413
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek
Assignee: Marton Elek


HDDS-1469 introduced a new method to handle configuration. Configuration can be 
injected directly to java objects which makes all the java constants 
unnecessary.

 

Almost.

 

To read the configuration it's enough to have an annotated java object. For 
example:

 
{code:java}
@ConfigGroup(prefix = "hdds.scm")
public class ScmConfig {
  private String principal;
  private String keytab;  @Config(key = "kerberos.principal",
type = ConfigType.STRING,
defaultValue = "",
tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
description = "This Kerberos principal is used by the SCM service."
  )
  public void setKerberosPrincipal(String kerberosPrincipal) {
this.principal = kerberosPrincipal; {code}
And the configuration can be set in ozone-site.xml

Unfortunately during the unit testing we need to inject the configuration 
variables programmatically which requires a String constant:
{code:java}
configuration.set(ScmConfig.ConfigStrings.HDDS_SCM_KERBEROS_PRINCIPAL_KEY,  
"scm/" + host + "@" + realm); {code}
I propose to implement a simple setter in the OzoneConfiguration which may help 
to set configuration based on an annotated configuration object instance:
{code:java}
OzoneConfiguration conf = new OzoneConfiguration();

SCMHTTPServerConfig httpConfig = SCMHTTPServerConfig(principal1,...);

conf.setFromObject(httpConfig){code}
This is the opposite direction of the existing OzoneConfiguration.getObject() 
and can be implemented with a similar approach.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2412) Define description/topics/merge strategy for the github repository with .asf.yaml

2019-11-07 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2412:
-

 Summary: Define description/topics/merge strategy for the github 
repository with .asf.yaml
 Key: HDDS-2412
 URL: https://issues.apache.org/jira/browse/HDDS-2412
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek
Assignee: Marton Elek


.asf.yaml helps to set different parameters on github repositories without 
admin privileges:

[https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories]

This basic .asf.yaml defines description/url/topics and the allowed merge 
buttons.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2327) Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis

2019-11-07 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2327.
---
Resolution: Fixed

> Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis
> --
>
> Key: HDDS-2327
> URL: https://issues.apache.org/jira/browse/HDDS-2327
> Project: Hadoop Distributed Data Store
>  Issue Type: New Feature
>  Components: freon
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> [~xyao] suggested during an offline talk to implement one additional Freon 
> test to test the ratis part only.
> It can use XceiverClientManager which creates a pure XceiverClientRatis. The 
> client can be used to generate chunks as the datanode accepts any container 
> id / block id.
> With this approach we can stress-test one selected ratis pipeline without 
> having full end2end overhead of the key creation (OM, SCM, etc.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2411) Create DataChunkValidator Freon test

2019-11-07 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2411:
-

 Summary: Create DataChunkValidator Freon test
 Key: HDDS-2411
 URL: https://issues.apache.org/jira/browse/HDDS-2411
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: freon
Reporter: Marton Elek


HDDS-2327 introduced a new load test which generates a lot of WriteChunk 
request.

As with other freon test (for example with. 
HadoopFsGenerator/HadoopFsValidator) we need an other load test for 
validation/read path.

It should be almost the same DatanodeChunkGenerator but it should read the 
first chunk and compare all the others (very similar to the HadoopFsValidator 
or OzoneClientKeyValidator)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2369) Fix typo in param description.

2019-11-07 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2369.
---
Resolution: Fixed

> Fix typo in param description.
> --
>
> Key: HDDS-2369
> URL: https://issues.apache.org/jira/browse/HDDS-2369
> Project: Hadoop Distributed Data Store
>  Issue Type: Task
>Reporter: YiSheng Lien
>Priority: Trivial
>  Labels: newbie, pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In many addAcl(), the annotation param acl should be
> {code}
> ozone acl to be added.
> {code}
> but now is 
> {code}
> ozone acl top be added.
> {code}
> The files as follows:
> {code}
> hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java
> 614:   * @param acl ozone acl top be added.
> hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
> 1029:   * @param acl ozone acl top be added.
> hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ObjectStore.java
> 453:   * @param acl ozone acl top be added.
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/IOzoneAcl.java
> 36:   * @param acl ozone acl top be added.
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/PrefixManagerImpl.java
> 96:   * @param acl ozone acl top be added.
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
> 481:   * @param acl ozone acl top be added.
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
> 379:   * @param acl ozone acl top be added.
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
> 1475:   * @param acl ozone acl top be added.
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
> 2868:   * @param acl ozone acl top be added.
> hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
> 486:   * @param acl ozone acl top be added.
> hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java
> 1405:   * @param acl ozone acl top be added.
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2410) ozoneperf docker-compose should use privileged containers

2019-11-07 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2410:
-

 Summary: ozoneperf docker-compose should use privileged containers
 Key: HDDS-2410
 URL: https://issues.apache.org/jira/browse/HDDS-2410
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek
Assignee: Marton Elek


The profiler 
[servlet|https://github.com/elek/hadoop-ozone/blob/master/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ProfileServlet.java]
 (which helps to run java profiler in the background and publishes the result 
on the web interface) requires privileged docker containers.

 

This flag is missing from the ozoneperf docker-compose cluster (which is 
designed to run performance tests).

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1515) Create ozone dev-support script to check hadolint violiations

2019-11-04 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-1515.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

Merged to the master. Thanks [~akkidx] the contribution.

> Create ozone dev-support script to check hadolint violiations
> -
>
> Key: HDDS-1515
> URL: https://issues.apache.org/jira/browse/HDDS-1515
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>    Reporter: Marton Elek
>Assignee: Akshesh Doshi
>Priority: Major
>  Labels: newbie, pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> hadoop-ozone/dev-support/checks/ directory contains helper scripts to execute 
> different code quality checks locally.
> They are different from yetus as they can be executed in an easy way and they 
> check _ALL_ the violation of the current code base. 
> We need to create a new script to check the 
> [hadolint|https://github.com/hadolint/hadolint] errors in the hadoop-ozone 
> and hadoop-hdds projects.
> The contracts of the check scripts:
> #  Exit code should define the result (0: passed, <>0 failed)
> # Violation should be printed out to the stdout
> We can assume that the hadolint is part of the development environment. For 
> jenkins we can put it to the image of the dev builds.
> As the check introduce zero-tolerance for the hadolint violations the biggest 
> issue here is to eliminate all of the existing issues.
> Thanks to [~eyang] for reporting that it's still missing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2400) Enable github actions based builds for oyone

2019-11-04 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2400:
-

 Summary: Enable github actions based builds for oyone
 Key: HDDS-2400
 URL: https://issues.apache.org/jira/browse/HDDS-2400
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: build
Reporter: Marton Elek
Assignee: Marton Elek


Current PR checks are executed in a private branch based on the scripts in 
[https://github.com/elek/argo-ozone]

but the results are stored in a public repositories:

[https://github.com/elek/ozone-ci-q4|https://github.com/elek/ozone-ci-q3]

[https://github.com/elek/ozone-ci-03]

 

As we discussed during the community builds, it would be great to use github 
actions (or any other cloud based build) to make all the build definitions more 
accessible for the community.

[~vivekratnavel] checked CircleCI which has better reporting capabilities. But 
INFRA has concerns about the permission model of circle-ci:
{quote}it is highly unlikley we will allow a bot to be able to commit code 
(whether or not that is the intention, allowing circle-ci will make this 
possible, and is a complete no)
{quote}
See:

https://issues.apache.org/jira/browse/INFRA-18131

[https://lists.apache.org/thread.html/af52e2a3e865c01596d46374e8b294f2740587dbd59d85e132429b6c@%3Cbuilds.apache.org%3E]

 

Fortunately we have a clear contract. Or build scripts are stored under 
_hadoop-ozone/dev-support/checks_ (return code show the result, details are 
printed out to the console output). It's very easy to experiment with different 
build systems.

 

Github action seems to be an obvious choice: it's integrated well with GitHub 
and it has more generous resource limitations.

 

With this Jira I propose to enable github actions based PR checks for a few 
tests (author, rat, unit, acceptance, checkstyle, findbugs) as an experiment.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2399) Update mailing list information in CONTRIBUTION and README files

2019-11-04 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2399:
-

 Summary: Update mailing list information in CONTRIBUTION and 
README files
 Key: HDDS-2399
 URL: https://issues.apache.org/jira/browse/HDDS-2399
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


We have new mailing lists:

 [ozone-...@hadoop.apache.org|mailto:ozone-...@hadoop.apache.org]

[ozone-iss...@hadoop.apache.org|mailto:ozone-iss...@hadoop.apache.org]

[ozone-comm...@hadoop.apache.org|mailto:ozone-comm...@hadoop.apache.org]

 

We need to update CONTRIBUTION.md and README.md to use ozone-dev instead of 
hdfs-dev (optionally we can mention the issues/commits lists, but only in 
CONTRIBUTION.md)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2219) Move all the ozone dist scripts/configs to one location

2019-10-31 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2219.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> Move all the ozone dist scripts/configs to one location
> ---
>
> Key: HDDS-2219
> URL: https://issues.apache.org/jira/browse/HDDS-2219
> Project: Hadoop Distributed Data Store
>  Issue Type: Task
>  Components: build
>    Reporter: Marton Elek
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: newbe, pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The hadoop distribution tar file contains jar files scripts and default 
> configuration files.
> The scripts and configuration files are stored in multiple separated projects 
> without any reason:
> {code:java}
> ls hadoop-hdds/common/src/main/bin/
> hadoop-config.cmd  hadoop-config.sh  hadoop-daemons.sh  hadoop-functions.sh  
> workers.sh
> ls hadoop-ozone/common/src/main/bin 
> ozone  ozone-config.sh  start-ozone.sh  stop-ozone.sh
> ls hadoop-ozone/common/src/main/shellprofile.d 
> hadoop-ozone.sh
> ls hadoop-ozone/dist/src/main/conf 
> dn-audit-log4j2.properties  log4j.properties  om-audit-log4j2.properties  
> ozone-shell-log4j.properties  ozone-site.xml  scm-audit-log4j2.properties
>  {code}
> All of these scripts can be moved to the hadoop-ozone/dist/src/shell
> hadoop-ozone/dist/dev-support/bin/dist-layout-stitching also should be 
> updated to copy all of them to the right place in the tar.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2348) remove log4j properties for org.apache.hadoop.ozone

2019-10-31 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2348.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> remove log4j properties for org.apache.hadoop.ozone
> ---
>
> Key: HDDS-2348
> URL: https://issues.apache.org/jira/browse/HDDS-2348
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Manager
>Affects Versions: 0.5.0
>Reporter: luhuachao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Properties int log4j.properties cause logger in package 
> org.apache.hadoop.ozone cannot write log to .log file ;such as OM startup_msg 
> .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2372) Datanode pipeline is failing with NoSuchFileException

2019-10-28 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2372:
-

 Summary: Datanode pipeline is failing with NoSuchFileException
 Key: HDDS-2372
 URL: https://issues.apache.org/jira/browse/HDDS-2372
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Marton Elek


Found it on a k8s based test cluster using a simple 3 node cluster and 
HDDS-2327 freon test. After a while the StateMachine become unhealthy after 
this error:
{code:java}
datanode-0 datanode java.util.concurrent.ExecutionException: 
java.util.concurrent.ExecutionException: 
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: 
java.nio.file.NoSuchFileException: 
/data/storage/hdds/2a77fab9-9dc5-4f73-9501-b5347ac6145c/current/containerDir0/1/chunks/gGYYgiTTeg_testdata_chunk_13931.tmp.2.20830
 {code}
Can be reproduced.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2371) Print out the ozone version during the startup instead of hadoop version

2019-10-28 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2371:
-

 Summary: Print out the ozone version during the startup instead of 
hadoop version
 Key: HDDS-2371
 URL: https://issues.apache.org/jira/browse/HDDS-2371
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


Ozone components printing out the current version during the startup:

 
{code:java}
STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG:   host = om/10.8.0.145
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 3.2.0
STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r 
e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-{code}
But as it's visible the build / compiled information is about hadoop not about 
hadoop-ozone.

(And personally I prefer to use a github compatible url instead of the SVN 
style -r. Something like:
{code:java}
STARTUP_MSG: build =  
https://github.com/apache/hadoop-ozone/commit/8541c5694efebb58f53cf4665d3e4e6e4a12845c
 ; compiled by '' on ...{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2287) Move ozone source code to apache/hadoop-ozone from apache/hadoop

2019-10-23 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2287.
---
Resolution: Fixed

> Move ozone source code to apache/hadoop-ozone from apache/hadoop
> 
>
> Key: HDDS-2287
> URL: https://issues.apache.org/jira/browse/HDDS-2287
> Project: Hadoop Distributed Data Store
>  Issue Type: Task
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>
> *This issue is created to use the assigned number for any technical commits 
> to make it easy to follow the root reason of the commit...*
>  
> As discussed and voted on the mailing lists, Apache Hadoop Ozone source code 
> will be removed from the hadoop trunk and stored in a separated repository.
>  
> Original discussion is here:
> [https://lists.apache.org/thread.html/ef01b7def94ba58f746875999e419e10645437423ab9af19b32821e7@%3Chdfs-dev.hadoop.apache.org%3E]
> (It's started as a discussion but as everybody started to vote it's finished 
> with a call to a lazy consensus vote)
>  
> Technical proposal is shared on the wiki: 
> [https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Ozone+source+tree+split]
>  
> Discussed on the community meeting: 
> [https://cwiki.apache.org/confluence/display/HADOOP/2019-09-30+Meeting+notes]
>  
> Which is shared on the mailing list to get more feedback: 
> [https://lists.apache.org/thread.html/ed608c708ea302675ae5e39636ed73613f47a93c2ddfbd3c9e24dbae@%3Chdfs-dev.hadoop.apache.org%3E]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2327) Provide new Freon test to test Ratis pipeline with pure XceiverClientRatis

2019-10-18 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2327:
-

 Summary: Provide new Freon test to test Ratis pipeline with pure 
XceiverClientRatis
 Key: HDDS-2327
 URL: https://issues.apache.org/jira/browse/HDDS-2327
 Project: Hadoop Distributed Data Store
  Issue Type: New Feature
  Components: freon
Reporter: Marton Elek
Assignee: Marton Elek


[~xyao] suggested during an offline talk to implement one additional Freon test 
to test the ratis part only.

It can use XceiverClientManager which creates a pure XceiverClientRatis. The 
client can be used to generate chunks as the datanode accepts any container id 
/ block id.

With this approach we can stress-test one selected ratis pipeline without 
having full end2end overhead of the key creation (OM, SCM, etc.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2326) Http server of Freon is not started for new Freon tests

2019-10-18 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2326:
-

 Summary: Http server of Freon is not started for new Freon tests
 Key: HDDS-2326
 URL: https://issues.apache.org/jira/browse/HDDS-2326
 Project: Hadoop Distributed Data Store
  Issue Type: New Feature
  Components: freon
Reporter: Marton Elek
Assignee: Marton Elek


HDDS-2022 introduced new Freon tests but the Freon http server is not started 
for the new tests.

Freon includes a http server which can be turned on with the '–server' flag. It 
helps to monitor and profile the freon as the http server contains by default 
the prometheus and profiler servlets.

The server should be started if's requested.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2325) BenchMarkDatanodeDispatcher genesis test is failing with NPE

2019-10-18 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2325:
-

 Summary: BenchMarkDatanodeDispatcher genesis test is failing with 
NPE
 Key: HDDS-2325
 URL: https://issues.apache.org/jira/browse/HDDS-2325
 Project: Hadoop Distributed Data Store
  Issue Type: New Feature
Reporter: Marton Elek
Assignee: Marton Elek


## What changes were proposed in this pull request?

Genesis is a microbenchmark tool for Ozone based on JMH 
([https://openjdk.java.net/projects/code-tools/jmh/).]

 

Due to the recent Datanode changes the BenchMarkDatanodeDispatcher is failing 
with NPE:

 
{code:java}
java.lang.NullPointerException
at 
org.apache.hadoop.ozone.container.common.interfaces.Handler.(Handler.java:69)
at 
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.(KeyValueHandler.java:114)
at 
org.apache.hadoop.ozone.container.common.interfaces.Handler.getHandlerForContainerType(Handler.java:78)
at 
org.apache.hadoop.ozone.genesis.BenchMarkDatanodeDispatcher.initialize(BenchMarkDatanodeDispatcher.java:115)
at 
org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest._jmh_tryInit_f_benchmarkdatanodedispatcher0_G(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:438)
at 
org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest.createContainer_Throughput(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
at 
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
 {code}
And this is the just the biggest problem there are a few other problems. I 
propose the following fixes:

*fix 1*: NPE is thrown because the 'context' object is required by 
KeyValueHandler/Handler classes.

In fact the context is not required, we need two functionalities/info from the 
context: the ability to send icr (IncrementalContainerReport) and the ID of the 
datanode.

Law of Demeter principle suggests to have only the minimum required information 
from other classes.

For example instead of having context but using only 
context.getParent().getDatanodeDetails().getUuidString() we can have only the 
UUID string which makes more easy to test (unit and benchmark) the 
Handler/KeyValueHandler.

This is the biggest (but still small change) in this patch: I started to use 
the datanodeId and an icrSender instead of having the full context.

*fix 2,3:* There were a few other problems. The scmId was missing if the 
writeChunk was called from Benchmark and and the Checksum was also missing.

*fix 4:* I also had a few other problems: very huge containers are used 
(default 5G) and as the benchmark starts with creating 100 containers it 
requires 500G space by default. I adjusted the container size to make it 
possible to run on local machine.

 

## How this patch can be tested?
{code:java}
./ozone genesis -benchmark=BenchMarkDatanodeDispatcher.writeChunk{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2221) Monitor datanodes in ozoneperf compose cluster

2019-10-17 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2221.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> Monitor datanodes in ozoneperf compose cluster
> --
>
> Key: HDDS-2221
> URL: https://issues.apache.org/jira/browse/HDDS-2221
> Project: Hadoop Distributed Data Store
>  Issue Type: Task
>  Components: docker
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> ozoneperf compose cluster contains a prometheus but as of now it collects the 
> data only from scm and om.
> We don't know the exact number of datanodes (can be scaled up and down) 
> therefor it's harder to configure the datanode host names. I would suggest to 
> configure the first 10 datanodes (which covers most of the use cases)
> How to test?
> {code:java}
> cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozoneperf
> docker-compose up -d
> firefox http://localhost:9090/targets
>   {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2316) Support to skip recon and/or ozonefs during the build

2019-10-16 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2316:
-

 Summary: Support to skip recon and/or ozonefs during the build
 Key: HDDS-2316
 URL: https://issues.apache.org/jira/browse/HDDS-2316
 Project: Hadoop Distributed Data Store
  Issue Type: New Feature
Reporter: Anu Engineer
Assignee: Marton Elek


(I almost use this Jira summary: "Fast-lane to ozone build" It was very hard to 
resist...)

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2312) Fix typo in ozone command

2019-10-16 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2312.
---
Fix Version/s: 0.5.0
   Resolution: Fixed

> Fix typo in ozone command
> -
>
> Key: HDDS-2312
> URL: https://issues.apache.org/jira/browse/HDDS-2312
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Trivial
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> {noformat:title=ozone}
> Usage: ozone [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
> ...
> insight   tool to get runtime opeartion information
> ...
> {noformat}
> Should be "operation".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2308) Switch to centos with the apache/ozone-build docker image

2019-10-15 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2308:
-

 Summary: Switch to centos with the apache/ozone-build docker image
 Key: HDDS-2308
 URL: https://issues.apache.org/jira/browse/HDDS-2308
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek
 Attachments: hs_err_pid16346.log

I realized multiple JVM crashes in the daily builds:

 
{code:java}

ERROR] ExecutionException The forked VM terminated without properly saying 
goodbye. VM crash or System.exit called?
  
  
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && 
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError -jar 
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
 /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 
surefire7569723928289175829tmp surefire_947955725320624341206tmp
  
  
[ERROR] Error occurred in starting fork, check output in log
  
  
[ERROR] Process Exit Code: 139
  
  
[ERROR] Crashed tests:
  
  
[ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
  
  
[ERROR] ExecutionException The forked VM terminated without properly 
saying goodbye. VM crash or System.exit called?
  
  
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && 
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError -jar 
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar
 /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 
surefire7227403571189445391tmp surefire_1011197392458143645283tmp
  
  
[ERROR] Error occurred in starting fork, check output in log
  
  
[ERROR] Process Exit Code: 139
  
  
[ERROR] Crashed tests:
  
  
[ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp
  
  
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: 
ExecutionException The forked VM terminated without properly saying goodbye. VM 
crash or System.exit called?
  
  
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && 
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError -jar 
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar
 /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 
surefire3938612864214747736tmp surefire_933162535733309260236tmp
  
  
[ERROR] Error occurred in starting fork, check output in log
  
  
[ERROR] Process Exit Code: 139
  
  
[ERROR] ExecutionException The forked VM terminated without properly 
saying goodbye. VM crash or System.exit called?
  
  
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && 
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError -jar 
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
 /workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1 
surefire7569723928289175829tmp surefire_947955725320624341206tmp
  
  
[ERROR] Error occurred in starting fork, check output in log
  
  
[ERROR] Process Exit Code: 139 {code}
 

Based on the crash log (uploaded) it's related to the rocksdb JNI interface.

In the current ozone-build docker image (which provides the environment for 
build) we use alpine where musl libc is used instead of the main glibc. I think 
it would be more safe to use the same glibc what is used in production.

I tested with centos based docker image and it seems to be more stable. Didn't 
see any more JVM crashes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2302) Manage common pom versions in one common place

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2302:
-

 Summary: Manage common pom versions in one common place
 Key: HDDS-2302
 URL: https://issues.apache.org/jira/browse/HDDS-2302
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: build
Reporter: Marton Elek
Assignee: Marton Elek


Some of the versions (eg. ozone.version, hdds.version, ratis.version) are 
required for both ozone and hdds subprojects. As we have a common pom.xml it 
can be safer to manage them in one common place at the root pom.xml instead of 
managing them multiple times.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2300) Publish normalized Ratis metrics via the prometheus endpoint

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2300:
-

 Summary: Publish normalized Ratis metrics via the prometheus 
endpoint
 Key: HDDS-2300
 URL: https://issues.apache.org/jira/browse/HDDS-2300
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Marton Elek
Assignee: Marton Elek


Latest Ratis contains very good metrics about the status of the ratis ring.

After RATIS-702 it will be possible to adjust the repoter of the Dropwizard 
based ratis metrics and export them directly to the /prom http endpoint (used 
by ozone insight and ratis).

Unfortunately Dropwizard is very simple, there is no tag support. All of the 
instance specific strings are part of the metric name. For example:
{code:java}
"ratis_grpc.log_appender.72caaf3a-fb1c-4da4-9cc0-a2ce21bb8e67@group"
 + "-72caaf3a-fb1c-4da4-9cc0-a2ce21bb8e67"
 + ".grpc_log_appender_follower_75fa730a-59f0-4547"
 + "-bd68-216162c263eb_latency", {code}
In this patch I will use a simple method: during the export of the dropwizard 
metrics based on the well known format of the ratis metrics, they are converted 
to proper prometheus metrics where the instance information is included as tags:
{code:java}
ratis_grpc.log_appender.grpc_log_appender_follower_latency{instance="72caaf3a-fb1c-4da4-9cc0-a2ce21bb8e67"}
 {code}
With this approach we can:

 1. monitor easily all the Ratis pipelines with one simple query

 2. Use the metrics for ozone insight which will show health state of the Ratis 
pipeline



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2298) Fix maven warning about duplicated metrics-core jar

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2298:
-

 Summary: Fix maven warning about duplicated metrics-core jar
 Key: HDDS-2298
 URL: https://issues.apache.org/jira/browse/HDDS-2298
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: build
Reporter: Marton Elek
Assignee: Marton Elek


Maven build of Ozone is starting with a warning:
{code:java}
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hadoop:hadoop-ozone-tools:jar:0.5.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must 
be unique: io.dropwizard.metrics:metrics-core:jar -> version 3.2.4 vs (?) @ 
line 94, column 17
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten 
the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support 
building such malformed projects.
[WARNING] 
 {code}
It's better to avoid it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2297) Enable Opentracing for new Freon tests

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2297:
-

 Summary: Enable Opentracing for new Freon tests
 Key: HDDS-2297
 URL: https://issues.apache.org/jira/browse/HDDS-2297
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: freon
Reporter: Marton Elek
Assignee: Marton Elek


HDDS-2022 introduced new freon tests, but the initial root span of opentracing 
is not created before the test execution. We need to enable opentracing to get 
better view about the executions of the new freon test.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2296) ozoneperf compose cluster shouln't start freon by default

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2296:
-

 Summary: ozoneperf compose cluster shouln't start freon by default
 Key: HDDS-2296
 URL: https://issues.apache.org/jira/browse/HDDS-2296
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: docker
Reporter: Marton Elek
Assignee: Marton Elek


During the original creation of the compose/ozoneperf we added an example freon 
execution to make it clean how the data can be generated. This freon process 
starts all the time when ozoneperf cluster is started (usually I notice it when 
my CPU starts to use 100% of the available resources).

Since the creation of this cluster definition we implemented multiple type of 
freon tests and it's hard predict which tests should be executed. I propose to 
remove the default execution of the random key generation but keep the 
opportunity to run any of the tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2295) Display log of freon on the standard output

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2295:
-

 Summary: Display log of freon on the standard output
 Key: HDDS-2295
 URL: https://issues.apache.org/jira/browse/HDDS-2295
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek


HDDS-2042 disabled the console logging for all of the ozone command line tools 
including freon.

But freon is different, it has a different error handling model. For freon we 
need all the log on the console.

 1. To follow all the different errors

 2. To get information about the used (random) prefix which can be reused 
during the validation phase.

 

I propose to restore the original behavior for Ozone.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2294) Create a new HISTORY.md in the new repository

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2294:
-

 Summary: Create a new HISTORY.md in the new repository
 Key: HDDS-2294
 URL: https://issues.apache.org/jira/browse/HDDS-2294
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek


During the apache/hadoop.git --> apache/hadoop-ozone.git move we rewrote the 
history to simplify the work. Unfortunately some of the early work of HDDS-7280 
is not part of the hadoop-ozone repository just the hadoop repository. We need 
to explain it in a separated file and show how the origin of Ozone can be found.

 

cc [~aengineer]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2293) Create a new CONTRIBUTION.md for the new repository

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2293:
-

 Summary: Create a new CONTRIBUTION.md for the new repository
 Key: HDDS-2293
 URL: https://issues.apache.org/jira/browse/HDDS-2293
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek


Github supports CONTIRUTION.md which is displayed during the creation of a new 
Github PR. We can copy the content of the wiki page about how to contribut / 
how to build.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2292) Create Ozone specific README.md to the new hadoop-ozone repository

2019-10-14 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2292:
-

 Summary: Create Ozone specific README.md to the new hadoop-ozone 
repository
 Key: HDDS-2292
 URL: https://issues.apache.org/jira/browse/HDDS-2292
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek


The current README is main Hadoop specific. We can create an ozone specific.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2289) Put testing information and a problem description to the github PR template

2019-10-13 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2289:
-

 Summary: Put testing information and a problem description to the 
github PR template
 Key: HDDS-2289
 URL: https://issues.apache.org/jira/browse/HDDS-2289
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Anu Engineer
Assignee: Marton Elek


This is suggested by [~aengineer] during an offline discussion to add more 
information to the github PR template based on the template of ambari (by 
Vivek):

https://github.com/apache/ambari/commit/579cec8cf5bcfe1a1a0feacf055ed6569f674e6a



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2288) Delete hadoop-ozone and hadoop-hdds subprojects from apache trunk

2019-10-12 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2288:
-

 Summary: Delete hadoop-ozone and hadoop-hdds subprojects from 
apache trunk
 Key: HDDS-2288
 URL: https://issues.apache.org/jira/browse/HDDS-2288
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek
Assignee: Marton Elek


As described in the HDDS-2287 ozone/hdds sources are moving to the 
apache/hadoop-ozone git repository.

All the remaining ozone/hdds files can be removed from trunk (including hdds 
profile in main pom.xml)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2287) Move ozone source code to apache/hadoop-ozone from apache/hadoop

2019-10-12 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2287:
-

 Summary: Move ozone source code to apache/hadoop-ozone from 
apache/hadoop
 Key: HDDS-2287
 URL: https://issues.apache.org/jira/browse/HDDS-2287
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek
Assignee: Marton Elek


As discussed and voted on the mailing lists, Apache Hadoop Ozone source code 
will be removed from the hadoop trunk and stored in a separated repository.

 

Original discussion is here:

[https://lists.apache.org/thread.html/ef01b7def94ba58f746875999e419e10645437423ab9af19b32821e7@%3Chdfs-dev.hadoop.apache.org%3E]

(It's started as a discussion but as everybody started to vote it's finished 
with a call to a lazy consensus vote)

 

Technical proposal is shared on the wiki: 
[https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Ozone+source+tree+split]

 

Discussed on the community meeting: 
[https://cwiki.apache.org/confluence/display/HADOOP/2019-09-30+Meeting+notes]

 

Which is shared on the mailing list to get more feedback: 
[https://lists.apache.org/thread.html/ed608c708ea302675ae5e39636ed73613f47a93c2ddfbd3c9e24dbae@%3Chdfs-dev.hadoop.apache.org%3E]

 

This issue is created to use the assigned number for any technical commits to 
make it easy to follow the root reason of the commit...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2251) Add an option to customize unit.sh and integration.sh parameters

2019-10-05 Thread Marton Elek (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Elek resolved HDDS-2251.
---
Resolution: Fixed

> Add an option to customize unit.sh and integration.sh parameters
> 
>
> Key: HDDS-2251
> URL: https://issues.apache.org/jira/browse/HDDS-2251
> Project: Hadoop Distributed Data Store
>  Issue Type: Task
>    Reporter: Marton Elek
>    Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> hadoop-ozone/dev-support/checks/unit.sh (and same with integration) provides 
> an easy entrypoint to execute all the unit/integration test. But in same 
> cases it would be great to use the script but further specify the scope of 
> the test.
> With this simple patch it will be possible to adjust the surefire parameters.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2251) Add an option to customize unit.sh and integration.sh parameters

2019-10-04 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2251:
-

 Summary: Add an option to customize unit.sh and integration.sh 
parameters
 Key: HDDS-2251
 URL: https://issues.apache.org/jira/browse/HDDS-2251
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek
Assignee: Marton Elek


hadoop-ozone/dev-support/checks/unit.sh (and same with integration) provides an 
easy entrypoint to execute all the unit/integration test. But in same cases it 
would be great to use the script but further specify the scope of the test.

With this simple patch it will be possible to adjust the surefire parameters.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2237) KeyDeletingService throws NPE if it's started too early

2019-10-03 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2237:
-

 Summary: KeyDeletingService throws NPE if it's started too early
 Key: HDDS-2237
 URL: https://issues.apache.org/jira/browse/HDDS-2237
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: om
Reporter: Marton Elek
Assignee: Marton Elek


1. OzoneManager starts KeyManager

2. KeyManager starts KeyDeletingService

3. KeyDeletingService uses OzoneManager.isLeader()

4. OzoneManager.isLeader() uses omRatisServer

5. omRatisServer can be null (bumm)

 

Now the initialization order in OzoneManager:

 

new KeymanagerServer() *Includes start()*

omRatisServer initialization

start() (includes KeyManager.start())

 

The solution seems to be easy: start the key manager only from the 
OzoneManager.start() and not from the OzoneManager.instantiateServices()



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2232) ozone-build docker image is failing due to a missing entrypoint script

2019-10-02 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2232:
-

 Summary: ozone-build docker image is failing due to a missing 
entrypoint script
 Key: HDDS-2232
 URL: https://issues.apache.org/jira/browse/HDDS-2232
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: build, docker
Reporter: Marton Elek
Assignee: Marton Elek


We have a dedicated apache/ozone-build image which contains all of the required 
build and test tools to build ozone.

Unfortunately  it's not working as one shell script was not added to the 
original patch.

This patch (to the hadoop-docker-ozone repo!!) remove the requirement of the 
entrypoint.sh (no more docker in docker)

 

And installs additional tools (blockade, kubectl, mailsend)

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2221) Monitor datanodes in ozoneperf compose cluster

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2221:
-

 Summary: Monitor datanodes in ozoneperf compose cluster
 Key: HDDS-2221
 URL: https://issues.apache.org/jira/browse/HDDS-2221
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: docker
Reporter: Marton Elek


ozoneperf compose cluster contains a prometheus but as of now it collects the 
data only from scm and om.

We don't know the exact number of datanodes (can be scaled up and down) 
therefor it's harder to configure the datanode host names. I would suggest to 
configure the first 10 datanodes (which covers most of the use cases)

How to test?
{code:java}
cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozoneperf
docker-compose up -d
firefox http://localhost:9090/targets
  {code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2220) HddsVolume needs a toString method

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2220:
-

 Summary: HddsVolume needs a toString method
 Key: HDDS-2220
 URL: https://issues.apache.org/jira/browse/HDDS-2220
 Project: Hadoop Distributed Data Store
  Issue Type: Task
Reporter: Marton Elek


This is logged to the console of datanodes:
{code:java}
2019-10-01 11:37:59 INFO  HddsVolumeChecker:202 - Scheduled health check for 
volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 11:52:59 INFO  ThrottledAsyncChecker:139 - Scheduling a check for 
org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 11:52:59 INFO  HddsVolumeChecker:202 - Scheduled health check for 
volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:07:59 INFO  ThrottledAsyncChecker:139 - Scheduling a check for 
org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:07:59 INFO  HddsVolumeChecker:202 - Scheduled health check for 
volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:22:59 INFO  ThrottledAsyncChecker:139 - Scheduling a check for 
org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:22:59 INFO  HddsVolumeChecker:202 - Scheduled health check for 
volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:37:59 INFO  ThrottledAsyncChecker:139 - Scheduling a check for 
org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:37:59 INFO  HddsVolumeChecker:202 - Scheduled health check for 
volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:52:59 INFO  ThrottledAsyncChecker:139 - Scheduling a check for 
org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a
2019-10-01 12:52:59 INFO  HddsVolumeChecker:202 - Scheduled health check for 
volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5460cf3a 
{code}
Without a proper HddsVolume.toString it's hard to say which volume is checked...

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2219) Move all the ozone dist scripts/configs to one location

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2219:
-

 Summary: Move all the ozone dist scripts/configs to one location
 Key: HDDS-2219
 URL: https://issues.apache.org/jira/browse/HDDS-2219
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: build
Reporter: Marton Elek


The hadoop distribution tar file contains jar files scripts and default 
configuration files.

The scripts and configuration files are stored in multiple separated projects 
without any reason:
{code:java}
ls hadoop-hdds/common/src/main/bin/
hadoop-config.cmd  hadoop-config.sh  hadoop-daemons.sh  hadoop-functions.sh  
workers.sh

ls hadoop-ozone/common/src/main/bin 
ozone  ozone-config.sh  start-ozone.sh  stop-ozone.sh

ls hadoop-ozone/common/src/main/shellprofile.d 
hadoop-ozone.sh

ls hadoop-ozone/dist/src/main/conf 
dn-audit-log4j2.properties  log4j.properties  om-audit-log4j2.properties  
ozone-shell-log4j.properties  ozone-site.xml  scm-audit-log4j2.properties
 {code}
All of these scripts can be moved to the hadoop-ozone/dist/src/shell

hadoop-ozone/dist/dev-support/bin/dist-layout-stitching also should be updated 
to copy all of them to the right place in the tar.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2218) Use OZONE_CLASSPATH instead of HADOOP_CLASSPATH

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2218:
-

 Summary: Use OZONE_CLASSPATH instead of HADOOP_CLASSPATH
 Key: HDDS-2218
 URL: https://issues.apache.org/jira/browse/HDDS-2218
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: docker
Reporter: Marton Elek


HADOOP_CLASSPATH is the standard way to add additional jar files to the 
classpath of the mapreduce/spark/.. .jobs. If something is added to the 
HADOOP_CLASSPATH, than it should be on the classpath of the classic hadoop 
daemons.

But for the Ozone components we don't need any new jar files (cloud connectors, 
libraries). I think it's more safe to separated HADOOP_CLASSPATH from 
OZONE_CLASSPATH. If something is really need on the classpath for Ozone daemons 
the dedicated environment variable should be used.

 

Most probably it can be fixed in

hadoop-hdds/common/src/main/bin/hadoop-functions.sh

And the hadoop-ozone/dev/src/main/compose files also should be checked (some of 
them contain HADOOP_CLASSPATH



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2217) Remove log4j and audit configuration from the docker-config files

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2217:
-

 Summary: Remove log4j and audit configuration from the 
docker-config files
 Key: HDDS-2217
 URL: https://issues.apache.org/jira/browse/HDDS-2217
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: docker
Reporter: Marton Elek


Log4j configuration lines are added to the docker-config under 
hadoop-ozone/dist/src/main/compose/...

Mainly to make it easier to reconfigure the log level of any components.

As we already have a "ozone insight" tool which can help us to modify the log 
level at runtime we don't need these lines any more.
{code:java}

LOG4J.PROPERTIES_log4j.rootLogger=INFO, stdout
LOG4J.PROPERTIES_log4j.appender.stdout=org.apache.log4j.ConsoleAppender
LOG4J.PROPERTIES_log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern=%d{-MM-dd 
HH:mm:ss} %-5p %c{1}:%L - %m%n
LOG4J.PROPERTIES_log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
LOG4J.PROPERTIES_log4j.logger.org.apache.ratis.conf.ConfUtils=WARN
LOG4J.PROPERTIES_log4j.logger.org.apache.hadoop.security.ShellBasedUnixGroupsMapping=ERROR
LOG4J.PROPERTIES_log4j.logger.org.apache.ratis.grpc.client.GrpcClientProtocolClient=WARN
LOG4J.PROPERTIES_log4j.logger.http.requests.s3gateway=INFO,s3gatewayrequestlog
LOG4J.PROPERTIES_log4j.appender.s3gatewayrequestlog=org.apache.hadoop.http.HttpRequestLogAppender
LOG4J.PROPERTIES_log4j.appender.s3gatewayrequestlog.Filename=/tmp/jetty-s3gateway-_mm_dd.log
LOG4J.PROPERTIES_log4j.appender.s3gatewayrequestlog.RetainDays=3 {code}
We can remove them together with the audit log entries as we already have a 
default log4j.propertes / audit log4j2 config.

After the remove the clusters should be tested: Ozone CLI should not print and 
confusing log messages (such as NativeLib is missing or anything else). AFAIK 
they are already turned off in the etc/hadoop/etc log4j.properties.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2216) Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in compose .env files

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2216:
-

 Summary: Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in 
compose .env files
 Key: HDDS-2216
 URL: https://issues.apache.org/jira/browse/HDDS-2216
 Project: Hadoop Distributed Data Store
  Issue Type: Task
  Components: docker
Reporter: Marton Elek


In HDDS-1698 we replaced our apache/hadoop-runner base image to 
apache/ozone-runner base image.

 

The version of the image is set by the .env files under the 
hadoop-ozone/dist/src/main/compose directories
{code:java}
cd hadoop-ozone/dist/src/main/compose
grep -r HADOOP_RUNNER .
./ozoneperf/docker-compose.yaml:  image: 
apache/ozone-runner:${HADOOP_RUNNER_VERSION}
./ozoneperf/docker-compose.yaml:  image: 
apache/ozone-runner:${HADOOP_RUNNER_VERSION}
./ozoneperf/docker-compose.yaml:  image: 
apache/ozone-runner:${HADOOP_RUNNER_VERSION}
 {code}
But the name of the variable is HADOOP_RUNNER_VERSION instead of 
OZONE_RUNNER_VERSION.

Would be great to rename it the OZONE_RUNNER_VERSION.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2214) TestSCMContainerPlacementRackAware has an intermittent failure

2019-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2214:
-

 Summary: TestSCMContainerPlacementRackAware has an intermittent 
failure
 Key: HDDS-2214
 URL: https://issues.apache.org/jira/browse/HDDS-2214
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek


For example from the nightly build:
{code:java}
  
  
  
java.lang.AssertionError
   
  
at org.junit.Assert.fail(Assert.java:86)
  
  
at org.junit.Assert.assertTrue(Assert.java:41)
  
  
at org.junit.Assert.assertTrue(Assert.java:52)
  
  
at 
org.apache.hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware.testNoFallback(TestSCMContainerPlacementRackAware.java:276)
  
  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  
  
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  
  
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  
  
at java.lang.reflect.Method.invoke(Method.java:498)
  
  
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 {code}
The problem is in the testNoFallback:

Let's say we have 11 nodes (from parameter) and we would like to choose 5 nodes 
(hard coded in the test).

As the first two replicas are chosen from the same rack an all the other from 
different racks it's not possible, so we except a failure.

But we have an assertion that the success count is at least 3. But this is true 
only if the first two replicas are placed to the rack1 (5 nodes) or rack2 
(5nodes). If the replica is placed to the rack3 (one node) it will fail 
immediately:

 

Lucky case when we have success count > 3
{code:java}
 rack1 -- node1 
 rack1 -- node2 -- FIRST replica
 rack1 -- node3 -- SECOND replica
 rack1 -- node4
 rack1 -- node5 
 rack2 -- node6
 rack2 -- node7 -- THIRD replica
 rack2 -- node8
 rack2 -- node9 
 rack2 -- node10
 rack3 -- node11 -- FOURTH replica{code}
 The specific case when we have success count == 1, as we can't choose the 
second replica on rack3 (This is when the test is failing)
{code:java}
 rack1 -- node1 
 rack1 -- node2
 rack1 -- node3
 rack1 -- node4
 rack1 -- node5 
 rack2 -- node6
 rack2 -- node7
 rack2 -- node8
 rack2 -- node9 
 rack2 -- node10
 rack3 -- node11 -- FIRST replica{code}
 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2209) Checkstyle issue in OmUtils on trunk

2019-09-30 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2209:
-

 Summary: Checkstyle issue in OmUtils on  trunk
 Key: HDDS-2209
 URL: https://issues.apache.org/jira/browse/HDDS-2209
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Marton Elek


HDDS-2174 introduced a new checkstyle error:
{code:java}
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
 49: Unused import - org.apache.hadoop.ozone.om.OMMetadataManager. {code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2184) Rename ozone scmcli to ozone admin

2019-09-26 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2184:
-

 Summary: Rename ozone scmcli to ozone admin
 Key: HDDS-2184
 URL: https://issues.apache.org/jira/browse/HDDS-2184
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


Originally ozone scmcli designed to be used only by the developers. A very 
cryptic name is chosen intentionally to frighten away the beginner users.

As we realized recently we started to use "ozone scmcli" as a generic admin 
tool. More and more tools has been added which are useful not only for the 
developers but for the administrators.

Therefore I suggest to rename "ozone scmcli" to something more meaningful.

For example to "ozone admin"

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2183) Container and pipline subcommands of scmcli should be grouped

2019-09-26 Thread Marton Elek (Jira) 
Marton Elek created HDDS-2183:
-

 Summary: Container and pipline subcommands of scmcli should be 
grouped
 Key: HDDS-2183
 URL: https://issues.apache.org/jira/browse/HDDS-2183
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek


Once upon an time when we had only a few subcommands under `ozone scmcli` to 
manage containers.

 

Now we have many admin commands some of them are grouped to a subcommand (eg. 
safemode, replicationmanager) some of are not.

 

I propose to group the container and pipeline related commands:

 

Instead of "ozone scmcli info" use "ozone scmcli container info"

Instead of "ozone scmcli list" use "ozone scmcli container list"

Instead of "ozone scmcli listPipelines" use "ozone scmcli pipeline list"

 

And so on...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Marton, Elek 

+1 (non-binding)

* Built from the source
* Installed dockerized YARN/HADOOP cluster to a 20 node cluster 
(scheduled with nomad, configured from consul, docker host networking)
* Started example yarn jobs (teragen/terasort) and hdfs dfs commands + 
checking UIs


I noticed only two very minor issues (changelog of HADOOP-9902 didn't 
mention that I need a writable 'logs' dir, even with custom 
log4j.properties; and there was a space typo in yarn error message 
https://issues.apache.org/jira/browse/YARN-7279)


Marton

On 09/29/2017 02:04 AM, Andrew Wang wrote:

Hi all,

Let me start, as always, by thanking the many, many contributors who helped
with this release! I've prepared an RC0 for 3.0.0-beta1:

http://home.apache.org/~wang/3.0.0-beta1-RC0/

This vote will run five days, ending on Nov 3rd at 5PM Pacific.

beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
improvements, and feature enhancements. Notable additions include the
addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
shaded client, and HDFS erasure coding pluggable policy support.

I've done the traditional testing of running a Pi job on a pseudo cluster.
My +1 to start.

We're working internally on getting this run through our integration test
rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.

Best,
Andrew



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-22 Thread Marton, Elek 

Thanks all the feedbacks.

I created an issue:
https://issues.apache.org/jira/browse/HADOOP-14898

Let's continue the discussion there.

Thanks,
Marton

On 09/08/2017 02:45 PM, Marton, Elek wrote:


TL;DR: I propose to create official hadoop images and upload them to the 
dockerhub.


GOAL/SCOPE: I would like improve the existing documentation with 
easy-to-use docker based recipes to start hadoop clusters with various 
configuration.


The images also could be used to test experimental features. For example 
ozone could be tested easily with these compose file and configuration:


https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml 



I would like to create separated example compose files for federation, 
ha, metrics usage, etc. to make it easier to try out and understand the 
features.


CONTEXT: There is an existing Jira 
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images 
(multiple types, in a flexible way). If no objections, I will create a 
separated issue to create simplified docker images for rapid prototyping 
and investigating new features. And register the branch to the dockerhub 
to create the images automatically.


MY BACKGROUND: I am working with docker based hadoop/spark clusters 
quite a while and run them succesfully in different environments 
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is 
available from here: https://github.com/flokkr but they could handle 
more complex use cases (eg. instrumenting java processes with btrace, or 
read/reload configuration from consul).
  And IMHO in the official hadoop documentation it’s better to suggest 
to use official apache docker images and not external ones (which could 
be changed).


Please let me know if you have any comments.

Marton

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] official docker image(s) for hadoop

2017-09-08 Thread Marton, Elek 


TL;DR: I propose to create official hadoop images and upload them to the 
dockerhub.


GOAL/SCOPE: I would like improve the existing documentation with 
easy-to-use docker based recipes to start hadoop clusters with various 
configuration.


The images also could be used to test experimental features. For example 
ozone could be tested easily with these compose file and configuration:


https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml

I would like to create separated example compose files for federation, 
ha, metrics usage, etc. to make it easier to try out and understand the 
features.


CONTEXT: There is an existing Jira 
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images 
(multiple types, in a flexible way). If no objections, I will create a 
separated issue to create simplified docker images for rapid prototyping 
and investigating new features. And register the branch to the dockerhub 
to create the images automatically.


MY BACKGROUND: I am working with docker based hadoop/spark clusters 
quite a while and run them succesfully in different environments 
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is 
available from here: https://github.com/flokkr but they could handle 
more complex use cases (eg. instrumenting java processes with btrace, or 
read/reload configuration from consul).
 And IMHO in the official hadoop documentation it’s better to suggest 
to use official apache docker images and not external ones (which could 
be changed).


Please let me know if you have any comments.

Marton

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

hadoop roadmaps

2017-09-08 Thread Marton, Elek 

Hi,

I tried to summarize all of the information from different mail threads 
about the upcomming releases:


https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

Please fix it / let me know if you see any invalid data. I will try to 
follow the conversations and update accordingly.


Two administrative questions:

 * Is there any information about which wiki should be used? Or about 
the migration process? As I see the new pages are created on the cwiki 
recently.


 * Could you please give me permission (user: elek) to the old wiki. I 
would like to update the old Roadmap page 
(https://wiki.apache.org/hadoop/Roadmap)


Thanks
Marton

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Running HDFS from source broken since HDFS-11596

2017-07-27 Thread Marton, Elek 
Yeah, I suffer from the same problem as well. I will be very happy if 
it's fixed.


Usually I run all of the components from my IDE (namenode, datanode, 
ozone components etc.). I have two problems:


1. the provided scopes which was mentioned.

2. The components couldn't be started because the httpserver tries to 
locate the files on the classpath.


Usually I have branch with one commit which fixes all these problems 
(together with some default configuration files). I cherry-pick this fix 
to my feature branches and remove before the patch generation (as I am 
the only user of my private branches I can do interactive rebases and 
modify the history. (My current dirty commit is here: 
https://github.com/elek/hadoop/commit/5c14cb0a1697f50cfd359a3d57421376c92c7ce8)


I am just interested how the web development is handled by others. Do 
you compile the project every time? Do you run the components from IDE?


If you think it's usefull, I would be happy to post the first 
modification (HttpServer2 from 
https://github.com/elek/hadoop/commit/5c14cb0a1697f50cfd359a3d57421376c92c7ce8#diff-1d06d3e6ca2c0754572b021eb81a084d) 
as a patch. But my impression was that there should be some existing 
solution for the problem (improve the css/html parts of the servers from 
IDE without restart the component).


Marton



On 07/26/2017 01:51 AM, Andrew Wang wrote:

I looked into this more and filed HDFS-12197 with a possible solution.
Thanks for the report, Lars!

On Fri, Jul 21, 2017 at 12:51 AM, Lars Francke 
wrote:


Thanks John, that was helpful. I see that you're using the hadoop-dist
directory while the wiki points directly to the project folders (e.g.
hadoop-hdfs-project etc.).

The former works the latter doesn't. So I guess it's a matter of updating
the wiki.


On Thu, Jul 20, 2017 at 9:09 AM, John Zhuge  wrote:


Hi Lars,

I am able to run pseudo-distributed mode from a dev tree. Here is the
wiki: https://hadoop.apache.org/docs/current/hadoop-
project-dist/hadoop-common/SingleCluster.html#Pseudo-

Distributed_Operation

.

Check out my script pseudo_dist
 to

start/stop a pseudo-distributed cluster.

Here are the steps:

1. mvn install -DskipTests -DskipShade -Dmaven.javadoc.skip -Pdist
-Dtar
2. pseudo_dist start ~/hadoop-sanity-tests/config/insecure/
3. test_env hdfs dfs -ls /tmp

Thanks,

On Wed, Jul 19, 2017 at 11:49 PM, Lars Francke 
wrote:


I've already asked in )

and

as used to be possible before the patch. This is because the hdfs client
classes (e.g. ClientProtocol is the first one that HDFS complains about
during startup) are not in the classpath anymore.

I wonder how all of you are running Hadoop these days from source? I've
always followed the Wiki instructions but maybe they are out of date and
there's a better way?

Thanks,
Lars





--
John







-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-20 Thread Marton Elek 
+1 (non-binding)

Tested from the released binary package
* 5 node cluster running from dockerized containers (every 
namenode/datanode/nodemanager, etc. are running in separated containers) 
* Bitcoin bockchain data (~100Gb) parsed and imported to HBase (1.2.4)
* Spark (2.1.0 with included hadoop) job (executing on YARN) to query the data 
from HBase and write the results to HDFS

Looks good.

Marton


> On Mar 19, 2017, at 6:01 PM, Sunil Govind  wrote:
> 
> +1 (non-binding). Thanks Junping for the effort.
> 
> I have used release package and verified below cases
> - Ran MR sleep job and wordcount successfully in where nodes are configured
> with labels.
> - Verified application priority feature and I could see high priority apps
> are getting resource over lower priority apps when configured
> - Verified RM web UI pages and looks fine (priority could be seen)
> - Intra-queue preemption related to app priority also seems fine
> 
> Thanks
> Sunil
> 
> 
> On Fri, Mar 17, 2017 at 2:48 PM Junping Du  wrote:
> 
>> Hi all,
>> With fix of HDFS-11431 get in, I've created a new release candidate
>> (RC3) for Apache Hadoop 2.8.0.
>> 
>> This is the next minor release to follow up 2.7.0 which has been
>> released for more than 1 year. It comprises 2,900+ fixes, improvements, and
>> new features. Most of these commits are released for the first time in
>> branch-2.
>> 
>>  More information about the 2.8.0 release plan can be found here:
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>> 
>>  New RC is available at:
>> http://home.apache.org/~junping_du/hadoop-2.8.0-RC3
>> 
>>  The RC tag in git is: release-2.8.0-RC3, and the latest commit id
>> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>> 
>>  The maven artifacts are available via repository.apache.org at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1057
>> 
>>  Please try the release and vote; the vote will run for the usual 5
>> days, ending on 03/22/2017 PDT time.
>> 
>> Thanks,
>> 
>> Junping
>> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

2017-03-08 Thread Marton Elek 
I think the main point here is the testing of the release script, not the 
creation of the official release.

I think there should be an option to configure the release tool to use a forked 
github repo and/or a private playground nexus instead of official apache repos. 
In this case it would be easy to test regularly the tool, even by a 
non-committer (or even from Jenkins). But it would be just a smoketest of the 
release script...

Marton
   

From: Allen Wittenauer 
Sent: Wednesday, March 08, 2017 2:24 AM
To: Andrew Wang
Cc: Hadoop Common; yarn-...@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

> On Mar 7, 2017, at 2:51 PM, Andrew Wang  wrote:
> I think it'd be nice to
> have a nightly Jenkins job that builds an RC,

Just a reminder that any such build cannot be used for an actual 
release:

http://www.apache.org/legal/release-policy.html#owned-controlled-hardware



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

2017-03-08 Thread Marton Elek 

Thank you very much for your feedback. I am opening the following JIRAs:

1. Create dev-support scripts to do the bulk jira updates required by the 
releases (check remaining jiras, update fix versions, etc.)

2. Create a 'wizzard' like script which guide through the release process (all 
the steps from the wiki pages, not just a build. But it may be an extension of 
the existing script):

  Goals:
  * It would work even without the apache infrastructure: with custom 
configuration (forked repositories/alternative nexus), it would be possible to 
test the scripts even by a non-commiter.  
  * every step which could be automated should be scripted (create git 
branches, build,...). if something could be not automated there an explanation 
could be printed out, and wait for confirmation
  * Before dangerous steps (eg. bulk jira update) we can ask for confirmation 
and explain what will be happened (eg. the following jira items will be 
changed: ) 
  * The run should be idempontent (and there should be an option to continue 
the release from any steps).   

3. Migrate the forrest based home page to a use a modern static site generator.

  Goals: * existing links should work (or at least redirected)
 * It should be easy to add more content required by a release 
automatically

4. It's not about the release, but I think the current maven site theme also 
could be updated to use a (more modern) theme, which could be similar to the 
main site from step 3.

Let me know if you have any other suggestion for actionable items. Or comment 
the Jiras if you have more specific requirements.

Marton

ps: Vinod, I will contact with you, soon.




From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Tuesday, March 07, 2017 11:58 PM
To: Sangjin Lee
Cc: Marton Elek; common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
Hdfs-dev; mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

I was planning to take this up, celebrating my return from my paternity leave 
of absence for quite a while.

Marton, let me know if you do want to take this up instead and we can work 
together.

Thanks
+Vinod

> On Mar 7, 2017, at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> If we have a volunteer for releasing 2.7.4, we should go full speed
> ahead. We still need a volunteer from a PMC member or a committer as some
> tasks may require certain privileges, but I don't think it precludes
> working with others to close down the release.


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

2017-03-07 Thread Marton Elek 
Is there any reason to wait for 2.8 with 2.7.4?

Unfortunately the previous  thread about release cadence has been ended without 
final decision. But if I understood well, there was more or less an agreement 
about that it would be great to achieve more frequent releases, if possible 
(with or without written rules and EOL policy).

I personally prefer to be more closer to the scheduling part of the proposal:

"A minor release on the latest major line should be every 6 months, and a 
maintenance release on a minor release (as there may be concurrently
maintained minor releases) every 2 months".

I don't know what is the hardest part of creating new minor/maintenance 
releases. But if the problems are technical (smoketesting, unit tests, old 
release script, anything else) I would be happy to do any task for new 
maintenance releases (or more frequent releases).

Regards,
Marton

 

From: Akira Ajisaka 
Sent: Tuesday, March 07, 2017 7:34 AM
To: Brahma Reddy Battula; Hadoop Common; yarn-...@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Probably 2.8.0 will be released soon.
https://issues.apache.org/jira/browse/HADOOP-13866?focusedCommentId=15898379=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15898379

I'm thinking 2.7.4 release process starts after 2.8.0 release,
so 2.7.4 will be released in April or May. (hopefully)

Thoughts?

Regards,
Akira

On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> Hi All
>
> It has been six months for branch-2.7 release.. is there any near plan for 
> 2.7.4..?
>
>
> Thanks
> Brahma Reddy Battula
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

2017-01-25 Thread Marton Elek 
 Ran MapReduce jobs Pi
5. Verified Hadoop version command output is correct.

Best,

Yufei

On Tue, Jan 24, 2017 at 3:02 AM, Marton Elek 
<me...@hortonworks.com<mailto:me...@hortonworks.com>>
wrote:

]>
minicluster is kind of weird on filesystems that don't support mixed
case, like OS X's default HFS+.

$  jar tf hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar | grep
-i
license
LICENSE.txt
license/
license/LICENSE
license/LICENSE.dom-documentation.txt
license/LICENSE.dom-software.txt
license/LICENSE.sax.txt
license/NOTICE
license/README.dom.txt
license/README.sax.txt
LICENSE
Grizzly_THIRDPARTYLICENSEREADME.txt


I added a patch to https://issues.apache.org/jira/browse/HADOOP-14018 to
add the missing META-INF/LICENSE.txt to the shaded files.

Question: what should be done with the other LICENSE files in the
minicluster. Can we just exclude them (from legal point of view)?

Regards,
Marton

-
To unsubscribe, e-mail: 
yarn-dev-unsubscr...@hadoop.apache.org<mailto:yarn-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
yarn-dev-h...@hadoop.apache.org<mailto:yarn-dev-h...@hadoop.apache.org>

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

2017-01-24 Thread Marton Elek 
]> 
> minicluster is kind of weird on filesystems that don't support mixed case, 
> like OS X's default HFS+.
> 
> $  jar tf hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar | grep -i 
> license
> LICENSE.txt
> license/
> license/LICENSE
> license/LICENSE.dom-documentation.txt
> license/LICENSE.dom-software.txt
> license/LICENSE.sax.txt
> license/NOTICE
> license/README.dom.txt
> license/README.sax.txt
> LICENSE
> Grizzly_THIRDPARTYLICENSEREADME.txt


I added a patch to https://issues.apache.org/jira/browse/HADOOP-14018 to add 
the missing META-INF/LICENSE.txt to the shaded files.

Question: what should be done with the other LICENSE files in the minicluster. 
Can we just exclude them (from legal point of view)?

Regards,
Marton

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

2017-01-21 Thread Marton Elek 
hadoop-3.0.0-alpha2.tar.gz is much more smaller than 
hadoop-3.0.0-alpha1.tar.gz. (246M vs 316M)

The big difference is the generated source documentation:

find -name src-html
./hadoop-2.7.3/share/doc/hadoop/api/src-html
./hadoop-2.7.3/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/src-html
./hadoop-3.0.0-alpha1/share/doc/hadoop/api/src-html
./hadoop-3.0.0-alpha1/share/doc/hadoop/hadoop-hdfs-httpfs/apidocs/src-html
./hadoop-3.0.0-alpha1/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/api/src-html
./hadoop-3.0.0-alpha1/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs-client/api/src-html
./hadoop-3.0.0-alpha1/share/doc/hadoop/hadoop-project-dist/hadoop-common/api/src-html
(./hadoop-3.0.0-alpha-2 --> no match)

I am just wondering if it's intentional or not as I can't find any related jira 
or mail thread (maybe I missed it)

Regards,
Marton

On 01/20/2017 11:36 PM, Andrew Wang wrote:
> Hi all,
> 
> With heartfelt thanks to many contributors, the RC0 for 3.0.0-alpha2 is
> ready.
> 
> 3.0.0-alpha2 is the second alpha in the planned 3.0.0 release line leading
> up to a 3.0.0 GA. It comprises 857 fixes, improvements, and new features
> since alpha1 was released on September 3rd, 2016.
> 
> More information about the 3.0.0 release plan can be found here:
> 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
> 
> The artifacts can be found here:
> 
> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
> 
> This vote will run 5 days, ending on 01/25/2017 at 2PM pacific.
> 
> I ran basic validation with a local pseudo cluster and a Pi job. RAT output
> was clean.
> 
> My +1 to start.
> 
> Thanks,
> Andrew
> 

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org