Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2163872859
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/BytemanKeywords.robot:
##
@@ -0,0 +1,52 @@
+*** Settings ***
+Library../lib/BytemanLibrary.py
+LibraryString
+
+*** Variables ***
+${BYTEMAN_PORT} 9091
+${DATANODE1_BYTEMAN_HOST_PORT}datanode1:${BYTEMAN_PORT}
+${DATANODE2_BYTEMAN_HOST_PORT}datanode2:${BYTEMAN_PORT}
+${DATANODE3_BYTEMAN_HOST_PORT}datanode3:${BYTEMAN_PORT}
+${OM1_BYTEMAN_HOST_PORT} om1:${BYTEMAN_PORT}
+${OM2_BYTEMAN_HOST_PORT} om2:${BYTEMAN_PORT}
+${OM3_BYTEMAN_HOST_PORT} om3:${BYTEMAN_PORT}
+${RECON_BYTEMAN_HOST_PORT}recon:${BYTEMAN_PORT}
+${SCM1_BYTEMAN_HOST_PORT} scm1.org:${BYTEMAN_PORT}
+${SCM2_BYTEMAN_HOST_PORT} scm2.org:${BYTEMAN_PORT}
+${SCM3_BYTEMAN_HOST_PORT} scm3:${BYTEMAN_PORT}
+${HTTPFS_BYTEMAN_HOST_PORT} httpfs:${BYTEMAN_PORT}
+${S3G_BYTEMAN_HOST_PORT} s3g:${BYTEMAN_PORT}
+
+*** Keywords ***
+Setup Byteman For Component
+[Arguments]${component}${host_port}
+${host}${port} =Split String${host_port}:
+Connect To Byteman Agent${component}${host}${port}
+
+Setup All Byteman Agents
+Log Inside Setup All Byteman Agents
+Setup Byteman For Componentdatanode1 ${DATANODE1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentdatanode2 ${DATANODE2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentdatanode3 ${DATANODE3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom1 ${OM1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom2 ${OM2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom3 ${OM3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentrecon ${RECON_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm1${SCM1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm2${SCM2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm3${SCM3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componenthttps ${HTTPFS_BYTEMAN_HOST_PORT}
+Setup Byteman For Components3g ${S3G_BYTEMAN_HOST_PORT}
Review Comment:
Cleaned up
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot:
##
@@ -0,0 +1,32 @@
+*** Variables ***
+${RULE}/opt/hadoop/share/ozone/byteman/skip-put-block.btm
+${vol} vol1
+${buck}buck1
+${key} key1
+${SECURITY_ENABLED} true
+
+*** Settings ***
+ResourceBytemanKeywords.robot
+Resourceos.robot
+Resource../lib/os.robot
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2163873647
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot:
##
@@ -0,0 +1,32 @@
+*** Variables ***
+${RULE}/opt/hadoop/share/ozone/byteman/skip-put-block.btm
+${vol} vol1
+${buck}buck1
+${key} key1
+${SECURITY_ENABLED} true
+
+*** Settings ***
+ResourceBytemanKeywords.robot
+Resourceos.robot
+Resource../lib/os.robot
+Resource../commonlib.robot
+Resource../ozone-lib/shell.robot
+Suite Setup Setup Suite
+Suite Teardown Teardown Suite
+
+
+*** Keywords ***
+Setup Suite
+Setup All Byteman Agents
+Inject Fault Into Componentdatanode1${RULE}
+Log To Console Kinit
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2163875486
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/byteman.yaml:
##
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+x-byteman-config:
+ &byteman-config
+ environment:
+BYTEMAN_HOME: /opt/byteman/
+OZONE_OPTS:
-javaagent:/opt/byteman.jar=listener:true,interface:0.0.0.0,port:${BYTEMAN_PORT}
Review Comment:
Sure.
address is the new keywork in jdk21. Updated the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2163874682
##
hadoop-ozone/dist/src/main/smoketest/lib/BytemanLibrary.py:
##
@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional consolermation
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import subprocess
+import json
+from robot.api import logger
+
+class BytemanLibrary:
+def __init__(self):
+self.byteman_clients = {}
+
+def connect_to_byteman_agent(self, component_name, host="localhost",
port=9091):
+"""Connect to Byteman agent on specific component"""
+self.byteman_clients[component_name] = {'host': host, 'port': port,
+'base_url':
f"http://{host}:{port}"}
+logger.console(f"Connected to Byteman agent for {component_name} at
{host}:{port}")
+
+def add_byteman_rule(self, component_name, rule_file):
+"""Add Byteman rule into specific component"""
+client = self.byteman_clients[component_name]
+
+# Use bmsubmit command to load rule
+cmd = ["bmsubmit", "-h", component_name, "-p", str(client['port']),
"-l", rule_file]
+result = subprocess.run(cmd, capture_output=True, text=True)
+if result.returncode != 0:
+raise RuntimeError(f"Failed to load rule: {result.stderr}")
+logger.console(f"Loaded Byteman rule {rule_file} into
{component_name}")
+
+def remove_byteman_rule(self, component_name, rule_file):
+"""Remove Byteman rule for specific component"""
+client = self.byteman_clients[component_name]
+
+cmd = ["bmsubmit", "-h", component_name, "-p", str(client['port']),
"-u", rule_file]
+logger.console(cmd)
+result = subprocess.run(cmd, capture_output=True, text=True)
+if result.returncode != 0:
+raise RuntimeError(f"Failed to unload rule: {result.stderr}")
+logger.console(f"Unloaded Byteman rule {rule_file} from
{component_name}")
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2163874227
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot:
##
@@ -0,0 +1,32 @@
+*** Variables ***
+${RULE}/opt/hadoop/share/ozone/byteman/skip-put-block.btm
+${vol} vol1
+${buck}buck1
+${key} key1
+${SECURITY_ENABLED} true
+
+*** Settings ***
+ResourceBytemanKeywords.robot
+Resourceos.robot
+Resource../lib/os.robot
+Resource../commonlib.robot
+Resource../ozone-lib/shell.robot
+Suite Setup Setup Suite
+Suite Teardown Teardown Suite
+
+
+*** Keywords ***
+Setup Suite
+Setup All Byteman Agents
+Inject Fault Into Componentdatanode1${RULE}
+Log To Console Kinit
+Execute And Ignore Error kinit -kt
/etc/security/keytabs/testuser.keytab testuser/om
Review Comment:
Sure. That was a sample test. I am removing these anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2163537380
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/BytemanKeywords.robot:
##
@@ -0,0 +1,52 @@
+*** Settings ***
+Library../lib/BytemanLibrary.py
+LibraryString
+
+*** Variables ***
+${BYTEMAN_PORT} 9091
+${DATANODE1_BYTEMAN_HOST_PORT}datanode1:${BYTEMAN_PORT}
+${DATANODE2_BYTEMAN_HOST_PORT}datanode2:${BYTEMAN_PORT}
+${DATANODE3_BYTEMAN_HOST_PORT}datanode3:${BYTEMAN_PORT}
+${OM1_BYTEMAN_HOST_PORT} om1:${BYTEMAN_PORT}
+${OM2_BYTEMAN_HOST_PORT} om2:${BYTEMAN_PORT}
+${OM3_BYTEMAN_HOST_PORT} om3:${BYTEMAN_PORT}
+${RECON_BYTEMAN_HOST_PORT}recon:${BYTEMAN_PORT}
+${SCM1_BYTEMAN_HOST_PORT} scm1.org:${BYTEMAN_PORT}
+${SCM2_BYTEMAN_HOST_PORT} scm2.org:${BYTEMAN_PORT}
+${SCM3_BYTEMAN_HOST_PORT} scm3:${BYTEMAN_PORT}
+${HTTPFS_BYTEMAN_HOST_PORT} httpfs:${BYTEMAN_PORT}
+${S3G_BYTEMAN_HOST_PORT} s3g:${BYTEMAN_PORT}
+
+*** Keywords ***
+Setup Byteman For Component
+[Arguments]${component}${host_port}
+${host}${port} =Split String${host_port}:
+Connect To Byteman Agent${component}${host}${port}
+
+Setup All Byteman Agents
+Log Inside Setup All Byteman Agents
+Setup Byteman For Componentdatanode1 ${DATANODE1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentdatanode2 ${DATANODE2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentdatanode3 ${DATANODE3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom1 ${OM1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom2 ${OM2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom3 ${OM3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentrecon ${RECON_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm1${SCM1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm2${SCM2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm3${SCM3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componenthttps ${HTTPFS_BYTEMAN_HOST_PORT}
+Setup Byteman For Components3g ${S3G_BYTEMAN_HOST_PORT}
Review Comment:
Sure
##
hadoop-ozone/dist/src/main/smoketest/lib/BytemanLibrary.py:
##
@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional consolermation
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import subprocess
+import json
+from robot.api import logger
+
+class BytemanLibrary:
+def __init__(self):
+self.byteman_clients = {}
+
+def connect_to_byteman_agent(self, component_name, host="localhost",
port=9091):
+"""Connect to Byteman agent on specific component"""
+self.byteman_clients[component_name] = {'host': host, 'port': port,
+'base_url':
f"http://{host}:{port}"}
+logger.console(f"Connected to Byteman agent for {component_name} at
{host}:{port}")
+
+def add_byteman_rule(self, component_name, rule_file):
+"""Add Byteman rule into specific component"""
+client = self.byteman_clients[component_name]
+
+# Use bmsubmit command to load rule
+cmd = ["bmsubmit", "-h", component_name, "-p", str(client['port']),
"-l", rule_file]
+result = subprocess.run(cmd, capture_output=True, text=True)
+if result.returncode != 0:
+raise RuntimeError(f"Failed to load rule: {result.stderr}")
+logger.console(f"Loaded Byteman rule {rule_file} into
{component_name}")
+
+def remove_byteman_rule(self, component_name, rule_file):
+"""Remove Byteman rule for specific component"""
+client = self.byteman_clients[component_name]
+
+cmd = ["bmsubmit", "-h", component_name, "-p", str(client['port']),
"-u", rule_file]
+logger.console(cmd)
+result = subprocess.run(cmd, capture_output=True, text=True)
+if result.returncode != 0:
+raise RuntimeError(f"Failed to unload rule: {result.stderr}")
+logger.console(f"Unloaded Byteman rule {rule_file} from
{component_name}")
Review Comment:
Sure
--
This is a
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
adoroszlai commented on PR #8654: URL: https://github.com/apache/ozone/pull/8654#issuecomment-2998730450 > This PR is required to be merged before [apache/ozone-docker-runner#47](https://github.com/apache/ozone-docker-runner/pull/47) I think the dependency is the other way around: we need the new image with `bmsubmit` before this PR. 1. merge the `ozone-runner` change 2. tag new image 3. update `ozone-runner` version in this PR Until then, let's keep it draft. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
adoroszlai commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2162963412
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot:
##
@@ -0,0 +1,32 @@
+*** Variables ***
+${RULE}/opt/hadoop/share/ozone/byteman/skip-put-block.btm
+${vol} vol1
+${buck}buck1
+${key} key1
+${SECURITY_ENABLED} true
+
+*** Settings ***
+ResourceBytemanKeywords.robot
+Resourceos.robot
+Resource../lib/os.robot
Review Comment:
```suggestion
Resource../lib/os.robot
```
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/byteman.yaml:
##
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+x-byteman-config:
+ &byteman-config
+ environment:
+BYTEMAN_HOME: /opt/byteman/
+OZONE_OPTS:
-javaagent:/opt/byteman.jar=listener:true,interface:0.0.0.0,port:${BYTEMAN_PORT}
Review Comment:
`OZONE_OPTS` is applied to all `ozone` commands. `ozone sh`, `ozone admin`,
etc. will show error due to `Address already in use`. We'll need to add a way
to set this only for server processes (HDDS-13325).
Also got `illegal agent argument : interface:0.0.0.0`. Byteman is listening
only on localhost (`127.0.0.1:9091`), thus cannot connect to other container's
listener.
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/BytemanKeywords.robot:
##
@@ -0,0 +1,52 @@
+*** Settings ***
+Library../lib/BytemanLibrary.py
+LibraryString
+
+*** Variables ***
+${BYTEMAN_PORT} 9091
+${DATANODE1_BYTEMAN_HOST_PORT}datanode1:${BYTEMAN_PORT}
+${DATANODE2_BYTEMAN_HOST_PORT}datanode2:${BYTEMAN_PORT}
+${DATANODE3_BYTEMAN_HOST_PORT}datanode3:${BYTEMAN_PORT}
+${OM1_BYTEMAN_HOST_PORT} om1:${BYTEMAN_PORT}
+${OM2_BYTEMAN_HOST_PORT} om2:${BYTEMAN_PORT}
+${OM3_BYTEMAN_HOST_PORT} om3:${BYTEMAN_PORT}
+${RECON_BYTEMAN_HOST_PORT}recon:${BYTEMAN_PORT}
+${SCM1_BYTEMAN_HOST_PORT} scm1.org:${BYTEMAN_PORT}
+${SCM2_BYTEMAN_HOST_PORT} scm2.org:${BYTEMAN_PORT}
+${SCM3_BYTEMAN_HOST_PORT} scm3:${BYTEMAN_PORT}
+${HTTPFS_BYTEMAN_HOST_PORT} httpfs:${BYTEMAN_PORT}
+${S3G_BYTEMAN_HOST_PORT} s3g:${BYTEMAN_PORT}
+
+*** Keywords ***
+Setup Byteman For Component
+[Arguments]${component}${host_port}
+${host}${port} =Split String${host_port}:
+Connect To Byteman Agent${component}${host}${port}
+
+Setup All Byteman Agents
+Log Inside Setup All Byteman Agents
+Setup Byteman For Componentdatanode1 ${DATANODE1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentdatanode2 ${DATANODE2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentdatanode3 ${DATANODE3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom1 ${OM1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom2 ${OM2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentom3 ${OM3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentrecon ${RECON_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm1${SCM1_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm2${SCM2_BYTEMAN_HOST_PORT}
+Setup Byteman For Componentscm3${SCM3_BYTEMAN_HOST_PORT}
+Setup Byteman For Componenthttps ${HTTPFS_BYTEMAN_HOST_PORT}
+Setup Byteman For Components3g ${S3G_BYTEMAN_HOST_PORT}
Review Comment:
I think this is unnecessary.
1. `Connect To Byteman Agent` just stores the `component => host:port`
mapping. This can be skipped, `Inject Fault` and `Remove Fault` can use
hostname directly, and take port from the environment variable
`%{BYTEMAN_PORT}`.
2. `Setup Byteman For Component` splits its `${host_port}` argument, so
creating all these variables in `host:port` format is not needed.
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot:
##
@@ -0,0 +1,32 @@
+*** Variables ***
+${RULE}/opt/hadoop/share/ozone/byteman/skip-put-block.btm
+${vol} vol1
+${buck}buck1
+${key} key1
+${SECURITY_ENABLED} true
+
+*** Settings ***
+ResourceBytemanKey
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2162438344
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/docker-compose.yaml:
##
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-common-config:
+ &common-config
+ dns_search: .
+ image: ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+- ../_keytabs:/etc/security/keytabs
+- ./krb5.conf:/etc/krb5.conf
+- ./byteman-scripts:/opt/byteman/scripts:ro
+ env_file:
+- docker-config
+
+services:
+ kdc:
+image: ${OZONE_TESTKRB5_IMAGE}
+hostname: kdc
+dns_search: .
+volumes:
+ - ../..:/opt/hadoop
+ - ../_keytabs:/etc/security/keytabs
+command: ["/opt/hadoop/compose/common/init-kdc.sh"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.100
+ kms:
+image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
+dns_search: .
+ports:
+ - 9600:9600
+env_file:
+ - ./docker-config
+volumes:
+ - ../_keytabs:/etc/security/keytabs
+ - ./krb5.conf:/etc/krb5.conf
+ - ../../libexec/transformation.py:/opt/transformation.py
+environment:
+ HADOOP_CONF_DIR: /opt/hadoop/etc/hadoop
+command: ["hadoop", "kms"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.101
+ datanode1:
+<<: *common-config
+ports:
+ - 19864:
+ - 9090:9090
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9090
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.102
+ datanode2:
+<<: *common-config
+ports:
+ - 9866:
+ - 9091:9091
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9091
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.103
+ datanode3:
+<<: *common-config
+ports:
+ - 9868:
+ - 9092:9092
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9092
Review Comment:
I am not updating this as this is fixated on a particular rule but I want a
dynamic update of rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654: URL: https://github.com/apache/ozone/pull/8654#discussion_r2162435025 ## hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot: ## @@ -0,0 +1,16 @@ +*** Settings *** +ResourceBytemanKeywords.robot +Suite SetupSetup All Byteman Agents + +*** Test Cases *** +Test Skip Put Block on the datanode +Inject Fault Into Componentdatanode1 /opt/byteman/scripts/skip-put-block.btm + +# Run your Ozone operations that should be affected by delay +Execute Ozone Commandozone sh volume create /vol1 Review Comment: Updated the test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2162434741
##
hadoop-ozone/dist/src/main/smoketest/ozone-fi/BytemanKeywords.robot:
##
@@ -0,0 +1,49 @@
+*** Settings ***
+LibraryBytemanLibrary
+
+*** Variables ***
+${DATANODE1_BYTEMAN_HOST_PORT}172.25.0.102:9090
+${DATANODE2_BYTEMAN_HOST_PORT}172.25.0.103:9091
+${DATANODE3_BYTEMAN_HOST_PORT}172.25.0.104:9092
+${OM1_BYTEMAN_HOST_PORT} 172.25.0.111:9093
+${OM2_BYTEMAN_HOST_PORT} 172.25.0.112:9094
+${OM3_BYTEMAN_HOST_PORT} 172.25.0.113:9095
+${RECON_BYTEMAN_HOST_PORT}172.25.0.115:9096
+${SCM1_BYTEMAN_HOST_PORT} 172.25.0.116:9097
+${SCM2_BYTEMAN_HOST_PORT} 172.25.0.117:9098
+${SCM3_BYTEMAN_HOST_PORT} 172.25.0.118:9099
+${HTTPFS_BYTEMAN_HOST_PORT} 172.25.0.119:9100
+${S3G_BYTEMAN_HOST_PORT} 172.25.0.120:9101
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654: URL: https://github.com/apache/ozone/pull/8654#discussion_r2162443380 ## hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/byteman-scripts/skip-notify-group-remove.btm: ## Review Comment: Done ## hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/.env: ## Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on PR #8654: URL: https://github.com/apache/ozone/pull/8654#issuecomment-2997806866 This PR is required to be merged before https://github.com/apache/ozone-docker-runner/pull/47 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2162443864
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/docker-compose.yaml:
##
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-common-config:
+ &common-config
+ dns_search: .
+ image: ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+- ../_keytabs:/etc/security/keytabs
+- ./krb5.conf:/etc/krb5.conf
+- ./byteman-scripts:/opt/byteman/scripts:ro
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
ssulav commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2162438624
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/docker-compose.yaml:
##
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-common-config:
+ &common-config
+ dns_search: .
+ image: ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+- ../_keytabs:/etc/security/keytabs
+- ./krb5.conf:/etc/krb5.conf
+- ./byteman-scripts:/opt/byteman/scripts:ro
+ env_file:
+- docker-config
+
+services:
+ kdc:
+image: ${OZONE_TESTKRB5_IMAGE}
+hostname: kdc
+dns_search: .
+volumes:
+ - ../..:/opt/hadoop
+ - ../_keytabs:/etc/security/keytabs
+command: ["/opt/hadoop/compose/common/init-kdc.sh"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.100
+ kms:
+image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
+dns_search: .
+ports:
+ - 9600:9600
+env_file:
+ - ./docker-config
+volumes:
+ - ../_keytabs:/etc/security/keytabs
+ - ./krb5.conf:/etc/krb5.conf
+ - ../../libexec/transformation.py:/opt/transformation.py
+environment:
+ HADOOP_CONF_DIR: /opt/hadoop/etc/hadoop
+command: ["hadoop", "kms"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.101
+ datanode1:
+<<: *common-config
+ports:
+ - 19864:
+ - 9090:9090
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9090
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.102
+ datanode2:
+<<: *common-config
+ports:
+ - 9866:
+ - 9091:9091
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9091
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.103
+ datanode3:
+<<: *common-config
+ports:
+ - 9868:
+ - 9092:9092
Review Comment:
Yes. Resolved
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/docker-compose.yaml:
##
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-common-config:
+ &common-config
+ dns_search: .
+ image: ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+- ../_keytabs:/etc/security/keytabs
+- ./krb5.conf:/etc/krb5.conf
+- ./byteman-scripts:/opt/byteman/scripts:ro
+ env_file:
+- docker-config
+
+services:
+ kdc:
+image: ${OZONE_TESTKRB5_IMAGE}
+hostname: kdc
+dns_search: .
+volumes:
+ - ../..:/opt/hadoop
+ - ../_keytabs:/etc/security/keytabs
+command: ["/opt/hadoop/compose/common/init-kdc.sh"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.100
+ kms:
+image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
+dns_search: .
+ports:
+ - 9600:9600
+env_file:
+ - ./docker-config
+volumes:
+ - ../_keytabs:/etc/security/keytabs
+ - ./krb5.conf:/etc/krb5.conf
+ - ../../libexec/transformation.py:/opt/transformation.py
+environment:
+ HADOOP_CONF_DIR: /opt/ha
Re: [PR] HDDS-13251. Adding Byteman support in acceptance test suites via new docker-compose [ozone]
adoroszlai commented on code in PR #8654:
URL: https://github.com/apache/ozone/pull/8654#discussion_r2154450491
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/docker-compose.yaml:
##
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-common-config:
+ &common-config
+ dns_search: .
+ image: ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+- ../_keytabs:/etc/security/keytabs
+- ./krb5.conf:/etc/krb5.conf
+- ./byteman-scripts:/opt/byteman/scripts:ro
Review Comment:
This extra mount is not necessary, the same files are already available in
the container at `/opt/hadoop/compose/byteman-scripts`.
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/byteman-scripts/skip-notify-group-remove.btm:
##
Review Comment:
Existing byteman scripts are at `/opt/hadoop/share/ozone/byteman` in the
container, sources at `dev-support/byteman`.
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/docker-compose.yaml:
##
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+x-common-config:
+ &common-config
+ dns_search: .
+ image: ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+ volumes:
+- ../..:/opt/hadoop
+- ../_keytabs:/etc/security/keytabs
+- ./krb5.conf:/etc/krb5.conf
+- ./byteman-scripts:/opt/byteman/scripts:ro
+ env_file:
+- docker-config
+
+services:
+ kdc:
+image: ${OZONE_TESTKRB5_IMAGE}
+hostname: kdc
+dns_search: .
+volumes:
+ - ../..:/opt/hadoop
+ - ../_keytabs:/etc/security/keytabs
+command: ["/opt/hadoop/compose/common/init-kdc.sh"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.100
+ kms:
+image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
+dns_search: .
+ports:
+ - 9600:9600
+env_file:
+ - ./docker-config
+volumes:
+ - ../_keytabs:/etc/security/keytabs
+ - ./krb5.conf:/etc/krb5.conf
+ - ../../libexec/transformation.py:/opt/transformation.py
+environment:
+ HADOOP_CONF_DIR: /opt/hadoop/etc/hadoop
+command: ["hadoop", "kms"]
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.101
+ datanode1:
+<<: *common-config
+ports:
+ - 19864:
+ - 9090:9090
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9090
+networks:
+ ozone_net:
+ipv4_address: 172.25.0.102
+ datanode2:
+<<: *common-config
+ports:
+ - 9866:
+ - 9091:9091
+command: ["/opt/hadoop/bin/ozone","datanode"]
+extra_hosts:
+ - "scm1.org=172.25.0.116"
+ - "scm2.org=172.25.0.117"
+ - "scm3.org=172.25.0.118"
+ - "recon=172.25.0.115"
+environment:
+ WAITFOR: scm3.org:9894
+ OZONE_OPTS: ${BYTEMAN_OPTS},port:9091
Review Comment:
Byteman port should be the same in all containers, so this per-service
definition is unnecessary.
##
hadoop-ozone/dist/src/main/compose/ozonesecure-ha-fi/.env:
##
Review Comment:
Please don't duplicate an existing environment (`ozonesecure-ha`). If some
customization is needed, create an add-on. There are multiple examples for
that, one of them is Vault, see
`hadoop-ozone/dist/src/main/compose/ozonesecure/*
