[PATCH v2 net-next 1/1] tc-testing: Clarify the use of tdc's -d option
The -d command line argument to tdc requires the name of a physical device
on the system where the tests will be run. If -d has not been used, tdc
will skip tests that require a physical device.
This patch is intended to better document what the -d option does and how
it is used.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/README | 4 +++-
tools/testing/selftests/tc-testing/tdc.py | 12
2 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/README
b/tools/testing/selftests/tc-testing/README
index 22e5da9..b0954c8 100644
--- a/tools/testing/selftests/tc-testing/README
+++ b/tools/testing/selftests/tc-testing/README
@@ -128,7 +128,9 @@ optional arguments:
-v, --verbose Show the commands that are being run
-N, --notap Suppress tap results for command under test
-d DEVICE, --device DEVICE
-Execute the test case in flower category
+Execute test cases that use a physical device, where
+DEVICE is its name. (If not defined, tests that require
+a physical device will be skipped)
-P, --pause Pause execution just before post-suite stage
selection:
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index f04321a..e566c70 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -356,12 +356,14 @@ def test_runner(pm, args, filtered_tests):
time.sleep(2)
for tidx in testlist:
if "flower" in tidx["category"] and args.device == None:
+errmsg = "Tests using the DEV2 variable must define the name of a "
+errmsg += "physical NIC with the -d option when running tdc.\n"
+errmsg += "Test has been skipped."
if args.verbose > 1:
-print('Not executing test {} {} because DEV2 not defined'.
- format(tidx['id'], tidx['name']))
+print(errmsg)
res = TestResult(tidx['id'], tidx['name'])
res.set_result(ResultState.skip)
-res.set_errormsg('Not executed because DEV2 is not defined')
+res.set_errormsg(errmsg)
tsr.add_resultdata(res)
continue
try:
@@ -499,7 +501,9 @@ def set_args(parser):
choices=['none', 'xunit', 'tap'],
help='Specify the format for test results. (Default: TAP)')
parser.add_argument('-d', '--device',
-help='Execute the test case in flower category')
+help='Execute test cases that use a physical device, '
+
+'where DEVICE is its name. (If not defined, tests ' +
+'that require a physical device will be skipped)')
parser.add_argument(
'-P', '--pause', action='store_true',
help='Pause execution just before post-suite stage')
--
2.7.4
[PATCH net-next 2/2] tc-testing: introduce scapyPlugin for basic traffic
The scapyPlugin allows for simple traffic generation in tdc to
test various tc features. It was tested with scapy v2.4.2, but
should work with any successive version.
In order to use the plugin's functionality, scapy must be
installed. This can be done with:
pip3 install scapy
or to install 2.4.2:
pip3 install scapy==2.4.2
If the plugin is unable to import the scapy module, it will
terminate the tdc run.
The plugin makes use of a new key in the test case data, 'scapy'.
This block contains three other elements: 'iface', 'count', and
'packet':
"scapy": {
"iface": "$DEV0",
"count": 1,
"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
},
* iface is the name of the device on the host machine from which
the packet(s) will be sent. Values contained within tdc_config.py's
NAMES dict can be used here - this is useful if paired with
nsPlugin
* count is the number of copies of this packet to be sent
* packet is a string detailing the different layers of the packet
to be sent. If a property isn't explicitly set, scapy will set
default values for you.
Layers in the packet info are separated by slashes. For info about
common TCP and IP properties, see:
https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf
Caution is advised when running tests using the scapy functionality,
since the plugin blindly sends the packet as defined in the test case
data.
See creating-testcases/scapy-example.json for sample test cases;
the first test is intended to pass while the second is intended to
fail.
Signed-off-by: Lucas Bates
---
.../creating-testcases/scapy-example.json | 98 ++
.../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++
2 files changed, 149 insertions(+)
create mode 100644
tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
diff --git
a/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
new file mode 100644
index 000..5a9377b
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
@@ -0,0 +1,98 @@
+[
+{
+"id": "b1e9",
+"name": "Test matching of source IP",
+"category": [
+"actions",
+"scapy"
+],
+"plugins": {
+"requires": [
+"nsPlugin",
+"scapyPlugin"
+]
+},
+"setup": [
+[
+"$TC qdisc del dev $DEV1 ingress",
+0,
+1,
+2,
+255
+],
+"$TC qdisc add dev $DEV1 ingress"
+],
+"cmdUnderTest": "$TC filter add dev $DEV1 parent : prio 3 protocol
ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+"scapy": {
+"iface": "$DEV0",
+"count": 1,
+"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+},
+"expExitCode": "0",
+"verifyCmd": "$TC -s -j filter ls dev $DEV1 ingress prio 3",
+"matchJSON": [
+{
+"path": [
+1,
+"options",
+"actions",
+0,
+"stats",
+"packets"
+],
+"value": 1
+}
+],
+"teardown": [
+"$TC qdisc del dev $DEV1 ingress"
+]
+},
+{
+"id": "e9c4",
+"name": "Test matching of source IP with wrong count",
+"category": [
+"actions",
+"scapy"
+],
+"plugins": {
+"requires": [
+"nsPlugin",
+"scapyPlugin"
+]
+},
+"setup": [
+[
+"$TC qdisc del dev $DEV1 ingress",
+0,
+1,
+2,
+255
+],
+"$TC qdisc add dev $DEV1 ingress"
+],
+"cmdUnderTest": "$TC filter add dev $DEV1 parent : prio 3 protocol
ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+"scapy&
[PATCH net-next 1/2] tc-testing: Allow tdc plugins to see test case data
Instead of only passing the test case name and ID, pass the
entire current test case down to the plugins. This change
allows plugins to start accepting commands and directives
from the test cases themselves, for greater flexibility
in testing.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/TdcPlugin.py | 5 ++---
tools/testing/selftests/tc-testing/tdc.py | 10 +-
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
b/tools/testing/selftests/tc-testing/TdcPlugin.py
index b980a56..79f3ca8 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,12 +18,11 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
-def pre_case(self, testid, test_name, test_skip):
+def pre_case(self, caseinfo, test_skip):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
-self.args.testid = testid
-self.args.test_name = test_name
+self.args.caseinfo = caseinfo
self.args.test_skip = test_skip
def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 1afa803..de7da9a 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -126,15 +126,15 @@ class PluginMgr:
for pgn_inst in reversed(self.plugin_instances):
pgn_inst.post_suite(index)
-def call_pre_case(self, testid, test_name, *, test_skip=False):
+def call_pre_case(self, caseinfo, *, test_skip=False):
for pgn_inst in self.plugin_instances:
try:
-pgn_inst.pre_case(testid, test_name, test_skip)
+pgn_inst.pre_case(caseinfo, test_skip)
except Exception as ee:
print('exception {} in call to pre_case for {} plugin'.
format(ee, pgn_inst.__class__))
print('test_ordinal is {}'.format(test_ordinal))
-print('testid is {}'.format(testid))
+print('testid is {}'.format(caseinfo['id']))
raise
def call_post_case(self):
@@ -379,14 +379,14 @@ def run_one_test(pm, args, index, tidx):
res = TestResult(tidx['id'], tidx['name'])
res.set_result(ResultState.skip)
res.set_errormsg('Test case designated as skipped.')
-pm.call_pre_case(tidx['id'], tidx['name'], test_skip=True)
+pm.call_pre_case(tidx, test_skip=True)
pm.call_post_execute()
return res
# populate NAMES with TESTID for this test
NAMES['TESTID'] = tidx['id']
-pm.call_pre_case(tidx['id'], tidx['name'])
+pm.call_pre_case(tidx)
prepare_env(args, pm, 'setup', "-> prepare stage", tidx["setup"])
if (args.verbose > 0):
--
2.7.4
[PATCH net-next 0/2] tc-testing: Add plugin for simple traffic generation
This series supersedes the previous submission that included a patch for test case verification using JSON output. It adds a new tdc plugin, scapyPlugin, as a way to send traffic to test tc filters and actions. The first patch makes a change to the TdcPlugin module that will allow tdc plugins to examine the test case currently being executed, so plugins can play a more active role in testing by accepting information or commands from the test case. This is required for scapyPlugin to work. The second patch adds scapyPlugin itself, and an example test case file to demonstrate how the scapy block works in the test cases. Lucas Bates (2): tc-testing: Allow tdc plugins to see test case data tc-testing: introduce scapyPlugin for basic traffic tools/testing/selftests/tc-testing/TdcPlugin.py| 5 +- .../creating-testcases/scapy-example.json | 98 ++ .../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++ tools/testing/selftests/tc-testing/tdc.py | 10 +-- 4 files changed, 156 insertions(+), 8 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py -- 2.7.4
Re: [PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic
Sorry Alex, I completely forgot about this email. On Thu, Jul 4, 2019 at 4:29 PM Alexander Aring wrote: > > Hi, > > On Wed, Jul 03, 2019 at 08:45:02PM -0400, Lucas Bates wrote: > > The scapyPlugin allows for simple traffic generation in tdc to > > test various tc features. It was tested with scapy v2.4.2, but > > should work with any successive version. > Is there a way to introduce thrid party scapy level descriptions which > are not upstream yet? Upstream to scapy? Not yet. This version of the plugin is extremely simple, and good for basic traffic. I'll add features to it so we can get more creative with the packets that can be sent, though. Lucas
Re: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc
On Mon, Jul 8, 2019 at 1:25 PM Alexander Aring wrote: > > Unless I'm off-base here? > > yes you need to know some python, complex code can be hidden by some > helper functionality I guess. > > I have no problem to let this patch in, it will not harm anything... I think I'm going to pull it for the moment - I started thinking about the patch today and I think it needs more testing against larger amounts of data. > Maybe I work on a matchEval and show some examples... in a human > readable way you can even concatenate bool expressions in combinations > with helpers. > > I just was curious, so I might add the matchEval or something to show > this approach. Go for it, I think you have a much better grasp on the use of eval than I do - and it could be very useful for test cases.
Re: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc
On Thu, Jul 4, 2019 at 4:21 PM Alexander Aring wrote: > why you just use eval() as pattern matching operation and let the user > define how to declare a matching mechanism instead you introduce another > static matching scheme based on a json description? > > Whereas in eval() you could directly use the python bool expression > parser to make whatever you want. > > I don't know, I see at some points you will hit limitations what you can > express with this matchFOO and we need to introduce another matchBAR, > whereas in providing the code it should be no problem expression > anything. If you want smaller shortcuts writing matching patterns you > can implement them and using in your eval() operation. Regarding hitting limitations: quite possibly, yes. Using eval() to provide code for matching is going to put more of a dependency on the test writer knowing Python. I know it's not a terribly difficult language to pick up, but it's still setting a higher barrier to entry. This is the primary reason I scrapped the work I had presented at Netdev 1.2 in Tokyo, where all the tests were coded using Python's unittest framework - I want to be sure it's as easy as possible for people to use tdc and write tests for it. Unless I'm off-base here? Lucas
[PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc
This patch allows tdc to process JSON output to perform secondary
verification of the command under test. If the verifyCmd generates
JSON, one can provide the 'matchJSON' key to process it
instead of a regex.
matchJSON has two elements: 'path' and 'value'. The 'path' key is a
list of integers and strings that provide the key values for tdc to
navigate the JSON information. The value is an integer or string
that tdc will compare against what it finds in the provided path.
If the numerical position of an element can vary, it's possible to
substitute an asterisk as a wildcard. tdc will search all possible
entries in the array.
Multiple matches are possible, but everything specified must
match for the test to pass.
If both matchPattern and matchJSON are present, tdc will only
operate on matchPattern. If neither are present, verification
is skipped.
Example:
"cmdUnderTest": "$TC actions add action pass index 8",
"verifyCmd": "$TC actions list action gact",
"matchJSON": [
{
"path": [
0,
"actions",
0,
"control action",
"type"
],
"value": "gact"
},
{
"path": [
0,
"actions",
0,
"index"
],
"value": 8
}
]
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tdc.py | 136 +++---
1 file changed, 123 insertions(+), 13 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 678182a..1afa803 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -35,6 +35,10 @@ class PluginMgrTestFail(Exception):
self.output = output
self.message = message
+class ElemNotFound(Exception):
+def __init__(self, path_element):
+self.path_element = path_element
+
class PluginMgr:
def __init__(self, argparser):
super().__init__()
@@ -167,6 +171,40 @@ class PluginMgr:
self.argparser = argparse.ArgumentParser(
description='Linux TC unit tests')
+def find_in_json(jsonobj, path):
+if type(jsonobj) == list:
+if type(path[0]) == int:
+if len(jsonobj) > path[0]:
+return find_in_json(jsonobj[path[0]], path[1:])
+else:
+raise ElemNotFound(path[0])
+elif path[0] == '*':
+res = []
+for index in jsonobj:
+try:
+res.append(find_in_json(index, path[1:]))
+except ElemNotFound:
+continue
+if len(res) == 0:
+raise ElemNotFound(path[0])
+else:
+return res
+elif type(jsonobj) == dict:
+if path[0] in jsonobj:
+if len(path) > 1:
+return find_in_json(jsonobj[path[0]], path[1:])
+return jsonobj[path[0]]
+else:
+raise ElemNotFound(path[0])
+else:
+# Assume we have found the correct depth in the object
+if len(path) >= 1:
+print('The remainder of the specified path cannot be found!')
+print('Path values: {}'.format(path))
+raise ElemNotFound(path[0])
+return jsonobj
+
+
def replace_keywords(cmd):
"""
For a given executable command, substitute any known
@@ -246,6 +284,86 @@ def prepare_env(args, pm, stage, prefix, cmdlist, output =
None):
stage, output,
'"{}" did not complete successfully'.format(prefix))
+def verify_by_regex(res, tidx, args, pm):
+if 'matchCount' not in tidx:
+res.set_result(ResultState.skip)
+fmsg = 'matchCount was not provided in the test case. '
+fmsg += 'Unable to complete pattern match.'
+res.set_failmsg(fmsg)
+print(fmsg)
+return res
+(p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
+match_pattern = re.compile(
+str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE)
+if procout:
+match_index = re.findall(match_pattern, procout)
+if len(match_index) != int(tidx["matchCount"]):
+res.set_result(ResultState.fail)
+fmsg = 'Verify stage failed because the output did not match '
+fmsg += 'the pattern in the test case.\nMatch pattern is:\n'
+fmsg += '\t{}\n'.format(tidx["matchPattern"])
+fmsg += 'Output generated by the verify command:\n'
+
[PATCH v2 net-next 0/3] tc-testing: Add JSON verification and simple traffic generation
This patchset introduces JSON as a verification method in tdc and adds a new plugin, scapyPlugin, as a way to send traffic to test tc filters and actions. This version includes the patch signoffs missing in the previous submission. The first patch adds the JSON verification to the core tdc script. The second patch makes a change to the TdcPlugin module that will allow tdc plugins to examine the test case currently being executed, such that plugins can play a more active role in testing. This feature is needed for the new plugin. The third patch adds the scapyPlugin itself, and an example test case file to demonstrate how the scapy block works. Lucas Bates (3): tc-testing: Add JSON verification to tdc tc-testing: Allow tdc plugins to see test case data tc-testing: introduce scapyPlugin for basic traffic tools/testing/selftests/tc-testing/TdcPlugin.py| 5 +- .../creating-testcases/scapy-example.json | 98 ++ .../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++ tools/testing/selftests/tc-testing/tdc.py | 146 ++--- 4 files changed, 279 insertions(+), 21 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py -- 2.7.4
[PATCH v2 net-next 2/3] tc-testing: Allow tdc plugins to see test case data
Instead of only passing the test case name and ID, pass the
entire current test case down to the plugins. This change
allows plugins to start accepting commands and directives
from the test cases themselves, for greater flexibility
in testing.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/TdcPlugin.py | 5 ++---
tools/testing/selftests/tc-testing/tdc.py | 10 +-
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
b/tools/testing/selftests/tc-testing/TdcPlugin.py
index b980a56..79f3ca8 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,12 +18,11 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
-def pre_case(self, testid, test_name, test_skip):
+def pre_case(self, caseinfo, test_skip):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
-self.args.testid = testid
-self.args.test_name = test_name
+self.args.caseinfo = caseinfo
self.args.test_skip = test_skip
def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 1afa803..de7da9a 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -126,15 +126,15 @@ class PluginMgr:
for pgn_inst in reversed(self.plugin_instances):
pgn_inst.post_suite(index)
-def call_pre_case(self, testid, test_name, *, test_skip=False):
+def call_pre_case(self, caseinfo, *, test_skip=False):
for pgn_inst in self.plugin_instances:
try:
-pgn_inst.pre_case(testid, test_name, test_skip)
+pgn_inst.pre_case(caseinfo, test_skip)
except Exception as ee:
print('exception {} in call to pre_case for {} plugin'.
format(ee, pgn_inst.__class__))
print('test_ordinal is {}'.format(test_ordinal))
-print('testid is {}'.format(testid))
+print('testid is {}'.format(caseinfo['id']))
raise
def call_post_case(self):
@@ -379,14 +379,14 @@ def run_one_test(pm, args, index, tidx):
res = TestResult(tidx['id'], tidx['name'])
res.set_result(ResultState.skip)
res.set_errormsg('Test case designated as skipped.')
-pm.call_pre_case(tidx['id'], tidx['name'], test_skip=True)
+pm.call_pre_case(tidx, test_skip=True)
pm.call_post_execute()
return res
# populate NAMES with TESTID for this test
NAMES['TESTID'] = tidx['id']
-pm.call_pre_case(tidx['id'], tidx['name'])
+pm.call_pre_case(tidx)
prepare_env(args, pm, 'setup', "-> prepare stage", tidx["setup"])
if (args.verbose > 0):
--
2.7.4
[PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic
The scapyPlugin allows for simple traffic generation in tdc to
test various tc features. It was tested with scapy v2.4.2, but
should work with any successive version.
In order to use the plugin's functionality, scapy must be
installed. This can be done with:
pip3 install scapy
or to install 2.4.2:
pip3 install scapy==2.4.2
If the plugin is unable to import the scapy module, it will
terminate the tdc run.
The plugin makes use of a new key in the test case data, 'scapy'.
This block contains three other elements: 'iface', 'count', and
'packet':
"scapy": {
"iface": "$DEV0",
"count": 1,
"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
},
* iface is the name of the device on the host machine from which
the packet(s) will be sent. Values contained within tdc_config.py's
NAMES dict can be used here - this is useful if paired with
nsPlugin
* count is the number of copies of this packet to be sent
* packet is a string detailing the different layers of the packet
to be sent. If a property isn't explicitly set, scapy will set
default values for you.
Layers in the packet info are separated by slashes. For info about
common TCP and IP properties, see:
https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf
Caution is advised when running tests using the scapy functionality,
since the plugin blindly sends the packet as defined in the test case
data.
See creating-testcases/scapy-example.json for sample test cases;
the first test is intended to pass while the second is intended to
fail. Consider using the matchJSON functionality for verification
when using scapy.
Signed-off-by: Lucas Bates
---
.../creating-testcases/scapy-example.json | 98 ++
.../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++
2 files changed, 149 insertions(+)
create mode 100644
tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
diff --git
a/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
new file mode 100644
index 000..5a9377b
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
@@ -0,0 +1,98 @@
+[
+{
+"id": "b1e9",
+"name": "Test matching of source IP",
+"category": [
+"actions",
+"scapy"
+],
+"plugins": {
+"requires": [
+"nsPlugin",
+"scapyPlugin"
+]
+},
+"setup": [
+[
+"$TC qdisc del dev $DEV1 ingress",
+0,
+1,
+2,
+255
+],
+"$TC qdisc add dev $DEV1 ingress"
+],
+"cmdUnderTest": "$TC filter add dev $DEV1 parent : prio 3 protocol
ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+"scapy": {
+"iface": "$DEV0",
+"count": 1,
+"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+},
+"expExitCode": "0",
+"verifyCmd": "$TC -s -j filter ls dev $DEV1 ingress prio 3",
+"matchJSON": [
+{
+"path": [
+1,
+"options",
+"actions",
+0,
+"stats",
+"packets"
+],
+"value": 1
+}
+],
+"teardown": [
+"$TC qdisc del dev $DEV1 ingress"
+]
+},
+{
+"id": "e9c4",
+"name": "Test matching of source IP with wrong count",
+"category": [
+"actions",
+"scapy"
+],
+"plugins": {
+"requires": [
+"nsPlugin",
+"scapyPlugin"
+]
+},
+"setup": [
+[
+"$TC qdisc del dev $DEV1 ingress",
+0,
+1,
+2,
+255
+],
+"$TC qdisc add dev $DEV1 ingress"
+],
+"cmdUnderTest": "$TC filter add dev $DEV1 parent : prio 3 protocol
i
[PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation
This patchset introduces JSON as a verification method in tdc and adds a new plugin, scapyPlugin, as a way to send traffic to test tc filters and actions. The first patch adds the JSON verification to the core tdc script. The second patch makes a change to the TdcPlugin module that will allow tdc plugins to examine the test case currently being executed, such that plugins can play a more active role in testing. This feature is needed for the new plugin. The third patch adds the scapyPlugin itself, and an example test case file to demonstrate how the scapy block works. Lucas Bates (3): tc-testing: Add JSON verification to tdc tc-testing: Allow tdc plugins to see test case data tc-testing: introduce scapyPlugin for basic traffic tools/testing/selftests/tc-testing/TdcPlugin.py| 5 +- .../creating-testcases/scapy-example.json | 98 ++ .../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++ tools/testing/selftests/tc-testing/tdc.py | 146 ++--- 4 files changed, 279 insertions(+), 21 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py -- 2.7.4
[PATCH net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic
The scapyPlugin allows for simple traffic generation in tdc to
test various tc features. It was tested with scapy v2.4.2, but
should work with any successive version.
In order to use the plugin's functionality, scapy must be
installed. This can be done with:
pip3 install scapy
or to install 2.4.2:
pip3 install scapy==2.4.2
If the plugin is unable to import the scapy module, it will
terminate the tdc run.
The plugin makes use of a new key in the test case data, 'scapy'.
This block contains three other elements: 'iface', 'count', and
'packet':
"scapy": {
"iface": "$DEV0",
"count": 1,
"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
},
* iface is the name of the device on the host machine from which
the packet(s) will be sent. Values contained within tdc_config.py's
NAMES dict can be used here - this is useful if paired with
nsPlugin
* count is the number of copies of this packet to be sent
* packet is a string detailing the different layers of the packet
to be sent. If a property isn't explicitly set, scapy will set
default values for you.
Layers in the packet info are separated by slashes. For info about
common TCP and IP properties, see:
https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf
Caution is advised when running tests using the scapy functionality,
since the plugin blindly sends the packet as defined in the test case
data.
See creating-testcases/scapy-example.json for sample test cases;
the first test is intended to pass while the second is intended to
fail. Consider using the matchJSON functionality for verification
when using scapy.
---
.../creating-testcases/scapy-example.json | 98 ++
.../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++
2 files changed, 149 insertions(+)
create mode 100644
tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
diff --git
a/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
new file mode 100644
index 000..5a9377b
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
@@ -0,0 +1,98 @@
+[
+{
+"id": "b1e9",
+"name": "Test matching of source IP",
+"category": [
+"actions",
+"scapy"
+],
+"plugins": {
+"requires": [
+"nsPlugin",
+"scapyPlugin"
+]
+},
+"setup": [
+[
+"$TC qdisc del dev $DEV1 ingress",
+0,
+1,
+2,
+255
+],
+"$TC qdisc add dev $DEV1 ingress"
+],
+"cmdUnderTest": "$TC filter add dev $DEV1 parent : prio 3 protocol
ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+"scapy": {
+"iface": "$DEV0",
+"count": 1,
+"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+},
+"expExitCode": "0",
+"verifyCmd": "$TC -s -j filter ls dev $DEV1 ingress prio 3",
+"matchJSON": [
+{
+"path": [
+1,
+"options",
+"actions",
+0,
+"stats",
+"packets"
+],
+"value": 1
+}
+],
+"teardown": [
+"$TC qdisc del dev $DEV1 ingress"
+]
+},
+{
+"id": "e9c4",
+"name": "Test matching of source IP with wrong count",
+"category": [
+"actions",
+"scapy"
+],
+"plugins": {
+"requires": [
+"nsPlugin",
+"scapyPlugin"
+]
+},
+"setup": [
+[
+"$TC qdisc del dev $DEV1 ingress",
+0,
+1,
+2,
+255
+],
+"$TC qdisc add dev $DEV1 ingress"
+],
+"cmdUnderTest": "$TC filter add dev $DEV1 parent : prio 3 protocol
ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+"scapy": {
+"iface": "$DEV0",
+"count": 3,
+"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+},
+"expExitCode": "0",
+"verifyCmd": "$TC -s -j filter ls dev $DEV1 parent :",
+"matchJSON": [
+{
+"path": [
+1,
+"options",
+"actions",
+0,
+"stats",
+"packets"
+],
+"value": 1
+}
+],
+"teardown": [
+"$TC qdisc d
[PATCH net-next 2/3] tc-testing: Allow tdc plugins to see test case data
Instead of only passing the test case name and ID, pass the
entire current test case down to the plugins. This change
allows plugins to start accepting commands and directives
from the test cases themselves, for greater flexibility
in testing.
---
tools/testing/selftests/tc-testing/TdcPlugin.py | 5 ++---
tools/testing/selftests/tc-testing/tdc.py | 10 +-
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
b/tools/testing/selftests/tc-testing/TdcPlugin.py
index b980a56..79f3ca8 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,12 +18,11 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
-def pre_case(self, testid, test_name, test_skip):
+def pre_case(self, caseinfo, test_skip):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
-self.args.testid = testid
-self.args.test_name = test_name
+self.args.caseinfo = caseinfo
self.args.test_skip = test_skip
def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 1afa803..de7da9a 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -126,15 +126,15 @@ class PluginMgr:
for pgn_inst in reversed(self.plugin_instances):
pgn_inst.post_suite(index)
-def call_pre_case(self, testid, test_name, *, test_skip=False):
+def call_pre_case(self, caseinfo, *, test_skip=False):
for pgn_inst in self.plugin_instances:
try:
-pgn_inst.pre_case(testid, test_name, test_skip)
+pgn_inst.pre_case(caseinfo, test_skip)
except Exception as ee:
print('exception {} in call to pre_case for {} plugin'.
format(ee, pgn_inst.__class__))
print('test_ordinal is {}'.format(test_ordinal))
-print('testid is {}'.format(testid))
+print('testid is {}'.format(caseinfo['id']))
raise
def call_post_case(self):
@@ -379,14 +379,14 @@ def run_one_test(pm, args, index, tidx):
res = TestResult(tidx['id'], tidx['name'])
res.set_result(ResultState.skip)
res.set_errormsg('Test case designated as skipped.')
-pm.call_pre_case(tidx['id'], tidx['name'], test_skip=True)
+pm.call_pre_case(tidx, test_skip=True)
pm.call_post_execute()
return res
# populate NAMES with TESTID for this test
NAMES['TESTID'] = tidx['id']
-pm.call_pre_case(tidx['id'], tidx['name'])
+pm.call_pre_case(tidx)
prepare_env(args, pm, 'setup', "-> prepare stage", tidx["setup"])
if (args.verbose > 0):
--
2.7.4
[PATCH net-next 1/3] tc-testing: Add JSON verification to tdc
This patch allows tdc to process JSON output to perform secondary
verification of the command under test. If the verifyCmd generates
JSON, one can provide the 'matchJSON' key to process it
instead of a regex.
matchJSON has two elements: 'path' and 'value'. The 'path' key is a
list of integers and strings that provide the key values for tdc to
navigate the JSON information. The value is an integer or string
that tdc will compare against what it finds in the provided path.
If the numerical position of an element can vary, it's possible to
substitute an asterisk as a wildcard. tdc will search all possible
entries in the array.
Multiple matches are possible, but everything specified must
match for the test to pass.
If both matchPattern and matchJSON are present, tdc will only
operate on matchPattern. If neither are present, verification
is skipped.
Example:
"cmdUnderTest": "$TC actions add action pass index 8",
"verifyCmd": "$TC actions list action gact",
"matchJSON": [
{
"path": [
0,
"actions",
0,
"control action",
"type"
],
"value": "gact"
},
{
"path": [
0,
"actions",
0,
"index"
],
"value": 8
}
]
---
tools/testing/selftests/tc-testing/tdc.py | 136 +++---
1 file changed, 123 insertions(+), 13 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 678182a..1afa803 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -35,6 +35,10 @@ class PluginMgrTestFail(Exception):
self.output = output
self.message = message
+class ElemNotFound(Exception):
+def __init__(self, path_element):
+self.path_element = path_element
+
class PluginMgr:
def __init__(self, argparser):
super().__init__()
@@ -167,6 +171,40 @@ class PluginMgr:
self.argparser = argparse.ArgumentParser(
description='Linux TC unit tests')
+def find_in_json(jsonobj, path):
+if type(jsonobj) == list:
+if type(path[0]) == int:
+if len(jsonobj) > path[0]:
+return find_in_json(jsonobj[path[0]], path[1:])
+else:
+raise ElemNotFound(path[0])
+elif path[0] == '*':
+res = []
+for index in jsonobj:
+try:
+res.append(find_in_json(index, path[1:]))
+except ElemNotFound:
+continue
+if len(res) == 0:
+raise ElemNotFound(path[0])
+else:
+return res
+elif type(jsonobj) == dict:
+if path[0] in jsonobj:
+if len(path) > 1:
+return find_in_json(jsonobj[path[0]], path[1:])
+return jsonobj[path[0]]
+else:
+raise ElemNotFound(path[0])
+else:
+# Assume we have found the correct depth in the object
+if len(path) >= 1:
+print('The remainder of the specified path cannot be found!')
+print('Path values: {}'.format(path))
+raise ElemNotFound(path[0])
+return jsonobj
+
+
def replace_keywords(cmd):
"""
For a given executable command, substitute any known
@@ -246,6 +284,86 @@ def prepare_env(args, pm, stage, prefix, cmdlist, output =
None):
stage, output,
'"{}" did not complete successfully'.format(prefix))
+def verify_by_regex(res, tidx, args, pm):
+if 'matchCount' not in tidx:
+res.set_result(ResultState.skip)
+fmsg = 'matchCount was not provided in the test case. '
+fmsg += 'Unable to complete pattern match.'
+res.set_failmsg(fmsg)
+print(fmsg)
+return res
+(p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
+match_pattern = re.compile(
+str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE)
+if procout:
+match_index = re.findall(match_pattern, procout)
+if len(match_index) != int(tidx["matchCount"]):
+res.set_result(ResultState.fail)
+fmsg = 'Verify stage failed because the output did not match '
+fmsg += 'the pattern in the test case.\nMatch pattern is:\n'
+fmsg += '\t{}\n'.format(tidx["matchPattern"])
+fmsg += 'Output generated by the verify command:\n'
+fmsg += '{}\n'.format(procout)
+res.set_failmsg(fmsg)
+else:
+res.set_result(ResultState.success)
+elif int(tidx["matchCount"]) != 0:
+res.set_result(ResultState.fail)
+res.set_failmsg('No output generated by verify command.')
+else:
+res.set_result(ResultState.success)
+return res
+
+def verify_by_json(res, tidx, args, pm):
+# Validate the matchJSON struct
+f
[PATCH net-next 1/1] tc-testing: Restore original behaviour for namespaces in tdc
This patch restores the original behaviour for tdc prior to the
introduction of the plugin system, where the network namespace
functionality was split from the main script.
It introduces the concept of required plugins for testcases,
and will automatically load any plugin that isn't already
enabled when said plugin is required by even one testcase.
Additionally, the -n option for the nsPlugin is deprecated
so the default action is to make use of the namespaces.
Instead, we introduce -N to not use them, but still create
the veth pair.
buildebpfPlugin's -B option is also deprecated.
If a test cases requires the features of a specific plugin
in order to pass, it should instead include a new key/value
pair describing plugin interactions:
"plugins": {
"requires": "buildebpfPlugin"
},
A test case can have more than one required plugin: a list
can be inserted as the value for 'requires'.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/README | 22 ++-
.../tc-testing/plugin-lib/buildebpfPlugin.py | 5 +-
.../selftests/tc-testing/plugin-lib/nsPlugin.py| 26 +++-
.../selftests/tc-testing/tc-tests/actions/bpf.json | 6 +
.../selftests/tc-testing/tc-tests/filters/fw.json | 162 +
.../tc-testing/tc-tests/filters/tests.json | 12 ++
tools/testing/selftests/tc-testing/tdc.py | 78 +-
tools/testing/selftests/tc-testing/tdc_helper.py | 5 +-
8 files changed, 296 insertions(+), 20 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/README
b/tools/testing/selftests/tc-testing/README
index f9281e8..22e5da9 100644
--- a/tools/testing/selftests/tc-testing/README
+++ b/tools/testing/selftests/tc-testing/README
@@ -12,10 +12,10 @@ REQUIREMENTS
* Minimum Python version of 3.4. Earlier 3.X versions may work but are not
guaranteed.
-* The kernel must have network namespace support
+* The kernel must have network namespace support if using nsPlugin
* The kernel must have veth support available, as a veth pair is created
- prior to running the tests.
+ prior to running the tests when using nsPlugin.
* The kernel must have the appropriate infrastructure enabled to run all tdc
unit tests. See the config file in this directory for minimum required
@@ -53,8 +53,12 @@ commands being tested must be run as root. The code that
enforces
execution by root uid has been moved into a plugin (see PLUGIN
ARCHITECTURE, below).
-If nsPlugin is linked, all tests are executed inside a network
-namespace to prevent conflicts within the host.
+Tests that use a network device should have nsPlugin.py listed as a
+requirement for that test. nsPlugin executes all commands within a
+network namespace and creates a veth pair which may be used in those test
+cases. To disable execution within the namespace, pass the -N option
+to tdc when starting a test run; the veth pair will still be created
+by the plugin.
Running tdc without any arguments will run all tests. Refer to the section
on command line arguments for more information, or run:
@@ -154,8 +158,8 @@ action:
netns:
options for nsPlugin (run commands in net namespace)
- -n, --namespace
-Run commands in namespace as specified in tdc_config.py
+ -N, --no-namespace
+Do not run commands in a network namespace.
valgrind:
options for valgrindPlugin (run command under test under Valgrind)
@@ -171,7 +175,8 @@ was in the tdc.py script has been moved into the plugins.
The plugins are in the directory plugin-lib. The are executed from
directory plugins. Put symbolic links from plugins to plugin-lib,
-and name them according to the order you want them to run.
+and name them according to the order you want them to run. This is not
+necessary if a test case being run requires a specific plugin to work.
Example:
@@ -223,7 +228,8 @@ directory:
- rootPlugin.py:
implements the enforcement of running as root
- nsPlugin.py:
- sets up a network namespace and runs all commands in that namespace
+ sets up a network namespace and runs all commands in that namespace,
+ while also setting up dummy devices to be used in testing.
- valgrindPlugin.py
runs each command in the execute stage under valgrind,
and checks for leaks.
diff --git a/tools/testing/selftests/tc-testing/plugin-lib/buildebpfPlugin.py
b/tools/testing/selftests/tc-testing/plugin-lib/buildebpfPlugin.py
index 9f0ba10..e98c367 100644
--- a/tools/testing/selftests/tc-testing/plugin-lib/buildebpfPlugin.py
+++ b/tools/testing/selftests/tc-testing/plugin-lib/buildebpfPlugin.py
@@ -34,8 +34,9 @@ class SubPlugin(TdcPlugin):
'buildebpf',
'options for buildebpfPlugin')
self.argparser_group.add_argument(
-'-B', '--buildebpf', action='store_tr
Re: [RFC PATCH net-next 1/1] tc-testing: Restore original behaviour for namespaces in tdc
On Tue, Jun 18, 2019 at 4:52 AM Nicolas Dichtel wrote: > >> From my point of view, if all tests are not successful by default, it > >> scares > >> users and prevent them to use those tests suite to validate their patches. > > > > For me, explicitly telling the user that a test was skipped, and /why/ > > it was skipped, is far better than excluding the test from the > > results: I don't want to waste someone's time with troubleshooting the > > script if they're expecting to see results for those tests when > > running tdc and nothing appears, or worse yet, stop using it because > > they think it doesn't work properly. > > > > I do believe the skip message should be improved so it better > > indicates why those tests are being skipped. And the '-d' feature > > should be documented. How do these changes sound? > If the error message is clear enough, I agree with you. The skip message > should > not feel like an error message ;-) Very true. I think I just put that one in quickly and meant to come back to it later, but either way it's a bit too vague. I'll get that corrected, but I believe I'll add it to a separate patch after the requires functionality goes in. I want to update some of the documentation as well. Thanks, Lucas
Re: [RFC PATCH net-next 1/1] tc-testing: Restore original behaviour for namespaces in tdc
On Fri, Jun 14, 2019 at 5:37 AM Nicolas Dichtel wrote: > > Le 05/06/2019 à 23:08, Lucas Bates a écrit : > > Apologies for the delay in getting this out. I've been busy > > with other things and this change was a little trickier than > > I expected. > > > > This patch restores the original behaviour for tdc prior to the > > introduction of the plugin system, where the network namespace > > functionality was split from the main script. > > > > It introduces the concept of required plugins for testcases, > > and will automatically load any plugin that isn't already > > enabled when said plugin is required by even one testcase. > > > > Additionally, the -n option for the nsPlugin is deprecated > > so the default action is to make use of the namespaces. > > Instead, we introduce -N to not use them, but still create > > the veth pair. > > > > Comments welcome! > Thanks for the follow up. I successfully tested your patch, it fixes the > netns case. Good, thank you for checking it out. I have to add a few more changes to the patch for the BPF-related tests, but once that's done I'll submit the finished version. > Note that there is still a bunch of test that fails or are skipped after your > patch, for example: > ok 431 e41d - Add 1M flower filters with 10 parallel tc instances # skipped - > Not executed because DEV2 is not defined > The message is not really explicit, you have to dig into the code to > understand > that '-d ' is needed. > Is it not possible to use a dummy interface by default? The tests that make use of DEV2 are intended to be run with a physical NIC. This feature was originally submitted by Chris Mi from Mellanox back in 2017 (commit 31c2611b) to reproduce a kernel panic, with d052 being the first test case submitted. Originally they were silently skipped, but once I added TdcResults.py this changed so they would be tracked and reported as skipped. > From my point of view, if all tests are not successful by default, it scares > users and prevent them to use those tests suite to validate their patches. For me, explicitly telling the user that a test was skipped, and /why/ it was skipped, is far better than excluding the test from the results: I don't want to waste someone's time with troubleshooting the script if they're expecting to see results for those tests when running tdc and nothing appears, or worse yet, stop using it because they think it doesn't work properly. I do believe the skip message should be improved so it better indicates why those tests are being skipped. And the '-d' feature should be documented. How do these changes sound? Thanks, Lucas
[RFC PATCH net-next 1/1] tc-testing: Scapy plugin and JSON verification for tdc
This is a draft version of two new tdc features.
First, the scapy plugin. This requires the scapy Python module
installed on the system (plugin was tested against v2.4.2).
The intent is to install a given rule (as the command under test)
and then generate packets to create statistics for that rule.
The stats are checked in the verify phase.
A new "scapy" entry appears in the test cases, which currently
have three requirements: the source interface for the packets,
the number of packets to be sent, and a string that is processed
by scapy's eval() function to construct the packets.
Limitations: For now, only one type of packet can be crafted
per test case. Also, knowledge of scapy's syntax is required.
Secondly, we add JSON processing as a method of performing the
verification stage. Each test case can now have a "matchPattern"
or "matchJSON" field which governs the method tdc will use to
process the results. The intent is to make it easier to handle
the verify stage by not requiring complex regular expressions
matchJSON has two fields, path and value. Path is a list of
strings and integers which indicate the path through the nested
JSON data - an asterisk is also acceptable in place of
a number if the specific index of a list is unknown.
This structure may not be the best method of handling JSON
verification - suggestions have been made that include using a
third party module to process the JSON, but that creates an
external dependency for tdc.
To try the sample tests in this patch:
1) Ensure nsPlugin and scapyPlugin are linked in the plugins/
subdirectory
2) Run:
sudo ./tdc.py -f tc-tests/actions/scapy-example.json -n
The second test is designed to fail.
Comments and discussion are encouraged!
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/TdcPlugin.py| 5 +-
.../selftests/tc-testing/plugin-lib/scapyPlugin.py | 49 +++
.../tc-testing/tc-tests/actions/scapy-example.json | 86
tools/testing/selftests/tc-testing/tdc.py | 149 ++---
4 files changed, 268 insertions(+), 21 deletions(-)
create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/scapy-example.json
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
b/tools/testing/selftests/tc-testing/TdcPlugin.py
index b980a56..79f3ca8 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,12 +18,11 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
-def pre_case(self, testid, test_name, test_skip):
+def pre_case(self, caseinfo, test_skip):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
-self.args.testid = testid
-self.args.test_name = test_name
+self.args.caseinfo = caseinfo
self.args.test_skip = test_skip
def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
new file mode 100644
index 000..b3ffc24
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import signal
+from string import Template
+import subprocess
+import time
+from TdcPlugin import TdcPlugin
+
+from tdc_config import *
+
+try:
+from scapy.all import *
+except ImportError:
+print("Unable to import the scapy python module. Is it installed?")
+exit(1)
+
+class SubPlugin(TdcPlugin):
+def __init__(self):
+self.sub_class = 'scapy/SubPlugin'
+super().__init__()
+
+def post_execute(self):
+if 'scapy' not in self.args.caseinfo:
+if self.args.verbose:
+print('{}.post_execute: no scapy info in test
case'.format(self.sub_class))
+return
+
+# Check for required fields
+scapyinfo = self.args.caseinfo['scapy']
+scapy_keys = ['iface', 'count', 'eval']
+missing_keys = []
+keyfail = False
+for k in scapy_keys:
+if k not in scapyinfo:
+keyfail = True
+missing_keys.add(k)
+if keyfail:
+print('{}: Scapy block present in the test, but is missing info:'
+.format(self.sub_class))
+print('{}'.format(missing_keys))
+
+pkt = eval(scapyinfo['eval'])
+if '$' in scapyinfo['iface']:
+tpl = Template(scapyinfo['iface'])
+scapyinfo['iface'] = tpl.safe_subs
Re: [RFC PATCH net-next 1/1] tc-testing: Restore original behaviour for namespaces in tdc
On Fri, Jun 7, 2019 at 8:18 AM Davide Caratti wrote: > From what I see, it is a fix for the reported problem (e.g. tests failing > because of 'nsPlugin' uninstalled). And, I want to followup fixing the > bpf.json in tc-actions, so that > > # ./tdc.py -l -c bpf | grep eBPF > e939: (actions, bpf) Add eBPF action with valid object-file > 282d: (actions, bpf) Add eBPF action with invalid object-file > > require the buildebpfPlugin (unless anybody disagrees, I will also revert > the meaning of '-B' also, like you did for '-n') Yes, those should definitely be tagged. I probably should have included them in the patch. As for the -B option, that would probably be a good idea. Required plugins, I think, should always be on and require a user to explicitly not use them. So in that case, nsPlugin and buildepfPlugin, being required, should have options to explicitly disable their features. valgrindPlugin, since it's not explicitly required to run any given test, doesn't need that. Maybe we should separate the plugins into different directories based on this? > > few comments after a preliminary test: > 1) the patch still does not cover the two categories that use $DEV2 (i.e. > flower and concurrency still fail in my environment) I will have to try that out and see what is going on. I didn't try the $DEV2 tests against this. > 2) I've been reported, and reproduced with latest fedora, a problem in > nsPlugin.py. All tests in the 'filter' category still fail, unless I do > > # sed -i "s#ip#/sbin/ip#g" nsPlugin.py > > otherwise, the 'prepare' stage fails: > > # ./tdc.py -e 5339 > -- ns/SubPlugin.__init__ > Test 5339: Del entire fw filter > > -> prepare stage *** Could not execute: "$TC qdisc add dev $DEV1 ingress" > > -> prepare stage *** Error message: "/bin/sh: ip: command not found > " > returncode 127; expected [0] > > -> prepare stage *** Aborting test run. > > (maybe we should use a variable for that, instead of hardcoded command > name, like we do for $TC ?) Ooh... Yes, yes we should. Sounds like the new version of Fedora isn't including sbin on the path anymore? That's going to require another minor change that I think I'll submit in a separate patch. Thanks, Davide!
[RFC PATCH net-next 1/1] tc-testing: Restore original behaviour for namespaces in tdc
Apologies for the delay in getting this out. I've been busy with other things and this change was a little trickier than I expected. This patch restores the original behaviour for tdc prior to the introduction of the plugin system, where the network namespace functionality was split from the main script. It introduces the concept of required plugins for testcases, and will automatically load any plugin that isn't already enabled when said plugin is required by even one testcase. Additionally, the -n option for the nsPlugin is deprecated so the default action is to make use of the namespaces. Instead, we introduce -N to not use them, but still create the veth pair. Comments welcome! --- tools/testing/selftests/tc-testing/README | 22 ++- .../selftests/tc-testing/plugin-lib/nsPlugin.py| 26 +++- .../selftests/tc-testing/tc-tests/filters/fw.json | 162 + .../tc-testing/tc-tests/filters/tests.json | 12 ++ tools/testing/selftests/tc-testing/tdc.py | 78 +- tools/testing/selftests/tc-testing/tdc_helper.py | 5 +- 6 files changed, 287 insertions(+), 18 deletions(-) diff --git a/tools/testing/selftests/tc-testing/README b/tools/testing/selftests/tc-testing/README index f9281e8..22e5da9 100644 --- a/tools/testing/selftests/tc-testing/README +++ b/tools/testing/selftests/tc-testing/README @@ -12,10 +12,10 @@ REQUIREMENTS * Minimum Python version of 3.4. Earlier 3.X versions may work but are not guaranteed. -* The kernel must have network namespace support +* The kernel must have network namespace support if using nsPlugin * The kernel must have veth support available, as a veth pair is created - prior to running the tests. + prior to running the tests when using nsPlugin. * The kernel must have the appropriate infrastructure enabled to run all tdc unit tests. See the config file in this directory for minimum required @@ -53,8 +53,12 @@ commands being tested must be run as root. The code that enforces execution by root uid has been moved into a plugin (see PLUGIN ARCHITECTURE, below). -If nsPlugin is linked, all tests are executed inside a network -namespace to prevent conflicts within the host. +Tests that use a network device should have nsPlugin.py listed as a +requirement for that test. nsPlugin executes all commands within a +network namespace and creates a veth pair which may be used in those test +cases. To disable execution within the namespace, pass the -N option +to tdc when starting a test run; the veth pair will still be created +by the plugin. Running tdc without any arguments will run all tests. Refer to the section on command line arguments for more information, or run: @@ -154,8 +158,8 @@ action: netns: options for nsPlugin (run commands in net namespace) - -n, --namespace -Run commands in namespace as specified in tdc_config.py + -N, --no-namespace +Do not run commands in a network namespace. valgrind: options for valgrindPlugin (run command under test under Valgrind) @@ -171,7 +175,8 @@ was in the tdc.py script has been moved into the plugins. The plugins are in the directory plugin-lib. The are executed from directory plugins. Put symbolic links from plugins to plugin-lib, -and name them according to the order you want them to run. +and name them according to the order you want them to run. This is not +necessary if a test case being run requires a specific plugin to work. Example: @@ -223,7 +228,8 @@ directory: - rootPlugin.py: implements the enforcement of running as root - nsPlugin.py: - sets up a network namespace and runs all commands in that namespace + sets up a network namespace and runs all commands in that namespace, + while also setting up dummy devices to be used in testing. - valgrindPlugin.py runs each command in the execute stage under valgrind, and checks for leaks. diff --git a/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py b/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py index a194b1a..7042618 100644 --- a/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py +++ b/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py @@ -18,6 +18,8 @@ class SubPlugin(TdcPlugin): if self.args.namespace: self._ns_create() +else: +self._ports_create() def post_suite(self, index): '''run commands after test_runner goes into a test loop''' @@ -27,6 +29,8 @@ class SubPlugin(TdcPlugin): if self.args.namespace: self._ns_destroy() +else: +self._ports_destroy() def add_args(self, parser): super().add_args(parser) @@ -34,8 +38,8 @@ class SubPlugin(TdcPlugin): 'netns', 'options for nsPlugin(run commands in net namespace)') self.argparser_group.add_argument( -'-n', '--namespace', action='store_true', -
Re: [RFC net-next 1/1] tdc.py: Introduce required plugins
On Fri, Apr 12, 2019 at 7:37 PM Nicolas Dichtel
wrote:
>
> Le 12/04/2019 à 18:07, Lucas Bates a écrit :
> [snip]
> > So something like this? Note the usage of the keyword "requires",
> >
> > {
> > "id": "901f",
> > "name": "Add fw filter with prio at 32-bit maxixum",
> > "category": [
> > "filter",
> > "fw"
> > ],
> > "requires": ["nsPlugin"],
> Yes.
OK, thanks. I'll work on getting that together and have a patch out
this week to resolve it.
I'm still open to feedback on this issue if anyone would like to weigh in.
Thanks,
Lucas
Re: [RFC net-next 1/1] tdc.py: Introduce required plugins
On Fri, Apr 12, 2019 at 11:42 AM Nicolas Dichtel
wrote:
>
> Le 12/04/2019 à 17:21, Lucas Bates a écrit :
> > On Fri, Apr 12, 2019 at 4:31 AM Nicolas Dichtel
> > wrote:
> >>> in our tri-weekly tc test meeting and the general consensus to address
> >>> what you brought up is leaning towards the following:
> >>>
> >>> - adding a symlink to nsPlugin
> >> I don't understand why a symlink is needed. Just load it by default and
> >> use it
> >> when needed. A property can be added to each test to tell which plugin (in
> >> fact,
> >> which topology) is needed to run it.
> >> Thus, if a new complex test is added, it can define another topology.
> >
> > I'm loathe to hard-code it into the script, but we can avoid the
> > symlink if we specify some default plugins to load in the
> > tdc_config.py file.
> Why is it a problem to specify the plugin in the test description? The fw
> tests
> depend on a specific topology, thus it's better explicitly state that.
So something like this? Note the usage of the keyword "requires",
{
"id": "901f",
"name": "Add fw filter with prio at 32-bit maxixum",
"category": [
"filter",
"fw"
],
"requires": ["nsPlugin"],
"setup": [
"$TC qdisc add dev $DEV1 ingress"
],
"cmdUnderTest": "$TC filter add dev $DEV1 parent : handle
1 prio 65535 fw action ok",
"expExitCode": "0",
"verifyCmd": "$TC filter get dev $DEV1 parent : handle 1
prio 65535 protocol all fw",
"matchPattern": "pref 65535 fw.*handle 0x1.*gact action pass",
"matchCount": "1",
"teardown": [
"$TC qdisc del dev $DEV1 ingress"
]
},
> >
> >>> - Changing default behaviour so that unless an option is explicitly
> >>> specified, all the tests will be run under a namespace with automatic
> >>> creation of the ports
> >> Yes.
> >>
> >>> - If the user chooses /not/ to use namespaces, it will still create
> >>> the veth pair to use.
> >> In fact, I would say an option so that the user can choose another
> >> topology.
> >
> > Quite possibly, but for traffic generation (upcoming using scapy)
> > we're just reusing the simple topology right now.
> I was thinking you were arguing for this option. If nobody need it, let's just
> remove it.
>
> >
> >>>> After your patch, I got the following error:
> >>>> $ ./tdc.py
> >>>> Traceback (most recent call last):
> >>>> File "./tdc.py", line 740, in
> >>>> main()
> >>>> File "./tdc.py", line 734, in main
> >>>> set_operation_mode(pm, args)
> >>>> File "./tdc.py", line 692, in set_operation_mode
> >>>> check_required_plugins(pm, alltests)
> >>>> File "./tdc.py", line 583, in check_required_plugins
> >>>> os.chown('plugins/{}'.format(fname), uid=int(os.getenv('SUDO_UID')),
> >>>> TypeError: int() argument must be a string or a number, not 'NoneType'
> >>>
> >>> That would be expected if you aren't running tdc with sudo or as root
> >>> or as a user with network admin capability.
> >> I was root for the test.
> >
> > Ah. That environment variable is probably not present on your system.
> > Out of curiosity, what shell/distribution are you using?
> >
> It was on a debian 8.11 / bash.
>
Thanks. I'll look into it later.
Re: [RFC net-next 1/1] tdc.py: Introduce required plugins
On Fri, Apr 12, 2019 at 4:31 AM Nicolas Dichtel
wrote:
> > in our tri-weekly tc test meeting and the general consensus to address
> > what you brought up is leaning towards the following:
> >
> > - adding a symlink to nsPlugin
> I don't understand why a symlink is needed. Just load it by default and use it
> when needed. A property can be added to each test to tell which plugin (in
> fact,
> which topology) is needed to run it.
> Thus, if a new complex test is added, it can define another topology.
I'm loathe to hard-code it into the script, but we can avoid the
symlink if we specify some default plugins to load in the
tdc_config.py file.
> > - Changing default behaviour so that unless an option is explicitly
> > specified, all the tests will be run under a namespace with automatic
> > creation of the ports
> Yes.
>
> > - If the user chooses /not/ to use namespaces, it will still create
> > the veth pair to use.
> In fact, I would say an option so that the user can choose another topology.
Quite possibly, but for traffic generation (upcoming using scapy)
we're just reusing the simple topology right now.
> >> After your patch, I got the following error:
> >> $ ./tdc.py
> >> Traceback (most recent call last):
> >> File "./tdc.py", line 740, in
> >> main()
> >> File "./tdc.py", line 734, in main
> >> set_operation_mode(pm, args)
> >> File "./tdc.py", line 692, in set_operation_mode
> >> check_required_plugins(pm, alltests)
> >> File "./tdc.py", line 583, in check_required_plugins
> >> os.chown('plugins/{}'.format(fname), uid=int(os.getenv('SUDO_UID')),
> >> TypeError: int() argument must be a string or a number, not 'NoneType'
> >
> > That would be expected if you aren't running tdc with sudo or as root
> > or as a user with network admin capability.
> I was root for the test.
Ah. That environment variable is probably not present on your system.
Out of curiosity, what shell/distribution are you using?
Re: [RFC net-next 1/1] tdc.py: Introduce required plugins
On Wed, Apr 10, 2019 at 10:33 AM Nicolas Dichtel
wrote:
>
> Le 09/04/2019 à 23:44, Lucas Bates a écrit :
> > Some of the testcases (for example, all of the fw tests) in tdc
> > require activating the nsplugin. This RFC introduces a feature which
> > tags one such test with the keyword "requires". Anyone running a test
> > that requires nsplugin will now get a warning if they are missing
> > the plugin.
> >
> > After compiling the list of test cases to execute, tdc will
> > gather all of the required plugins for that run, and validate
> > that they have been enabled with a symlink in the plugins/
> > directory. If required plugins are missing, tdc will create the
> > symlink for the user and then terminate. (This is because plugin-
> > specific options may exist and need to be parsed at startup)
> I still don't understand the goal of this plugin. Why not simply include it in
> the core code (like it was some times ago)?
Because the use of namespaces and the veth pair are specific to some
test cases. We're also making it possible to create complex test cases
which will require more than one namespace. The starting point was to
isolate the namespace into its own plugin. We had a discussion today
in our tri-weekly tc test meeting and the general consensus to address
what you brought up is leaning towards the following:
- adding a symlink to nsPlugin
- Changing default behaviour so that unless an option is explicitly
specified, all the tests will be run under a namespace with automatic
creation of the ports
- If the user chooses /not/ to use namespaces, it will still create
the veth pair to use.
I'll send the patch tomorrow as another RFC.
What do you think of this?
> > Please provide feedback. If this is amenable to all, I will proceed
> > to submit for the rest.
> After your patch, I got the following error:
> $ ./tdc.py
> Traceback (most recent call last):
> File "./tdc.py", line 740, in
> main()
> File "./tdc.py", line 734, in main
> set_operation_mode(pm, args)
> File "./tdc.py", line 692, in set_operation_mode
> check_required_plugins(pm, alltests)
> File "./tdc.py", line 583, in check_required_plugins
> os.chown('plugins/{}'.format(fname), uid=int(os.getenv('SUDO_UID')),
> TypeError: int() argument must be a string or a number, not 'NoneType'
That would be expected if you aren't running tdc with sudo or as root
or as a user with network admin capability.
[RFC net-next 1/1] tdc.py: Introduce required plugins
Some of the testcases (for example, all of the fw tests) in tdc
require activating the nsplugin. This RFC introduces a feature which
tags one such test with the keyword "requires". Anyone running a test
that requires nsplugin will now get a warning if they are missing
the plugin.
After compiling the list of test cases to execute, tdc will
gather all of the required plugins for that run, and validate
that they have been enabled with a symlink in the plugins/
directory. If required plugins are missing, tdc will create the
symlink for the user and then terminate. (This is because plugin-
specific options may exist and need to be parsed at startup)
Please provide feedback. If this is amenable to all, I will proceed
to submit for the rest.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/README | 28 ++-
.../selftests/tc-testing/tc-tests/filters/fw.json | 1 +
tools/testing/selftests/tc-testing/tdc.py | 41 ++
3 files changed, 62 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/README
b/tools/testing/selftests/tc-testing/README
index f9281e8..b91ce8b 100644
--- a/tools/testing/selftests/tc-testing/README
+++ b/tools/testing/selftests/tc-testing/README
@@ -36,14 +36,20 @@ REQUIREMENTS
BEFORE YOU RUN
--
-The path to the tc executable that will be most commonly tested can be defined
-in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and
-define the path.
-If you need to test a different tc executable on the fly, you can do so by
-using the -p option when running tdc:
+* The path to the tc executable that will be most commonly tested can be
+ defined in the tdc_config.py file. Find the 'TC' entry in the NAMES
+ dictionary and define the path.
+
+ If you need to test a different tc executable on the fly, you can do so by
+ using the -p option when running tdc:
./tdc.py -p /path/to/tc
+* Tests may be executed in a network namespace with a port that is created
+ automatically. To use this feature, create a symlink in the plugins/
+ directory to plugin-lib/nsPlugin.py and use the '-n' option when
+ executing tdc. If this is not done, you MUST have a port with the same
+ name as the value of DEV1 in tdc_config.py before running tdc.
RUNNING TDC
---
@@ -53,9 +59,6 @@ commands being tested must be run as root. The code that
enforces
execution by root uid has been moved into a plugin (see PLUGIN
ARCHITECTURE, below).
-If nsPlugin is linked, all tests are executed inside a network
-namespace to prevent conflicts within the host.
-
Running tdc without any arguments will run all tests. Refer to the section
on command line arguments for more information, or run:
./tdc.py -h
@@ -235,6 +238,15 @@ directory:
- buildebpfPlugin.py:
builds all programs in $EBPFDIR.
+If a plugin is required for a test case to succeed, it can be defined in
+the test case definition as follows:
+ "requires": ["nsPlugin", "fooPlugin"]
+
+This will cause tdc to verify that these plugins are present before
+starting test execution. If they aren't, the symlinks will be created
+for you and tdc will terminate to be restarted. This is done as some
+plugins may have command line options that are needed for a successful run.
+
ACKNOWLEDGEMENTS
diff --git a/tools/testing/selftests/tc-testing/tc-tests/filters/fw.json
b/tools/testing/selftests/tc-testing/tc-tests/filters/fw.json
index 3b97cfd..dfba083 100644
--- a/tools/testing/selftests/tc-testing/tc-tests/filters/fw.json
+++ b/tools/testing/selftests/tc-testing/tc-tests/filters/fw.json
@@ -6,6 +6,7 @@
"filter",
"fw"
],
+ "requires": ["nsPlugin"],
"setup": [
"$TC qdisc add dev $DEV1 ingress"
],
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 5cee156..aa9a804 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -550,6 +550,46 @@ def filter_tests_by_category(args, testlist):
return answer
+def check_required_plugins(pm, testlist):
+'''
+Get all required plugins from the list of test cases and ensure
+they're enabled.
+'''
+reqs = []
+pnf = False
+plinked = False
+for t in testlist:
+try:
+reqs.extend(t['requires'])
+except KeyError:
+continue
+reqs = get_unique_item(reqs)
+
+for r in reqs:
+if r not in pm.plugins:
+import glob
+fname = '{}.py'.format(r)
+source_path = []
+source_path.extend(glob.glob('plugin-lib/{}'.format(fname)))
+source_path
Re: [PATCH net] selftests: add a tc matchall test case
On Sun, Apr 7, 2019 at 10:32 PM David Miller wrote: > > I'm not sure to understand the goal of this series: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?h=45b3a3762721 > > > > But after it, tc selftests are not intuitive anymore. When naively running > > them > > (./tdy.py), all fw filter tests fail. It's hard to understand that the > > problem > > is that they require the 'nsPlugin' plugin to be enabled (by adding a > > symlink > > *and* providing the '-n' option to tdc.py). > > > > Brenda, why putting the netns code into a plugin that is disabled by > > default, > > knowing that this code is mandatory to run the test? > > Brenda, please respond. This would be a documentation issue - the usage of nsPlugin is not covered in the README. I'll submit a patch by end of the day tomorrow to address it. The namespace code isn't mandatory, strictly speaking. It's not required at all when running the action tests, only with the filter tests as those need a specific device to be attached to. The actual problem is that the nsPlugin creates a veth pair using the names found in the config file (DEV0/DEV1) - but if you aren't using the plugin, then a port won't be created. The intention is that you'd create a port manually before starting the tests.
Re: [RFC PATCH net-next] selftests: tc-testing: Add pedit tests
On Fri, Mar 1, 2019 at 7:33 AM Dmytro Linkin wrote: > > Add 36 pedit action tests to check pedit options described in tc-pedit(8) > man page. Test cases can be specified by categories: actions, pedit, > raw_op, layered_op. RAW_OP cases check offset option for u8, u16 and u32 > offset size. LAYERED_OP cases check fields option for eth, ip, ip6, > tcp and udp headers. Thanks Dmytro, the tests look pretty complete. I like seeing all the options exercised even if the impact is low. Unless someone has an objection to any of the commands being tested (i.e. doesn't make sense) I'd say go ahead and submit. Thanks, Lucas > > Include following tests: > 377e - Add pedit action with RAW_OP offset u32 > a0ca - Add pedit action with RAW_OP offset u32 (INVALID) > dd8a - Add pedit action with RAW_OP offset u16 u16 > 53db - Add pedit action with RAW_OP offset u16 (INVALID) > 5c7e - Add pedit action with RAW_OP offset u8 add value > 2893 - Add pedit action with RAW_OP offset u8 quad > 3a07 - Add pedit action with RAW_OP offset u8-u16-u8 > ab0f - Add pedit action with RAW_OP offset u16-u8-u8 > 9d12 - Add pedit action with RAW_OP offset u32 set u16 clear u8 invert > ebfa - Add pedit action with RAW_OP offset overflow u32 (INVALID) > f512 - Add pedit action with RAW_OP offset u16 at offmask shift set > c2cb - Add pedit action with RAW_OP offset u32 retain value > 86d4 - Add pedit action with LAYERED_OP eth set src & dst > c715 - Add pedit action with LAYERED_OP eth set src (INVALID) > ba22 - Add pedit action with LAYERED_OP eth type set/clear sequence > 5810 - Add pedit action with LAYERED_OP ip set src & dst > 1092 - Add pedit action with LAYERED_OP ip set ihl & dsfield > 02d8 - Add pedit action with LAYERED_OP ip set ttl & protocol > 3e2d - Add pedit action with LAYERED_OP ip set ttl (INVALID) > 31ae - Add pedit action with LAYERED_OP ip ttl clear/set > 486f - Add pedit action with LAYERED_OP ip set duplicate fields > e790 - Add pedit action with LAYERED_OP ip set ce, df, mf, firstfrag, nofrag > fields > 6829 - Add pedit action with LAYERED_OP beyond ip set dport & sport > afd8 - Add pedit action with LAYERED_OP beyond ip set icmp_type & icmp_code > 3143 - Add pedit action with LAYERED_OP beyond ip set dport (INVALID) > fc1f - Add pedit action with LAYERED_OP ip6 set src & dst > 6d34 - Add pedit action with LAYERED_OP ip6 dst retain value (INVALID) > 6f5e - Add pedit action with LAYERED_OP ip6 flow_lbl > 6795 - Add pedit action with LAYERED_OP ip6 set payload_len, nexthdr, hoplimit > 1442 - Add pedit action with LAYERED_OP tcp set dport & sport > b7ac - Add pedit action with LAYERED_OP tcp sport set (INVALID) > cfcc - Add pedit action with LAYERED_OP tcp flags set > 3bc4 - Add pedit action with LAYERED_OP tcp set dport, sport & flags fields > f1c8 - Add pedit action with LAYERED_OP udp set dport & sport > d784 - Add pedit action with mixed RAW/LAYERED_OP #1 > 70ca - Add pedit action with mixed RAW/LAYERED_OP #2
[PATCH net-next] tc-testing: Allow test cases to be skipped
By adding a check for an optional key/value pair to the test case
data, individual test cases may be skipped to prevent tdc from
aborting a test run due to setup or teardown failure.
If a test case is skipped, it will still appear in the results
output to allow for a consistent number of executed tests in each
run. However, the test will be marked as skipped.
This support for skipping extends to any plugins that may generate
additional results for each executed test.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/TdcPlugin.py | 4 ++--
.../tc-testing/creating-testcases/AddingTestCases.txt| 5 +
.../selftests/tc-testing/plugin-lib/valgrindPlugin.py| 16
tools/testing/selftests/tc-testing/tdc.py| 15 ---
4 files changed, 27 insertions(+), 13 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
b/tools/testing/selftests/tc-testing/TdcPlugin.py
index 1d9e279..b980a56 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,13 +18,13 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
-def pre_case(self, test_ordinal, testid, test_name):
+def pre_case(self, testid, test_name, test_skip):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
self.args.testid = testid
self.args.test_name = test_name
-self.args.test_ordinal = test_ordinal
+self.args.test_skip = test_skip
def post_case(self):
'''run commands after test_runner does one test'''
diff --git
a/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
b/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
index 17b267d..a28571a 100644
--- a/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
+++ b/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
@@ -33,6 +33,11 @@ Each test case has required data:
id: A unique alphanumeric value to identify a particular test case
name: Descriptive name that explains the command under test
+skip: A completely optional key, if the corresponding value is "yes"
+ then tdc will not execute the test case in question. However,
+ this test case will still appear in the results output but
+ marked as skipped. This key can be placed anywhere inside the
+ test case at the top level.
category: A list of single-word descriptions covering what the command
under test is testing. Example: filter, actions, u32, gact, etc.
setup:The list of commands required to ensure the command under test
diff --git a/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
b/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
index e00c798..4bb8665 100644
--- a/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
+++ b/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
@@ -102,6 +102,14 @@ class SubPlugin(TdcPlugin):
if not self.args.valgrind:
return
+res = TestResult('{}-mem'.format(self.args.testid),
+ '{} memory leak check'.format(self.args.test_name))
+if self.args.test_skip:
+res.set_result(ResultState.skip)
+res.set_errormsg('Test case designated as skipped.')
+self._add_results(res)
+return
+
self.definitely_lost_re = re.compile(
r'definitely lost:\s+([,0-9]+)\s+bytes in\s+([,0-9]+)\sblocks',
re.MULTILINE | re.DOTALL)
self.indirectly_lost_re = re.compile(
@@ -134,8 +142,6 @@ class SubPlugin(TdcPlugin):
nle_num = int(nle_mo.group(1))
mem_results = ''
-res = TestResult('{}-mem'.format(self.args.testid),
- '{} memory leak check'.format(self.args.test_name))
if (def_num > 0) or (ind_num > 0) or (pos_num > 0) or (nle_num > 0):
mem_results += 'not '
res.set_result(ResultState.fail)
@@ -146,12 +152,6 @@ class SubPlugin(TdcPlugin):
self._add_results(res)
-mem_results += 'ok {} - {}-mem # {}\n'.format(
-self.args.test_ordinal, self.args.testid, 'memory leak check')
-self._add_to_tap(mem_results)
-if mem_results.startswith('not '):
-print('{}'.format(content))
-self._add_to_tap(content)
def _add_results(self, res):
self._tsr.add_resultdata(res)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
Re: [Patch net] net_sched: refetch skb protocol for each filter
On Sat, Jan 12, 2019 at 7:23 AM Jamal Hadi Salim wrote:
>
> On 2019-01-11 9:55 p.m., Cong Wang wrote:
> > Martin reported a set of filters don't work after changing
> > from reclassify to continue. Looking into the code, it
> > looks like skb protocol is not always fetched for each
> > iteration of the filters. But, as demonstrated by Martin,
> > TC actions could modify skb->protocol, for example act_vlan,
> > this means we have to refetch skb protocol in each iteration,
> > rather than using the one we fetch in the beginning of the loop.
> >
> > This bug is _not_ introduced by commit 3b3ae880266d
> > ("net: sched: consolidate tc_classify{,_compat}"), technically,
> > if act_vlan is the only action that modifies skb protocol, then
> > it is commit c7e2b9689ef8 ("sched: introduce vlan action") which
> > introduced this bug.
> >
>
> +Cc Lucas
> Do we have a test case for a setup like this in tdc?
> i.e incoming tagged and then vlan popped by action?
> Otherwise a test with IFE which resets the ethertype
> would be sufficient i.e just something that will messup
> with skb->protocol.
Sorry, I've been a little caught up in things around here.
There isn't currently a test case like this, but I can look at the
example Cong provided and see if I can make it tdc-friendly.
- Lucas
[PATCH net-next 1/4] tc-testing: Add command timeout feature to tdc
Using an attribute set in the tdc_config.py file, limit the
amount of time tdc will wait for an executed command to
complete and prevent the script from hanging entirely.
This timeout will be applied to all executed commands.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tdc.py| 16 +++-
tools/testing/selftests/tc-testing/tdc_config.py | 2 ++
2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 7607ba3..b862ee5 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -131,12 +131,16 @@ def exec_cmd(args, pm, stage, command):
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=ENVIR)
-(rawout, serr) = proc.communicate()
-if proc.returncode != 0 and len(serr) > 0:
-foutput = serr.decode("utf-8", errors="ignore")
-else:
-foutput = rawout.decode("utf-8", errors="ignore")
+try:
+(rawout, serr) = proc.communicate(timeout=NAMES['TIMEOUT'])
+if proc.returncode != 0 and len(serr) > 0:
+foutput = serr.decode("utf-8", errors="ignore")
+else:
+foutput = rawout.decode("utf-8", errors="ignore")
+except subprocess.TimeoutExpired:
+foutput = "Command \"{}\" timed out\n".format(command)
+proc.returncode = 255
proc.stdout.close()
proc.stderr.close()
@@ -438,6 +442,8 @@ def check_default_settings(args, remaining, pm):
NAMES['TC'] = args.path
if args.device != None:
NAMES['DEV2'] = args.device
+if 'TIMEOUT' not in NAMES:
+NAMES['TIMEOUT'] = None
if not os.path.isfile(NAMES['TC']):
print("The specified tc path " + NAMES['TC'] + " does not exist.")
exit(1)
diff --git a/tools/testing/selftests/tc-testing/tdc_config.py
b/tools/testing/selftests/tc-testing/tdc_config.py
index d651bc1..6d91e48 100644
--- a/tools/testing/selftests/tc-testing/tdc_config.py
+++ b/tools/testing/selftests/tc-testing/tdc_config.py
@@ -15,6 +15,8 @@ NAMES = {
'DEV1': 'v0p1',
'DEV2': '',
'BATCH_FILE': './batch.txt',
+ # Length of time in seconds to wait before terminating a command
+ 'TIMEOUT': 12,
# Name of the namespace to use
'NS': 'tcut',
# Directory containing eBPF test programs
--
2.7.4
[PATCH net-next 3/4] tc-testing: Implement the TdcResults module in tdc
In tdc and the valgrind plugin, begin using the TdcResults module
to track executed test cases.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/TdcPlugin.py| 3 +-
.../tc-testing/plugin-lib/valgrindPlugin.py| 22 +++-
tools/testing/selftests/tc-testing/tdc.py | 117 -
3 files changed, 91 insertions(+), 51 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
b/tools/testing/selftests/tc-testing/TdcPlugin.py
index 3ee9a6d..1d9e279 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,11 +18,12 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
-def pre_case(self, test_ordinal, testid):
+def pre_case(self, test_ordinal, testid, test_name):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
self.args.testid = testid
+self.args.test_name = test_name
self.args.test_ordinal = test_ordinal
def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
b/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
index 477a7bd..e00c798 100644
--- a/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
+++ b/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
@@ -11,6 +11,7 @@ from string import Template
import subprocess
import time
from TdcPlugin import TdcPlugin
+from TdcResults import *
from tdc_config import *
@@ -21,6 +22,7 @@ class SubPlugin(TdcPlugin):
def __init__(self):
self.sub_class = 'valgrind/SubPlugin'
self.tap = ''
+self._tsr = TestSuiteReport()
super().__init__()
def pre_suite(self, testcount, testidlist):
@@ -34,10 +36,14 @@ class SubPlugin(TdcPlugin):
def post_suite(self, index):
'''run commands after test_runner goes into a test loop'''
super().post_suite(index)
-self._add_to_tap('\n|---\n')
if self.args.verbose > 1:
print('{}.post_suite'.format(self.sub_class))
-print('{}'.format(self.tap))
+#print('{}'.format(self.tap))
+for xx in range(index - 1, self.testcount):
+res = TestResult('{}-mem'.format(self.testidlist[xx]), 'Test
skipped')
+res.set_result(ResultState.skip)
+res.set_errormsg('Skipped because of prior setup/teardown failure')
+self._add_results(res)
if self.args.verbose < 4:
subprocess.check_output('rm -f vgnd-*.log', shell=True)
@@ -128,8 +134,17 @@ class SubPlugin(TdcPlugin):
nle_num = int(nle_mo.group(1))
mem_results = ''
+res = TestResult('{}-mem'.format(self.args.testid),
+ '{} memory leak check'.format(self.args.test_name))
if (def_num > 0) or (ind_num > 0) or (pos_num > 0) or (nle_num > 0):
mem_results += 'not '
+res.set_result(ResultState.fail)
+res.set_failmsg('Memory leak detected')
+res.append_failmsg(content)
+else:
+res.set_result(ResultState.success)
+
+self._add_results(res)
mem_results += 'ok {} - {}-mem # {}\n'.format(
self.args.test_ordinal, self.args.testid, 'memory leak check')
@@ -138,5 +153,8 @@ class SubPlugin(TdcPlugin):
print('{}'.format(content))
self._add_to_tap(content)
+def _add_results(self, res):
+self._tsr.add_resultdata(res)
+
def _add_to_tap(self, more_tap_output):
self.tap += more_tap_output
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index b862ee5..e6e4ce8 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -23,6 +23,7 @@ from tdc_config import *
from tdc_helper import *
import TdcPlugin
+from TdcResults import *
class PluginMgrTestFail(Exception):
@@ -60,10 +61,10 @@ class PluginMgr:
for pgn_inst in reversed(self.plugin_instances):
pgn_inst.post_suite(index)
-def call_pre_case(self, test_ordinal, testid):
+def call_pre_case(self, test_ordinal, testid, test_name):
for pgn_inst in self.plugin_instances:
try:
-pgn_inst.pre_case(test_ordinal, testid)
+pgn_inst.pre_case(test_ordinal, testid, test_name)
except Exception as ee:
print('exception {} in call to pre_case for {} plugin'.
[PATCH net-next 2/4] tc-testing: Add new TdcResults module
This module includes new classes for tdc to use in keeping track
of test case results, instead of generating and tracking a lengthy
string.
The new module can be extended to support multiple formal
test result formats to be friendlier to automation.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/TdcResults.py | 133 +++
1 file changed, 133 insertions(+)
create mode 100644 tools/testing/selftests/tc-testing/TdcResults.py
diff --git a/tools/testing/selftests/tc-testing/TdcResults.py
b/tools/testing/selftests/tc-testing/TdcResults.py
new file mode 100644
index 000..250fbb2
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/TdcResults.py
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+
+from enum import Enum
+
+class ResultState(Enum):
+noresult = -1
+skip = 0
+success = 1
+fail = 2
+
+class TestResult:
+def __init__(self, test_id="", test_name=""):
+ self.test_id = test_id
+ self.test_name = test_name
+ self.result = ResultState.noresult
+ self.failmsg = ""
+ self.errormsg = ""
+ self.steps = []
+
+def set_result(self, result):
+if (isinstance(result, ResultState)):
+self.result = result
+return True
+else:
+raise TypeError('Unknown result type, must be type ResultState')
+
+def get_result(self):
+return self.result
+
+def set_errormsg(self, errormsg):
+self.errormsg = errormsg
+return True
+
+def append_errormsg(self, errormsg):
+self.errormsg = '{}\n{}'.format(self.errormsg, errormsg)
+
+def get_errormsg(self):
+return self.errormsg
+
+def set_failmsg(self, failmsg):
+self.failmsg = failmsg
+return True
+
+def append_failmsg(self, failmsg):
+self.failmsg = '{}\n{}'.format(self.failmsg, failmsg)
+
+def get_failmsg(self):
+return self.failmsg
+
+def add_steps(self, newstep):
+if type(newstep) == list:
+self.steps.extend(newstep)
+elif type(newstep) == str:
+self.steps.append(step)
+else:
+raise TypeError('TdcResults.add_steps() requires a list or str')
+
+def get_executed_steps(self):
+return self.steps
+
+class TestSuiteReport():
+_testsuite = []
+
+def add_resultdata(self, result_data):
+if isinstance(result_data, TestResult):
+self._testsuite.append(result_data)
+return True
+
+def count_tests(self):
+return len(self._testsuite)
+
+def count_failures(self):
+return sum(1 for t in self._testsuite if t.result == ResultState.fail)
+
+def count_skips(self):
+return sum(1 for t in self._testsuite if t.result == ResultState.skip)
+
+def find_result(self, test_id):
+return next((tr for tr in self._testsuite if tr.test_id == test_id),
None)
+
+def update_result(self, result_data):
+orig = self.find_result(result_data.test_id)
+if orig != None:
+idx = self._testsuite.index(orig)
+self._testsuite[idx] = result_data
+else:
+self.add_resultdata(result_data)
+
+def format_tap(self):
+ftap = ""
+ftap += '1..{}\n'.format(self.count_tests())
+index = 1
+for t in self._testsuite:
+if t.result == ResultState.fail:
+ftap += 'not '
+ftap += 'ok {} {} - {}'.format(str(index), t.test_id, t.test_name)
+if t.result == ResultState.skip or t.result ==
ResultState.noresult:
+ftap += ' # skipped - {}\n'.format(t.errormsg)
+elif t.result == ResultState.fail:
+if len(t.steps) > 0:
+ftap += '\tCommands executed in this test case:'
+for step in t.steps:
+ftap += '\n\t\t{}'.format(step)
+ftap += '\n\t{}'.format(t.failmsg)
+ftap += '\n'
+index += 1
+return ftap
+
+def format_xunit(self):
+from xml.sax.saxutils import escape
+xunit = "\n"
+xunit += '\t\n'.format(self.count_tests(), self.count_skips())
+for t in self._testsuite:
+xunit += '\t\t 0:
+xunit += 'Commands executed in this test case:\n'
+for step in t.steps:
+xunit += '\t{}\n'.format(escape(step))
+xunit += 'FAILURE: {}\n'.format(escape(t.failmsg))
+xunit += '\t\t\t\n'
+if t.errormsg:
+xunit += '\t\t\t\n{}\n'.format(escape(t.errormsg))
+xunit += '\t\t\t\n'
+if t.result == ResultState.skip:
+xunit += '\t\t\t\n'
+xunit += '\t\t\n'
+xunit += '\t\n'
+xunit += '\n'
+return xunit
+
--
2.7.4
[PATCH net-next 0/4] tc-testing: implement command timeouts and better results tracking
Patch 1 adds a timeout feature for any command tdc launches in a subshell. This prevents tdc from hanging indefinitely. Patches 2-4 introduce a new method for tracking and generating test case results, and implements it across the core script and all applicable plugins. Lucas Bates (4): tc-testing: Add command timeout feature to tdc tc-testing: Add new TdcResults module tc-testing: Implement the TdcResults module in tdc tc-testing: gitignore, ignore generated test results tools/testing/selftests/tc-testing/.gitignore | 3 + tools/testing/selftests/tc-testing/TdcPlugin.py| 3 +- tools/testing/selftests/tc-testing/TdcResults.py | 133 + .../tc-testing/plugin-lib/valgrindPlugin.py| 22 +++- tools/testing/selftests/tc-testing/tdc.py | 133 + tools/testing/selftests/tc-testing/tdc_config.py | 2 + 6 files changed, 240 insertions(+), 56 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/TdcResults.py -- 2.7.4
[PATCH net-next 4/4] tc-testing: gitignore, ignore generated test results
Ignore any .tap or .xml test result files generated by tdc. Additionally, ignore plugin symlinks. Signed-off-by: Lucas Bates --- tools/testing/selftests/tc-testing/.gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tools/testing/selftests/tc-testing/.gitignore b/tools/testing/selftests/tc-testing/.gitignore index 7a60b85..c5cc160 100644 --- a/tools/testing/selftests/tc-testing/.gitignore +++ b/tools/testing/selftests/tc-testing/.gitignore @@ -1,2 +1,5 @@ __pycache__/ *.pyc +plugins/ +*.xml +*.tap -- 2.7.4
[PATCH net 1/2] tc-testing: tdc.py: ignore errors when decoding stdout/stderr
Prevent exceptions from being raised while decoding output
from an executed command. There is no impact on tdc's
execution and the verify command phase would fail the pattern
match.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tdc.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 87a04a8..9b3f414 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -134,9 +134,9 @@ def exec_cmd(args, pm, stage, command):
(rawout, serr) = proc.communicate()
if proc.returncode != 0 and len(serr) > 0:
-foutput = serr.decode("utf-8")
+foutput = serr.decode("utf-8", errors="ignore")
else:
-foutput = rawout.decode("utf-8")
+foutput = rawout.decode("utf-8", errors="ignore")
proc.stdout.close()
proc.stderr.close()
--
2.7.4
[PATCH net 0/2] Prevent uncaught exceptions in tdc
This patch series addresses two potential bugs in tdc that can cause exceptions to be raised in certain circumstances. These exceptions are generally not handled, so instead we will prevent them from being raised. Brenda J. Butler (1): tc-testing: tdc.py: Guard against lack of returncode in executed command Lucas Bates (1): tc-testing: tdc.py: ignore errors when decoding stdout/stderr tools/testing/selftests/tc-testing/tdc.py | 18 +- 1 file changed, 13 insertions(+), 5 deletions(-) -- 2.7.4
[PATCH net 2/2] tc-testing: tdc.py: Guard against lack of returncode in executed command
From: "Brenda J. Butler"
Add some defensive coding in case one of the subprocesses created by tdc
returns nothing. If no object is returned from exec_cmd, then tdc will
halt with an unhandled exception.
Signed-off-by: Brenda J. Butler
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tdc.py | 14 +++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 9b3f414..7607ba3 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -169,6 +169,8 @@ def prepare_env(args, pm, stage, prefix, cmdlist, output =
None):
file=sys.stderr)
print("\n{} *** Error message: \"{}\"".format(prefix, foutput),
file=sys.stderr)
+print("returncode {}; expected {}".format(proc.returncode,
+ exit_codes))
print("\n{} *** Aborting test run.".format(prefix),
file=sys.stderr)
print("\n\n{} *** stdout ***".format(proc.stdout), file=sys.stderr)
print("\n\n{} *** stderr ***".format(proc.stderr), file=sys.stderr)
@@ -195,12 +197,18 @@ def run_one_test(pm, args, index, tidx):
print('-> execute stage')
pm.call_pre_execute()
(p, procout) = exec_cmd(args, pm, 'execute', tidx["cmdUnderTest"])
-exit_code = p.returncode
+if p:
+exit_code = p.returncode
+else:
+exit_code = None
+
pm.call_post_execute()
-if (exit_code != int(tidx["expExitCode"])):
+if (exit_code is None or exit_code != int(tidx["expExitCode"])):
result = False
-print("exit:", exit_code, int(tidx["expExitCode"]))
+print("exit: {!r}".format(exit_code))
+print("exit: {}".format(int(tidx["expExitCode"])))
+#print("exit: {!r} {}".format(exit_code, int(tidx["expExitCode"])))
print(procout)
else:
if args.verbose > 0:
--
2.7.4
Re: Some suggestions for tc-tests
Hi Cong, > > 1. Create veth pair devices by its own. The most important thing for > tc-tests is to automate everything, it is not friendly for users to > create their own veth pair named v0p0 to just run the tests. tc-tests > should be able to create a veth pair with random names and clean up > them once it is finished. You can actually do this automatically in two steps: first, create a symlink to plugin-lib/nsPlugin.py in the plugins/ directory. Then, when running tdc, always invoke the '-n' option. This will execute all the commands inside a namespace *and* automatically create the veth pair that get used in the testing. It's referenced in the readme, but if you think it's useful to make it a default setup I could add an installation/setup script to tdc to create the symlink. > 2. Test iproute2 version or capability. Apparently my iproute2 doesn't > support tc filter chain yet, this makes many tests failed. Ideally, > each test should be able to check if the iproute2 supports the thing > it wants to test, if not just skip it, at least by default. So is this a version you compile yourself, or is it just the default /sbin/tc? Because you can specify the tc executable you want to use in tdc_config.py... But yes, we're looking at ways to make sure the support is there before running tests. We're hoping to send some patches soon. > 3. Is there anything in the tests that can be done only with Python3? > If we could lower the requirement to Python2, then it would be easier > to setup and run these tests. I'd have to go back and re-check to see what python 3-specific features I'm using, but there *are* some. Do you maybe have the ability to run a VM or Docker container on your system to run python 3? Thanks!
Re: [PATCH net-next] tc-testing: add geneve options in tunnel_key unit tests
On Tue, Jul 10, 2018 at 9:22 PM, Jakub Kicinski wrote: > From: Pieter Jansen van Vuuren > > Extend tc tunnel_key action unit tests with geneve options. Tests > include testing single and multiple geneve options, as well as > testing geneve options that are expected to fail. > > Signed-off-by: Pieter Jansen van Vuuren Acked-by: Lucas Bates
Re: [PATCH net-next 1/1] tc-testing: initial version of tunnel_key unit tests
On Tue, Jun 26, 2018 at 10:51 AM, Davide Caratti wrote:
> On Tue, 2018-06-26 at 09:17 -0400, Keara Leibovitz wrote:
>> Create unittests for the tc tunnel_key action.
>>
>>
>> Signed-off-by: Keara Leibovitz
>> ---
>> .../tc-testing/tc-tests/actions/tunnel_key.json| 676
>> +
>> 1 file changed, 676 insertions(+)
>> create mode 100644
>> tools/testing/selftests/tc-testing/tc-tests/actions/tunnel_key.json
>>
>> diff --git
>> a/tools/testing/selftests/tc-testing/tc-tests/actions/tunnel_key.json
>> b/tools/testing/selftests/tc-testing/tc-tests/actions/tunnel_key.json
>> new file mode 100644
>> index ..bfe522ac8177
>
> hello Keara!
>
> I think the 'teardown' stage in some of these tests should be reviewed.
> Those that are meant to test invalid configurations (like dc6b) should
> allow non-zero exit codes in the teardown stage, if the wrong
> configuration is catched by the userspace TC tool, before talking to the
> kernel.
>
> Otherwise, those tests will fail when they are invoked one by one with the
> act_tunnel_key module unloaded.
>
Hi Davide, I thought I'd weigh in here.
In the short term, I think this is reasonable, but it's not a feasible
long-term solution. Here's why:
Allowing non-zero exit codes on setup and teardown was a precaution
that needed to be implemented as flushing actions in a freshly-booted
kernel returned errors - certain actions would only allow you to flush
after that action had been added.
But, doing this on so many test cases means that we can lose control
of the test environment, especially since a lot of commands get copied
between test cases. One test's command under test becomes the next
test case's setup command, etc. This can cause false results and
potentially waste a lot of time for someone trying to track down a
bug... Or cause bugs to be missed.
So, how to fix: we've had some discussions about it already. Jiri had
requested the addition of a config file (like the one at
tools/testing/selftests/net/forwarding/config, and maybe an addition
to the README for tdc for explanation. People would then possibly be
restricted to running one test case file at a time based on what
options they had loaded... This is still not ideal.
I think the best possible fix is to add a new plugin for tdc to
exclude tests based on the kernel config. This would require the
addition of a new optional field to the test case format, where any
and all included modules required for the test to work would be
listed. The plugin would look at this information, do its best to
determine if the currently running kernel supports it, and allows the
test to run or be skipped as a result.
Let me show an example of the new field:
>> --- /dev/null
>> +++ b/tools/testing/selftests/tc-testing/tc-tests/actions/tunnel_key.json
>> @@ -0,0 +1,676 @@
>>
> ...
>
>> +{
>> +"id": "dc6b",
>> +"name": "Add tunnel_key set action with missing mandatory src_ip
>> parameter",
>> +"category": [
>> +"actions",
>> +"tunnel_key"
>> +],
"reqModules": [
"CONFIG_NET_ACT_TUNNEL_KEY"
],
>> +"setup": [
>> +[
>> +"$TC actions flush action tunnel_key",
>> +0,
>> +1,
>> +255
>> +]
>> +],
>> +"cmdUnderTest": "$TC actions add action tunnel_key set dst_ip
>> 20.20.20.2 id 100",
>> +"expExitCode": "255",
>> +"verifyCmd": "$TC actions list action tunnel_key",
>> +"matchPattern": "action order [0-9]+: tunnel_key set.*dst_ip
>> 20.20.20.2.*key_id 100",
>> +"matchCount": "0",
>> +"teardown": [
>> +"$TC actions flush action tunnel_key"
>> +]
>> +},
As we venture into more and more complicated tests, where different
modules would start getting mixed together, this might be the most
effective route.
This plugin will require some changes I've made to our local version
of tdc that I've been testing out - they change the way tdc handles
its test results, and also give it the ability to skip tests without
affecting the rest of the test run.
Until I'm able to submit everything, I'd be OK with having Keara add
the non-zero exit codes to the teardown on her tests. In the meantime
we'll get the README updated and config file added as well.
How does this sound?
- Lucas
Re: [PATCH 00/14] Modify action API for implementing lockless actions
On Tue, May 15, 2018 at 6:03 PM, Lucas Bates wrote: > On Tue, May 15, 2018 at 5:49 PM, Jamal Hadi Salim wrote: >>> Test 7d50: Add skbmod action to set destination mac >>> exit: 255 0 >>> dst MAC address <11:22:33:44:55:66> >>> RTNETLINK answers: No such file or directory >>> We have an error talking to the kernel >>> >> >> You may actually have broken something with your patches in this case. >> >> Lucas - does this test pass on latest net-next? > > Yes, 7d50 has been passing on our builds for at least the last month. Also, Vlad, you can look at the JSON to see the test case data, or run tdc.py -s | less and search for the ID to see the commands being run. I'm here if you need help using tdc.
Re: [PATCH 00/14] Modify action API for implementing lockless actions
On Tue, May 15, 2018 at 5:49 PM, Jamal Hadi Salim wrote: >> Test 7d50: Add skbmod action to set destination mac >> exit: 255 0 >> dst MAC address <11:22:33:44:55:66> >> RTNETLINK answers: No such file or directory >> We have an error talking to the kernel >> > > You may actually have broken something with your patches in this case. > > Lucas - does this test pass on latest net-next? Yes, 7d50 has been passing on our builds for at least the last month. - Lucas
Re: [PATCH net] tc-testing: fix tdc tests for 'bpf' action
On Wed, May 9, 2018 at 12:45 PM, Davide Caratti wrote:
> - correct a typo in the value of 'matchPattern' of test 282d, potentially
> causing false negative
> - allow errors when 'teardown' executes '$TC action flush action bpf' in
> test 282d, to fix false positive when it is run with act_bpf unloaded
> - correct the value of 'matchPattern' in test e939, causing false positive
> in case the BPF JIT is enabled
>
> Fixes: 440ea4ae1828 ("tc-testing: add selftests for 'bpf' action")
> Signed-off-by: Davide Caratti
> ---
> tools/testing/selftests/tc-testing/tc-tests/actions/bpf.json | 11 ---
> 1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/bpf.json
> b/tools/testing/selftests/tc-testing/tc-tests/actions/bpf.json
> index 5b012f4981d4..6f289a49e5ec 100644
> --- a/tools/testing/selftests/tc-testing/tc-tests/actions/bpf.json
> +++ b/tools/testing/selftests/tc-testing/tc-tests/actions/bpf.json
> @@ -66,7 +66,7 @@
> "cmdUnderTest": "$TC action add action bpf object-file _b.o index
> 667",
> "expExitCode": "0",
> "verifyCmd": "$TC action get action bpf index 667",
> -"matchPattern": "action order [0-9]*: bpf _b.o:\\[action\\] id
> [0-9]* tag 3b185187f1855c4c default-action pipe.*index 667 ref",
> +"matchPattern": "action order [0-9]*: bpf _b.o:\\[action\\] id
> [0-9]* tag 3b185187f1855c4c( jited)? default-action pipe.*index 667 ref",
> "matchCount": "1",
> "teardown": [
> "$TC action flush action bpf",
> @@ -92,10 +92,15 @@
> "cmdUnderTest": "$TC action add action bpf object-file _c.o index
> 667",
> "expExitCode": "255",
> "verifyCmd": "$TC action get action bpf index 667",
> -"matchPattern": "action order [0-9]*: bpf _b.o:\\[action\\] id
> [0-9].*index 667 ref",
> +"matchPattern": "action order [0-9]*: bpf _c.o:\\[action\\] id
> [0-9].*index 667 ref",
> "matchCount": "0",
> "teardown": [
> -"$TC action flush action bpf",
> +[
> +"$TC action flush action bpf",
> +0,
> +1,
> +255
> +],
> "rm -f _c.o"
> ]
> },
> --
> 2.14.3
>
Acked-by: Lucas Bates
Re: [PATCH net] net/sched: fix NULL dereference in the error path of tcf_bpf_init()
On Thu, Apr 5, 2018 at 7:19 PM, Davide Caratti wrote:
> when tcf_bpf_init_from_ops() fails (e.g. because of program having invalid
> number of instructions), tcf_bpf_cfg_cleanup() calls bpf_prog_put(NULL) or
> bpf_prog_destroy(NULL). Unless CONFIG_BPF_SYSCALL is unset, this causes
> the following error:
>
> BUG: unable to handle kernel NULL pointer dereference at 0020
> PGD 80007345a067 P4D 80007345a067 PUD 340e1067 PMD 0
> Oops: [#1] SMP PTI
> Modules linked in: act_bpf(E) ip6table_filter ip6_tables iptable_filter
> binfmt_misc ext4 mbcache jbd2 crct10dif_pclmul crc32_pclmul
> ghash_clmulni_intel snd_hda_codec_generic pcbc snd_hda_intel snd_hda_codec
> snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm aesni_intel crypto_simd
> glue_helper cryptd joydev snd_timer snd virtio_balloon pcspkr soundcore
> i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c
> ata_generic pata_acpi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt
> fb_sys_fops ttm virtio_blk drm virtio_net virtio_console i2c_core
> crc32c_intel serio_raw virtio_pci ata_piix libata virtio_ring floppy virtio
> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: act_bpf]
> CPU: 3 PID: 5654 Comm: tc Tainted: GE4.16.0.bpf_test+ #408
> Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
> RIP: 0010:__bpf_prog_put+0xc/0xc0
> RSP: 0018:9594003ef728 EFLAGS: 00010202
> RAX: RBX: 9594003ef758 RCX: 0024
> RDX: RSI: 0001 RDI:
> RBP: R08: 0001 R09: 0044
> R10: 0220 R11: 8a7ab9f17131 R12:
> R13: 8a7ab7c3c8e0 R14: 0001 R15: 8a7ab88f1054
> FS: 7fcb2f17c740() GS:8a7abfd8() knlGS:
> CS: 0010 DS: ES: CR0: 80050033
> CR2: 0020 CR3: 7c888006 CR4: 001606e0
> Call Trace:
> tcf_bpf_cfg_cleanup+0x2f/0x40 [act_bpf]
> tcf_bpf_cleanup+0x4c/0x70 [act_bpf]
> __tcf_idr_release+0x79/0x140
> tcf_bpf_init+0x125/0x330 [act_bpf]
> tcf_action_init_1+0x2cc/0x430
> ? get_page_from_freelist+0x3f0/0x11b0
> tcf_action_init+0xd3/0x1b0
> tc_ctl_action+0x18b/0x240
> rtnetlink_rcv_msg+0x29c/0x310
> ? _cond_resched+0x15/0x30
> ? __kmalloc_node_track_caller+0x1b9/0x270
> ? rtnl_calcit.isra.29+0x100/0x100
> netlink_rcv_skb+0xd2/0x110
> netlink_unicast+0x17c/0x230
> netlink_sendmsg+0x2cd/0x3c0
> sock_sendmsg+0x30/0x40
> ___sys_sendmsg+0x27a/0x290
> ? mem_cgroup_commit_charge+0x80/0x130
> ? page_add_new_anon_rmap+0x73/0xc0
> ? do_anonymous_page+0x2a2/0x560
> ? __handle_mm_fault+0xc75/0xe20
> __sys_sendmsg+0x58/0xa0
> do_syscall_64+0x6e/0x1a0
> entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> RIP: 0033:0x7fcb2e58eba0
> RSP: 002b:7ffc93c496c8 EFLAGS: 0246 ORIG_RAX: 002e
> RAX: ffda RBX: 7ffc93c497f0 RCX: 7fcb2e58eba0
> RDX: RSI: 7ffc93c49740 RDI: 0003
> RBP: 5ac6a646 R08: 0002 R09:
> R10: 7ffc93c49120 R11: 0246 R12:
> R13: 7ffc93c49804 R14: 0001 R15: 0066afa0
> Code: 5f 00 48 8b 43 20 48 c7 c7 70 2f 7c b8 c7 40 10 00 00 00 00 5b e9 a5
> 8b 61 00 0f 1f 44 00 00 0f 1f 44 00 00 41 54 55 48 89 fd 53 <48> 8b 47 20 f0
> ff 08 74 05 5b 5d 41 5c c3 41 89 f4 0f 1f 44 00
> RIP: __bpf_prog_put+0xc/0xc0 RSP: 9594003ef728
> CR2: 0020
>
> Fix it in tcf_bpf_cfg_cleanup(), ensuring that bpf_prog_{put,destroy}(f)
> is called only when f is not NULL.
>
> Fixes: bbc09e7842a5 ("net/sched: fix idr leak on the error path of
> tcf_bpf_init()")
> Reported-by: Lucas Bates
> Signed-off-by: Davide Caratti
That does the trick, thanks Davide.
Tested-by: Lucas Bates
Kernel bug from adding bpf actions in tc
Hi Davide, Our overnight tc test runs of net-next revealed a kernel bug on one of the BPF tests you submitted, d959. The add action completes successfully, but the bug occurs on the verify when tdc does a get of the action that was just added. Here's the text of the dump: [ 61.973632] BUG: unable to handle kernel NULL pointer dereference at 0020 [ 61.974366] PGD 800081881067 P4D 800081881067 PUD 83784067 PMD 0 [ 61.974986] Oops: [#1] SMP PTI [ 61.975309] Modules linked in: kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd psmouse glue_helper cryptd serio_raw [ 61.976800] CPU: 28 PID: 1087 Comm: tc Not tainted 4.16.0+ #28 [ 61.977329] RIP: 0010:__bpf_prog_put+0x5/0xe0 [ 61.977731] RSP: 0018:9647c4823788 EFLAGS: 00010202 [ 61.978204] RAX: RBX: 9647c48237a0 RCX: 176c [ 61.978845] RDX: RSI: 0001 RDI: [ 61.979484] RBP: R08: 00025be0 R09: a4794077 [ 61.980121] R10: dc1bc2130b00 R11: R12: [ 61.980763] R13: 0001 R14: 8889869969f0 R15: ffea [ 61.981398] FS: 7faa72489700() GS:888988f0() knlGS: [ 61.982114] CS: 0010 DS: ES: CR0: 80050033 [ 61.982627] CR2: 0020 CR3: 85938004 CR4: 003606a0 [ 61.983263] DR0: DR1: DR2: [ 61.983897] DR3: DR6: fffe0ff0 DR7: 0400 [ 61.984531] Call Trace: [ 61.984766] tcf_bpf_cfg_cleanup+0x2f/0x40 [ 61.985139] tcf_bpf_cleanup+0x3c/0x50 [ 61.985479] ? uncore_event_cpu_online+0x80/0x3c0 [ 61.985922] __tcf_idr_release+0x72/0x150 [ 61.986297] tcf_bpf_init+0x102/0x3e0 [ 61.986637] ? perf_trace_sched_process_exec+0xf4/0x140 [ 61.987108] tcf_action_init_1+0x36c/0x410 [ 61.987482] ? ___slab_alloc+0x218/0x4b0 [ 61.987841] tcf_action_init+0x106/0x190 [ 61.988204] tc_ctl_action+0x11a/0x220 [ 61.988551] rtnetlink_rcv_msg+0x243/0x2f0 [ 61.988931] ? _cond_resched+0x16/0x40 [ 61.989277] ? __kmalloc_node_track_caller+0x1e6/0x2a0 [ 61.989746] ? rtnl_calcit.isra.29+0xe0/0xe0 [ 61.990137] netlink_rcv_skb+0xde/0x110 [ 61.990494] netlink_unicast+0x16d/0x220 [ 61.990858] netlink_sendmsg+0x293/0x370 [ 61.991224] sock_sendmsg+0x36/0x40 [ 61.991559] ___sys_sendmsg+0x2cb/0x2e0 [ 61.991913] ? pagecache_get_page+0x27/0x220 [ 61.992302] ? filemap_fault+0xa2/0x650 [ 61.992651] ? page_add_file_rmap+0x108/0x200 [ 61.993057] ? alloc_set_pte+0x2aa/0x530 [ 61.993419] ? finish_fault+0x4e/0x70 [ 61.993758] ? __handle_mm_fault+0xbfc/0x1110 [ 61.994160] ? __sys_sendmsg+0x53/0x80 [ 61.994506] __sys_sendmsg+0x53/0x80 [ 61.994849] do_syscall_64+0x6e/0x120 [ 61.995189] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 61.995662] RIP: 0033:0x7faa71885bd0 [ 61.995991] RSP: 002b:7ffccf65bf28 EFLAGS: 0246 ORIG_RAX: 002e [ 61.996693] RAX: ffda RBX: 7ffccf65c050 RCX: 7faa71885bd0 [ 61.997350] RDX: RSI: 7ffccf65bfa0 RDI: 0003 [ 61.998001] RBP: 5ac513e0 R08: 0001 R09: [ 61.998649] R10: 05e7 R11: 0246 R12: [ 61.999287] R13: 7ffccf6600b0 R14: 0001 R15: 00674600 [ 61.42] Code: c6 72 00 48 8b 43 20 48 c7 c7 d0 61 9b a5 c7 40 10 00 00 00 00 5b e9 1b d9 74 00 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <48> 8b 47 20 f0 ff 08 74 01 c3 41 54 41 89 f4 55 48 89 fd 53 66 [ 62.001660] RIP: __bpf_prog_put+0x5/0xe0 RSP: 9647c4823788 [ 62.002183] CR2: 0020 [ 62.002488] ---[ end trace 19b56d1a66dd8e2a ]--- I'm sending this your way because you were the last one to touch this part of the code. Have you seen this in your own testing? (This can't be replicated by hand, only by running tdc). - Lucas
[PATCH net-next] tc-testing: Add newline when writing test case files
When using the -i feature to generate random ID numbers for test
cases in tdc, the function that writes the JSON to file doesn't
add a newline character to the end of the file, so we have to
add our own.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tdc.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index 44de4a2..87a04a8 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -490,6 +490,7 @@ def generate_case_ids(alltests):
testlist.append(t)
outfile = open(f, "w")
json.dump(testlist, outfile, indent=4)
+outfile.write("\n")
outfile.close()
def filter_tests_by_id(args, testlist):
--
2.7.4
[PATCH net-next 1/1] tc-testing: Correct compound statements for namespace execution
If tdc is executing test cases inside a namespace, only the
first command in a compound statement will be executed inside
the namespace by tdc. As a result, the subsequent commands
are not executed inside the namespace and the test will fail.
Example:
for i in {x..y}; do args="foo"; done && tc actions add $args
The namespace execution feature will prepend 'ip netns exec'
to the command:
ip netns exec tcut for i in {x..y}; do args="foo"; done && \
tc actions add $args
So the actual tc command is not parsed by the shell as being
part of the namespace execution.
Enclosing these compound statements inside a bash invocation
with proper escape characters resolves the problem by creating
a subshell inside the namespace.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tc-tests/actions/gact.json | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
index ae96d03..68c9102 100644
--- a/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
@@ -481,7 +481,7 @@
255
]
],
-"cmdUnderTest": "for i in `seq 1 32`; do cmd=\"action pass index $i
\"; args=\"$args$cmd\"; done && $TC actions add $args",
+"cmdUnderTest": "bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action
pass index \\$i \\\"; args=\"\\$args\\$cmd\"; done && $TC actions add
\\$args\"",
"expExitCode": "0",
"verifyCmd": "$TC actions list action gact",
"matchPattern": "^[ \t]+index [0-9]+ ref",
@@ -505,7 +505,7 @@
255
]
],
-"cmdUnderTest": "for i in `seq 1 32`; do cmd=\"action continue index
$i cookie aabbccddeeff112233445566778800a1 \"; args=\"$args$cmd\"; done && $TC
actions add $args",
+"cmdUnderTest": "bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action
continue index \\$i cookie aabbccddeeff112233445566778800a1 \\\";
args=\"\\$args\\$cmd\"; done && $TC actions add \\$args\"",
"expExitCode": "0",
"verifyCmd": "$TC actions list action gact",
"matchPattern": "^[ \t]+index [0-9]+ ref",
@@ -528,13 +528,13 @@
1,
255
],
-"for i in `seq 1 32`; do cmd=\"action continue index $i \";
args=\"$args$cmd\"; done && $TC actions add $args"
+"bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action continue
index \\$i \\\"; args=\\\"\\$args\\$cmd\\\"; done && $TC actions add \\$args\""
],
-"cmdUnderTest": "for i in `seq 1 32`; do cmd=\"action gact index $i
\"; args=\"$args$cmd\"; done && $TC actions del $args",
+"cmdUnderTest": "bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action
gact index \\$i \\\"; args=\"\\$args\\$cmd\"; done && $TC actions del
\\$args\"",
"expExitCode": "0",
"verifyCmd": "$TC actions list action gact",
"matchPattern": "^[ \t]+index [0-9]+ ref",
"matchCount": "0",
"teardown": []
}
-]
\ No newline at end of file
+]
--
2.7.4
Re: [PATCH net-next 1/1] tc-testing: Correct compound statements for namespace execution
On Thu, Mar 22, 2018 at 2:48 PM, David Miller wrote: > From: Lucas Bates > Date: Wed, 21 Mar 2018 11:49:40 -0400 > >> } >> -] >> \ No newline at end of file >> +] >> -- >> 2.7.4 > > Please fix this. This patch fixes the gact.json file that had no newline on the end.
[PATCH net-next 1/1] tc-testing: Correct compound statements for namespace execution
If tdc is executing test cases inside a namespace, only the
first command in a compound statement will be executed inside
the namespace by tdc. As a result, the subsequent commands
are not executed inside the namespace and the test will fail.
Example:
for i in {x..y}; do args="foo"; done && tc actions add $args
The namespace execution feature will prepend 'ip netns exec'
to the command:
ip netns exec tcut for i in {x..y}; do args="foo"; done && \
tc actions add $args
So the actual tc command is not parsed by the shell as being
part of the namespace execution.
Enclosing these compound statements inside a bash invocation
with proper escape characters resolves the problem by creating
a subshell inside the namespace.
Signed-off-by: Lucas Bates
---
tools/testing/selftests/tc-testing/tc-tests/actions/gact.json | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
index ae96d03..68c9102 100644
--- a/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
@@ -481,7 +481,7 @@
255
]
],
-"cmdUnderTest": "for i in `seq 1 32`; do cmd=\"action pass index $i
\"; args=\"$args$cmd\"; done && $TC actions add $args",
+"cmdUnderTest": "bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action
pass index \\$i \\\"; args=\"\\$args\\$cmd\"; done && $TC actions add
\\$args\"",
"expExitCode": "0",
"verifyCmd": "$TC actions list action gact",
"matchPattern": "^[ \t]+index [0-9]+ ref",
@@ -505,7 +505,7 @@
255
]
],
-"cmdUnderTest": "for i in `seq 1 32`; do cmd=\"action continue index
$i cookie aabbccddeeff112233445566778800a1 \"; args=\"$args$cmd\"; done && $TC
actions add $args",
+"cmdUnderTest": "bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action
continue index \\$i cookie aabbccddeeff112233445566778800a1 \\\";
args=\"\\$args\\$cmd\"; done && $TC actions add \\$args\"",
"expExitCode": "0",
"verifyCmd": "$TC actions list action gact",
"matchPattern": "^[ \t]+index [0-9]+ ref",
@@ -528,13 +528,13 @@
1,
255
],
-"for i in `seq 1 32`; do cmd=\"action continue index $i \";
args=\"$args$cmd\"; done && $TC actions add $args"
+"bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action continue
index \\$i \\\"; args=\\\"\\$args\\$cmd\\\"; done && $TC actions add \\$args\""
],
-"cmdUnderTest": "for i in `seq 1 32`; do cmd=\"action gact index $i
\"; args=\"$args$cmd\"; done && $TC actions del $args",
+"cmdUnderTest": "bash -c \"for i in \\`seq 1 32\\`; do cmd=\\\"action
gact index \\$i \\\"; args=\"\\$args\\$cmd\"; done && $TC actions del
\\$args\"",
"expExitCode": "0",
"verifyCmd": "$TC actions list action gact",
"matchPattern": "^[ \t]+index [0-9]+ ref",
"matchCount": "0",
"teardown": []
}
-]
\ No newline at end of file
+]
--
2.7.4
Re: [PATCH net-next 7/7] tools: tc-testing: Update README and TODO
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote: > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates > --- > tools/testing/selftests/tc-testing/README | 173 > +--- > tools/testing/selftests/tc-testing/TODO.txt | 25 +++- > 2 files changed, 179 insertions(+), 19 deletions(-) > > diff --git a/tools/testing/selftests/tc-testing/README > b/tools/testing/selftests/tc-testing/README > index 970ff294fec8..3a0336782d2d 100644 > --- a/tools/testing/selftests/tc-testing/README > +++ b/tools/testing/selftests/tc-testing/README > @@ -14,11 +14,11 @@ REQUIREMENTS > > * The kernel must have network namespace support > > -* The kernel must have veth support available, as a veth pair is created > +* The kernel must have veth support available, as a veth pair is created > prior to running the tests. > > -* All tc-related features must be built in or available as modules. > - To check what is required in current setup run: > +* All tc-related features being tested must be built in or available as > + modules. To check what is required in current setup run: > ./tdc.py -c > > Note: > @@ -44,10 +44,13 @@ using the -p option when running tdc: > RUNNING TDC > --- > > -To use tdc, root privileges are required. tdc will not run otherwise. > +To use tdc, root privileges are required. This is because the > +commands being tested must be run as root. The code that enforces > +execution by root uid has been moved into a plugin (see PLUGIN > +ARCHITECTURE, below). > > -All tests are executed inside a network namespace to prevent conflicts > -within the host. > +If nsPlugin is linked, all tests are executed inside a network > +namespace to prevent conflicts within the host. > > Running tdc without any arguments will run all tests. Refer to the section > on command line arguments for more information, or run: > @@ -59,6 +62,33 @@ output captured from the failing test will be printed > immediately following > the failed test in the TAP output. > > > +OVERVIEW OF TDC EXECUTION > +- > + > +One run of tests is considered a "test suite" (this will be refined in the > +future). A test suite has one or more test cases in it. > + > +A test case has four stages: > + > + - setup > + - execute > + - verify > + - teardown > + > +The setup and teardown stages can run zero or more commands. The setup > +stage does some setup if the test needs it. The teardown stage undoes > +the setup and returns the system to a "neutral" state so any other test > +can be run next. These two stages require any commands run to return > +success, but do not otherwise verify the results. > + > +The execute and verify stages each run one command. The execute stage > +tests the return code against one or more acceptable values. The > +verify stage checks the return code for success, and also compares > +the stdout with a regular expression. > + > +Each of the commands in any stage will run in a shell instance. > + > + > USER-DEFINED CONSTANTS > -- > > @@ -70,23 +100,132 @@ executed as part of the test. More will be added as > test cases require. > Example: > $TC qdisc add dev $DEV1 ingress > > +The NAMES values are used to substitute into the commands in the test cases. > + > > COMMAND LINE ARGUMENTS > -- > > Run tdc.py -h to see the full list of available arguments. > > --p PATH Specify the tc executable located at PATH to be used on > this > - test run > --cShow the available test case categories in this test file > --c CATEGORY Run only tests that belong to CATEGORY > --f FILE Read test cases from the JSON file named FILE > --l [CATEGORY] List all test cases in the JSON file. If CATEGORY is > - specified, list test cases matching that category. > --s ID Show the test case matching ID > --e ID Execute the test case identified by ID > --iGenerate unique ID numbers for test cases with no existing > - ID number > +usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]] > + [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] > + [-d DEVICE] [-n NS] [-V] > + > +Linux TC unit tests > + > +optional arguments: > + -h, --helpshow this help message and exit > + -p PATH, --path PATH The full path to the tc executable to use > + -v, --verbose Show the commands that are being run > + -d DEVICE, --device DEVICE > +Exec
Re: [PATCH net-next 6/7] tools: tc-testing: valgrindPlugin
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote:
> Run the command under test under valgrind. Produce an extra set of
> tap output for the memory check on each test.
>
> Signed-off-by: Brenda J. Butler
Acked-by: Lucas Bates
> ---
> .../tc-testing/plugin-lib/valgrindPlugin.py| 142
> +
> 1 file changed, 142 insertions(+)
> create mode 100644
> tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
>
> diff --git a/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
> b/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
> new file mode 100644
> index ..477a7bd7d7fb
> --- /dev/null
> +++ b/tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py
> @@ -0,0 +1,142 @@
> +'''
> +run the command under test, under valgrind and collect memory leak info
> +as a separate test.
> +'''
> +
> +
> +import os
> +import re
> +import signal
> +from string import Template
> +import subprocess
> +import time
> +from TdcPlugin import TdcPlugin
> +
> +from tdc_config import *
> +
> +def vp_extract_num_from_string(num_as_string_maybe_with_commas):
> +return int(num_as_string_maybe_with_commas.replace(',',''))
> +
> +class SubPlugin(TdcPlugin):
> +def __init__(self):
> +self.sub_class = 'valgrind/SubPlugin'
> +self.tap = ''
> +super().__init__()
> +
> +def pre_suite(self, testcount, testidlist):
> +'''run commands before test_runner goes into a test loop'''
> +super().pre_suite(testcount, testidlist)
> +if self.args.verbose > 1:
> +print('{}.pre_suite'.format(self.sub_class))
> +if self.args.valgrind:
> +self._add_to_tap('1..{}\n'.format(self.testcount))
> +
> +def post_suite(self, index):
> +'''run commands after test_runner goes into a test loop'''
> +super().post_suite(index)
> +self._add_to_tap('\n|---\n')
> +if self.args.verbose > 1:
> +print('{}.post_suite'.format(self.sub_class))
> +print('{}'.format(self.tap))
> +if self.args.verbose < 4:
> +subprocess.check_output('rm -f vgnd-*.log', shell=True)
> +
> +def add_args(self, parser):
> +super().add_args(parser)
> +self.argparser_group = self.argparser.add_argument_group(
> +'valgrind',
> +'options for valgrindPlugin (run command under test under
> Valgrind)')
> +
> +self.argparser_group.add_argument(
> +'-V', '--valgrind', action='store_true',
> +help='Run commands under valgrind')
> +
> +return self.argparser
> +
> +def adjust_command(self, stage, command):
> +super().adjust_command(stage, command)
> +cmdform = 'list'
> +cmdlist = list()
> +
> +if not self.args.valgrind:
> +return command
> +
> +if self.args.verbose > 1:
> +print('{}.adjust_command'.format(self.sub_class))
> +
> +if not isinstance(command, list):
> +cmdform = 'str'
> +cmdlist = command.split()
> +else:
> +cmdlist = command
> +
> +if stage == 'execute':
> +if self.args.verbose > 1:
> +print('adjust_command: stage is {}; inserting valgrind
> stuff in command [{}] list [{}]'.
> + format(stage, command, cmdlist))
> +cmdlist.insert(0, '--track-origins=yes')
> +cmdlist.insert(0, '--show-leak-kinds=definite,indirect')
> +cmdlist.insert(0, '--leak-check=full')
> +cmdlist.insert(0,
> '--log-file=vgnd-{}.log'.format(self.args.testid))
> +cmdlist.insert(0, '-v') # ask for summary of non-leak errors
> +cmdlist.insert(0, ENVIR['VALGRIND_BIN'])
> +else:
> +pass
> +
> +if cmdform == 'str':
> +command = ' '.join(cmdlist)
> +else:
> +command = cmdlist
> +
> +if self.args.verbose > 1:
> +print('adjust_command: return command [{}]'.format(command))
> +return command
> +
> +def post_execute(self):
> +if not self.args.valgrind:
> +return
> +
> +self.definitely_l
Re: [PATCH net-next 5/7] tools: tc-testing: nsPlugin
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote:
> Move the functionality of creating a namespace before the test suite
> and destroying it afterwards to a plugin.
>
> Signed-off-by: Brenda J. Butler
Acked-by: Lucas Bates
> ---
> .../selftests/tc-testing/plugin-lib/nsPlugin.py| 141
> +
> tools/testing/selftests/tc-testing/tdc.py | 45 +--
> 2 files changed, 142 insertions(+), 44 deletions(-)
> create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py
>
> diff --git a/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py
> b/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py
> new file mode 100644
> index ..a194b1af2b30
> --- /dev/null
> +++ b/tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py
> @@ -0,0 +1,141 @@
> +import os
> +import signal
> +from string import Template
> +import subprocess
> +import time
> +from TdcPlugin import TdcPlugin
> +
> +from tdc_config import *
> +
> +class SubPlugin(TdcPlugin):
> +def __init__(self):
> +self.sub_class = 'ns/SubPlugin'
> +super().__init__()
> +
> +def pre_suite(self, testcount, testidlist):
> +'''run commands before test_runner goes into a test loop'''
> +super().pre_suite(testcount, testidlist)
> +
> +if self.args.namespace:
> +self._ns_create()
> +
> +def post_suite(self, index):
> +'''run commands after test_runner goes into a test loop'''
> +super().post_suite(index)
> +if self.args.verbose:
> +print('{}.post_suite'.format(self.sub_class))
> +
> +if self.args.namespace:
> +self._ns_destroy()
> +
> +def add_args(self, parser):
> +super().add_args(parser)
> +self.argparser_group = self.argparser.add_argument_group(
> +'netns',
> +'options for nsPlugin(run commands in net namespace)')
> +self.argparser_group.add_argument(
> +'-n', '--namespace', action='store_true',
> +help='Run commands in namespace')
> +return self.argparser
> +
> +def adjust_command(self, stage, command):
> +super().adjust_command(stage, command)
> +cmdform = 'list'
> +cmdlist = list()
> +
> +if not self.args.namespace:
> +return command
> +
> +if self.args.verbose:
> +print('{}.adjust_command'.format(self.sub_class))
> +
> +if not isinstance(command, list):
> +cmdform = 'str'
> +cmdlist = command.split()
> +else:
> +cmdlist = command
> +if stage == 'setup' or stage == 'execute' or stage == 'verify' or
> stage == 'teardown':
> +if self.args.verbose:
> +print('adjust_command: stage is {}; inserting netns stuff
> in command [{}] list [{}]'.format(stage, command, cmdlist))
> +cmdlist.insert(0, self.args.NAMES['NS'])
> +cmdlist.insert(0, 'exec')
> +cmdlist.insert(0, 'netns')
> +cmdlist.insert(0, 'ip')
> +else:
> +pass
> +
> +if cmdform == 'str':
> +command = ' '.join(cmdlist)
> +else:
> +command = cmdlist
> +
> +if self.args.verbose:
> +print('adjust_command: return command [{}]'.format(command))
> +return command
> +
> +def _ns_create(self):
> +'''
> +Create the network namespace in which the tests will be run and set
> up
> +the required network devices for it.
> +'''
> +if self.args.namespace:
> +cmd = 'ip netns add {}'.format(self.args.NAMES['NS'])
> +self._exec_cmd('pre', cmd)
> +cmd = 'ip link add $DEV0 type veth peer name $DEV1'
> +self._exec_cmd('pre', cmd)
> +cmd = 'ip link set $DEV1 netns {}'.format(self.args.NAMES['NS'])
> +self._exec_cmd('pre', cmd)
> +cmd = 'ip link set $DEV0 up'
> +self._exec_cmd('pre', cmd)
> +cmd = 'ip -n {} link set $DEV1 up'.format(self.args.NAMES['NS'])
> +self._exec_cmd('pre', cmd)
> +if self.args.devi
Re: [PATCH net-next 3/7] tools: tc-testing: Introduce plugin architecture
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote:
> This should be a general test architecture, and yet allow specific
> tests to be done. Introduce a plugin architecture.
>
> An individual test has 4 stages, setup/execute/verify/teardown. Each
> plugin gets a chance to run a function at each stage, plus one call
> before all the tests are called ("pre" suite) and one after all the
> tests are called ("post" suite). In addition, just before each
> command is executed, the plugin gets a chance to modify the command
> using the "adjust_command" hook. This makes the test suite quite
> flexible.
>
> Future patches will take some functionality out of the tdc.py script and
> place it in plugins.
>
> To use the plugins, place the implementation in the plugins directory
> and run tdc.py. It will notice the plugins and use them.
>
> Signed-off-by: Brenda J. Butler
Acked-by: Lucas Bates
> ---
> tools/testing/selftests/tc-testing/TdcPlugin.py| 74 +++
> .../tc-testing/creating-plugins/AddingPlugins.txt | 104 ++
> .../selftests/tc-testing/plugin-lib/README-PLUGINS | 27 +++
> .../selftests/tc-testing/plugins/__init__.py | 0
> tools/testing/selftests/tc-testing/tdc.py | 221
> +++--
> 5 files changed, 368 insertions(+), 58 deletions(-)
> create mode 100644 tools/testing/selftests/tc-testing/TdcPlugin.py
> create mode 100644
> tools/testing/selftests/tc-testing/creating-plugins/AddingPlugins.txt
> create mode 100644
> tools/testing/selftests/tc-testing/plugin-lib/README-PLUGINS
> create mode 100644 tools/testing/selftests/tc-testing/plugins/__init__.py
>
> diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py
> b/tools/testing/selftests/tc-testing/TdcPlugin.py
> new file mode 100644
> index ..3ee9a6dacb52
> --- /dev/null
> +++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
> @@ -0,0 +1,74 @@
> +#!/usr/bin/env python3
> +
> +class TdcPlugin:
> +def __init__(self):
> +super().__init__()
> +print(' -- {}.__init__'.format(self.sub_class))
> +
> +def pre_suite(self, testcount, testidlist):
> +'''run commands before test_runner goes into a test loop'''
> +self.testcount = testcount
> +self.testidlist = testidlist
> +if self.args.verbose > 1:
> +print(' -- {}.pre_suite'.format(self.sub_class))
> +
> +def post_suite(self, index):
> +'''run commands after test_runner completes the test loop
> +index is the last ordinal number of test that was attempted'''
> +if self.args.verbose > 1:
> +print(' -- {}.post_suite'.format(self.sub_class))
> +
> +def pre_case(self, test_ordinal, testid):
> +'''run commands before test_runner does one test'''
> +if self.args.verbose > 1:
> +print(' -- {}.pre_case'.format(self.sub_class))
> +self.args.testid = testid
> +self.args.test_ordinal = test_ordinal
> +
> +def post_case(self):
> +'''run commands after test_runner does one test'''
> +if self.args.verbose > 1:
> +print(' -- {}.post_case'.format(self.sub_class))
> +
> +def pre_execute(self):
> +'''run command before test-runner does the execute step'''
> +if self.args.verbose > 1:
> +print(' -- {}.pre_execute'.format(self.sub_class))
> +
> +def post_execute(self):
> +'''run command after test-runner does the execute step'''
> +if self.args.verbose > 1:
> +print(' -- {}.post_execute'.format(self.sub_class))
> +
> +def adjust_command(self, stage, command):
> +'''adjust the command'''
> +if self.args.verbose > 1:
> +print(' -- {}.adjust_command {}'.format(self.sub_class, stage))
> +
> +# if stage == 'pre':
> +# pass
> +# elif stage == 'setup':
> +# pass
> +# elif stage == 'execute':
> +# pass
> +# elif stage == 'verify':
> +# pass
> +# elif stage == 'teardown':
> +# pass
> +# elif stage == 'post':
> +# pass
> +# else:
> +# pass
> +
> +return command
> +
> +def add_args(self, parser):
> +'''Get
Re: [PATCH net-next 4/7] tools: tc-testing: rootPlugin
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote:
> Move the functionality that checks for root permissions into a plugin.
>
> Signed-off-by: Brenda J. Butler
Acked-by: Lucas Bates
> ---
> .../selftests/tc-testing/plugin-lib/rootPlugin.py | 19
> +++
> tools/testing/selftests/tc-testing/tdc.py | 4
> 2 files changed, 19 insertions(+), 4 deletions(-)
> create mode 100644
> tools/testing/selftests/tc-testing/plugin-lib/rootPlugin.py
>
> diff --git a/tools/testing/selftests/tc-testing/plugin-lib/rootPlugin.py
> b/tools/testing/selftests/tc-testing/plugin-lib/rootPlugin.py
> new file mode 100644
> index ..e36775bd4d12
> --- /dev/null
> +++ b/tools/testing/selftests/tc-testing/plugin-lib/rootPlugin.py
> @@ -0,0 +1,19 @@
> +import os
> +import sys
> +from TdcPlugin import TdcPlugin
> +
> +from tdc_config import *
> +
> +
> +class SubPlugin(TdcPlugin):
> +def __init__(self):
> +self.sub_class = 'root/SubPlugin'
> +super().__init__()
> +
> +def pre_suite(self, testcount, testidlist):
> +# run commands before test_runner goes into a test loop
> +super().pre_suite(testcount, testidlist)
> +
> +if os.geteuid():
> +print('This script must be run with root privileges',
> file=sys.stderr)
> +exit(1)
> diff --git a/tools/testing/selftests/tc-testing/tdc.py
> b/tools/testing/selftests/tc-testing/tdc.py
> index 3e6f9f2e1691..a718d2b57739 100755
> --- a/tools/testing/selftests/tc-testing/tdc.py
> +++ b/tools/testing/selftests/tc-testing/tdc.py
> @@ -579,10 +579,6 @@ def set_operation_mode(pm, args):
> list_test_cases(alltests)
> exit(0)
>
> -if (os.geteuid() != 0):
> -print("This script must be run with root privileges.\n")
> -exit(1)
> -
> ns_create(args, pm)
>
> if len(alltests):
> --
> 2.15.1
>
Re: [PATCH net-next 2/7] tools: tc-testing: Refactor test-runner
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote:
> Split the test_runner function into the loop part (test_runner)
> and the contents (run_one_test) for maintainability.
> It makes it a little easier to catch exceptions
> in an individual test, and keep going (and flush a bunch
> of tap results for the skipped tests).
>
> Signed-off-by: Brenda J. Butler
Acked-by: Lucas Bates
> ---
> tools/testing/selftests/tc-testing/tdc.py | 81
> ---
> 1 file changed, 52 insertions(+), 29 deletions(-)
>
> diff --git a/tools/testing/selftests/tc-testing/tdc.py
> b/tools/testing/selftests/tc-testing/tdc.py
> index ef3a8881e458..a2624eda34db 100755
> --- a/tools/testing/selftests/tc-testing/tdc.py
> +++ b/tools/testing/selftests/tc-testing/tdc.py
> @@ -85,9 +85,42 @@ def prepare_env(cmdlist):
> print("\nError message:")
> print(foutput)
> print("\nAborting test run.")
> -ns_destroy()
> -exit(1)
> +# ns_destroy()
> +raise Exception('prepare_env did not complete successfully')
> +
> +def run_one_test(index, tidx):
> +result = True
> +tresult = ""
> +tap = ""
> +print("Test " + tidx["id"] + ": " + tidx["name"])
> +prepare_env(tidx["setup"])
> +(p, procout) = exec_cmd(tidx["cmdUnderTest"])
> +exit_code = p.returncode
> +
> +if (exit_code != int(tidx["expExitCode"])):
> +result = False
> +print("exit:", exit_code, int(tidx["expExitCode"]))
> +print(procout)
> +else:
> +match_pattern = re.compile(str(tidx["matchPattern"]),
> + re.DOTALL | re.MULTILINE)
> +(p, procout) = exec_cmd(tidx["verifyCmd"])
> +match_index = re.findall(match_pattern, procout)
> +if len(match_index) != int(tidx["matchCount"]):
> +result = False
> +
> +if not result:
> +tresult += "not "
> +tresult += "ok {} - {} # {}\n".format(str(index), tidx['id'],
> tidx["name"])
> +tap += tresult
>
> +if result == False:
> +tap += procout
> +
> +prepare_env(tidx["teardown"])
> +index += 1
> +
> +return tap
>
> def test_runner(filtered_tests, args):
> """
> @@ -104,37 +137,27 @@ def test_runner(filtered_tests, args):
> tap = str(index) + ".." + str(tcount) + "\n"
>
> for tidx in testlist:
> -result = True
> -tresult = ""
> if "flower" in tidx["category"] and args.device == None:
> continue
> -print("Test " + tidx["id"] + ": " + tidx["name"])
> -prepare_env(tidx["setup"])
> -(p, procout) = exec_cmd(tidx["cmdUnderTest"])
> -exit_code = p.returncode
> -
> -if (exit_code != int(tidx["expExitCode"])):
> -result = False
> -print("exit:", exit_code, int(tidx["expExitCode"]))
> -print(procout)
> -else:
> -match_pattern = re.compile(str(tidx["matchPattern"]), re.DOTALL)
> -(p, procout) = exec_cmd(tidx["verifyCmd"])
> -match_index = re.findall(match_pattern, procout)
> -if len(match_index) != int(tidx["matchCount"]):
> -result = False
> -
> -if result == True:
> -tresult += "ok "
> -else:
> -tresult += "not ok "
> -tap += tresult + str(index) + " " + tidx["id"] + " " + tidx["name"]
> + "\n"
> +try:
> +badtest = tidx # in case it goes bad
> +tap += run_one_test(index, tidx)
> +except Exception as ee:
> +print('Exception {} (caught in test_runner, running test {} {}
> {})'.
> + format(ee, index, tidx['id'], tidx['name']))
> +break
> +index += 1
>
> -if result == False:
> -tap += procout
> +count = index
> +tap += 'about to flush the tap output if tests need to be skipped\n'
> +if tcount + 1 != index:
> +for tidx in testlist[index - 1:]:
> +msg = 'skipped - previous setup or teardown failed'
> +tap += 'ok {} - {} # {} {} {} \n'.format(
> +count, tidx['id'], msg, index, badtest.get('id',
> '--Unknown--'))
> +count += 1
>
> -prepare_env(tidx["teardown"])
> -index += 1
> +tap += 'done flushing skipped test tap output\n'
>
> return tap
>
> --
> 2.15.1
>
Re: [PATCH net-next 1/7] tools: tc-testing: Command line parms
On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler wrote:
> Separate the functionality of the command line parameters into "selection"
> parameters, "action" parameters and other parameters.
>
> "Selection" parameters are for choosing which tests on which to act.
> "Action" parameters are for choosing what to do with the selected tests.
> "Other" parameters are for global effect (like "help" or "verbose").
>
> With this commit, we add the ability to name a directory as another
> selection mechanism. We can accumulate a number of tests by directory,
> file, category, or even by test id, instead of being constrained to
> run all tests in one collection or just one test.
>
> Signed-off-by: Brenda J. Butler
Acked-by: Lucas Bates
> ---
> .../creating-testcases/AddingTestCases.txt | 35 +++-
> tools/testing/selftests/tc-testing/tdc.py | 209
> +
> tools/testing/selftests/tc-testing/tdc_helper.py | 15 +-
> 3 files changed, 164 insertions(+), 95 deletions(-)
>
> diff --git
> a/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
> b/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
> index 00438331ba47..17b267dedbd9 100644
> ---
> a/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
> +++
> b/tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt
> @@ -12,14 +12,18 @@ template.json for the required JSON format for test cases.
> Include the 'id' field, but do not assign a value. Running tdc with the -i
> option will generate a unique ID for that test case.
>
> -tdc will recursively search the 'tc' subdirectory for .json files. Any
> -test case files you create in these directories will automatically be
> included.
> -If you wish to store your custom test cases elsewhere, be sure to run tdc
> -with the -f argument and the path to your file.
> +tdc will recursively search the 'tc-tests' subdirectory (or the
> +directories named with the -D option) for .json files. Any test case
> +files you create in these directories will automatically be included.
> +If you wish to store your custom test cases elsewhere, be sure to run
> +tdc with the -f argument and the path to your file, or the -D argument
> +and the path to your directory(ies).
>
> -Be aware of required escape characters in the JSON data - particularly when
> -defining the match pattern. Refer to the tctests.json file for examples when
> -in doubt.
> +Be aware of required escape characters in the JSON data - particularly
> +when defining the match pattern. Refer to the supplied json test files
> +for examples when in doubt. The match pattern is written in json, and
> +will be used by python. So the match pattern will be a python regular
> +expression, but should be written using json syntax.
>
>
> TEST CASE STRUCTURE
> @@ -69,7 +73,8 @@ SETUP/TEARDOWN ERRORS
> If an error is detected during the setup/teardown process, execution of the
> tests will immediately stop with an error message and the namespace in which
> the tests are run will be destroyed. This is to prevent inaccurate results
> -in the test cases.
> +in the test cases. tdc will output a series of TAP results for the skipped
> +tests.
>
> Repeated failures of the setup/teardown may indicate a problem with the test
> case, or possibly even a bug in one of the commands that are not being
> tested.
> @@ -79,3 +84,17 @@ so that it doesn't halt the script for an error that
> doesn't matter. Turn the
> individual command into a list, with the command being first, followed by all
> acceptable exit codes for the command.
>
> +Example:
> +
> +A pair of setup commands. The first can have exit code 0, 1 or 255, the
> +second must have exit code 0.
> +
> +"setup": [
> +[
> +"$TC actions flush action gact",
> +0,
> +1,
> +255
> +],
> +"$TC actions add action reclassify index 65536"
> +],
> diff --git a/tools/testing/selftests/tc-testing/tdc.py
> b/tools/testing/selftests/tc-testing/tdc.py
> index fc373fdf2bdc..ef3a8881e458 100755
> --- a/tools/testing/selftests/tc-testing/tdc.py
> +++ b/tools/testing/selftests/tc-testing/tdc.py
> @@ -209,20 +209,41 @@ def set_args(parser):
> """
> Set the command line arguments for tdc.
> """
> -parser.add_argument('-p', '--path', type=str,
> -help='The full path to the tc executable to us
Re: [Patch net 00/16] net_sched: fix races with RCU callbacks
On Tue, Oct 31, 2017 at 7:02 PM, Cong Wang wrote: > On Tue, Oct 31, 2017 at 3:09 PM, Cong Wang wrote: >> >> This almost rules out the guilty of this patchset. >> >> I will provide a patch for you to test, since I can't reproduce it here. >> > > Lucas, please test the attached patch, it applies to latest -net. > > Note, it is a combination of 3 patches which together close the > use-after-free you reported here, I hope. > > Please let me know if this works. My basic tests run well without > any stack traces or memory leaks. The results look good to me. I compiled with the patch and ran the test again in a loop. I set it to run 50 times in a row and didn't trigger the bug once (previous record was 4 times).
Re: [Patch net 00/16] net_sched: fix races with RCU callbacks
On Tue, Oct 31, 2017 at 2:55 PM, Lucas Bates wrote:
> On Tue, Oct 31, 2017 at 7:00 AM, Jamal Hadi Salim wrote:
>> On 17-10-31 01:44 AM, Cong Wang wrote:
>>>
>>> Can you try this patch? From your stack trace it is not clear where
>>> the cause is, but we know that the crash is in __tcf_idr_release(),
>>> this is how I came up with the following patch:
>>>
>>>
>>> diff --git a/include/net/act_api.h b/include/net/act_api.h
>>> index b944e0eb93be..5072446d5f06 100644
>>> --- a/include/net/act_api.h
>>> +++ b/include/net/act_api.h
>>> @@ -122,7 +122,9 @@ void tcf_idrinfo_destroy(const struct tc_action_ops
>>> *ops,
>>>
>>> static inline void tc_action_net_exit(struct tc_action_net *tn)
>>> {
>>> + rtnl_lock();
>>> tcf_idrinfo_destroy(tn->ops, tn->idrinfo);
>>> + rtnl_unlock();
>>> kfree(tn->idrinfo);
>>> }
>>>
>>>
>>
>> Looks like an excellent bet;-> tdc kills the container at the end of the
>> test.
>
> Unfortunately it doesn't seem to have had any effect, I'm still seeing
> the same bug as yesterday. At Jamal's suggestion I put in a delay
> after tdc completed running the tests but before it deleted the
> container - and I didn't run into the bug after many runs. I had no
> luck getting serial console access on any of our other systems so I
> have another stack trace photo which I'll send to you directly.
I forgot to mention: it appears the bug appears to be related to
deleting the container. Test d052 is the last to run before tdc does
its teardown and deletes the container; if I add the sleep the bug
never gets triggered.
Re: [Patch net 00/16] net_sched: fix races with RCU callbacks
On Tue, Oct 31, 2017 at 7:00 AM, Jamal Hadi Salim wrote:
> On 17-10-31 01:44 AM, Cong Wang wrote:
>>
>> Can you try this patch? From your stack trace it is not clear where
>> the cause is, but we know that the crash is in __tcf_idr_release(),
>> this is how I came up with the following patch:
>>
>>
>> diff --git a/include/net/act_api.h b/include/net/act_api.h
>> index b944e0eb93be..5072446d5f06 100644
>> --- a/include/net/act_api.h
>> +++ b/include/net/act_api.h
>> @@ -122,7 +122,9 @@ void tcf_idrinfo_destroy(const struct tc_action_ops
>> *ops,
>>
>> static inline void tc_action_net_exit(struct tc_action_net *tn)
>> {
>> + rtnl_lock();
>> tcf_idrinfo_destroy(tn->ops, tn->idrinfo);
>> + rtnl_unlock();
>> kfree(tn->idrinfo);
>> }
>>
>>
>
> Looks like an excellent bet;-> tdc kills the container at the end of the
> test.
Unfortunately it doesn't seem to have had any effect, I'm still seeing
the same bug as yesterday. At Jamal's suggestion I put in a delay
after tdc completed running the tests but before it deleted the
container - and I didn't run into the bug after many runs. I had no
luck getting serial console access on any of our other systems so I
have another stack trace photo which I'll send to you directly.
Re: [PATCH net-next] tc-testing: better test case file error reporting
On Tue, Oct 31, 2017 at 2:29 PM, Brenda J. Butler wrote: > tdc.py reads a bunch of test cases in json files. When a json file > cannot be parsed, tdc just exits and does not run any tests. > > This patch will cause tdc to print a message with the file name and > line number, then that file will be ignored and the rest of the tests > will be processed. > > Signed-off-by: Brenda J. Butler With great pleasure: Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: "show" action shows all selected tests
On Tue, Oct 31, 2017 at 2:30 PM, Brenda J. Butler wrote: > The -s "show" option takes an argument to specify which test > to show. Change the option so it does not take an argument, > but outputs all the tests that were selected with the other > arguments. > > It is still possible to output a single test, by using the -e > option, but now more than one test can be displayed with > one command. > > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: better check if thing is list
On Tue, Oct 31, 2017 at 2:28 PM, Brenda J. Butler wrote: > Check if tcase[k] is an instance of a list (is or is derived from list) > instead of checking if it is a list. > > This will be useful if the data structures change to be something > that implements list, instead of being an actual list. In that > case, this code will not have to change. > > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: correction to docstring in get_unique_item
On Tue, Oct 31, 2017 at 2:28 PM, Brenda J. Butler wrote: > Signed-off-by: Brenda J. Butler > --- > tools/testing/selftests/tc-testing/tdc_helper.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/tools/testing/selftests/tc-testing/tdc_helper.py > b/tools/testing/selftests/tc-testing/tdc_helper.py > index c3254f861fb2..6f99a4efe761 100644 > --- a/tools/testing/selftests/tc-testing/tdc_helper.py > +++ b/tools/testing/selftests/tc-testing/tdc_helper.py > @@ -15,7 +15,7 @@ def get_categorized_testlist(alltests, ucat): > > > def get_unique_item(lst): > -""" For a list, return a set of the unique items in the list. """ > +""" For a list, return a list of the unique items in the list. """ > return list(set(lst)) > > > -- > 2.15.0.rc0 > Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: split config file
On Tue, Oct 31, 2017 at 2:27 PM, Brenda J. Butler wrote: > Move the config customization into a site-local file > tdc_config_local.py, so that updates of the tdc test > software does not require hand-editing of the config. > This patch includes a template for the site-local > customization file. > > In addition, this makes it easy to revert to a stock > tdc environment for testing the test framework and/or > the core tests. > > Also it makes it harder for any custom config to be > submitted back to the kernel tdc. > > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: gitignore, ignore standard python artifacts
On Tue, Oct 31, 2017 at 2:26 PM, Brenda J. Butler wrote: > Ignore .pyc files, "python compiled" files, that get created > when a python script is run. They should never be committed. > > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: very simple example test cases
On Tue, Oct 31, 2017 at 2:25 PM, Brenda J. Butler wrote: > As part of documentation, supply some very simple test cases > to illustrate how test cases work. One test case shows > commands in the setup, command, verify and teardown stages. > Other test cases show how to have a working test case that > does not have commands in the setup, verify and/or teardown > stages. > > Specifically, the command lists for setup and teardown can > be empty. And the verify command must have a command, but > it can be /bin/true. The regex must have a string, we > recommend a single space, and the count of matches must be > zero if you do not want to use the match feature of verify. > Verify will always look for a return code of success (0) > so we give /bin/true when we do not want to make a check > there. > > Also, update the documentation for testcases to be more > specific in the cases of: > > - accepting non-success return codes in setup and > teardown stages > - how to write the test when no setup, teardown > and/or verify are desired. > > To run the example test cases: > > $ sudo -E ./tdc.py -f creating-testcases/example.json -l > 1f: (example) simple test to test framework > 2f: (example) simple test, no need for verify > 3f: (example) simple test, no need for setup or teardown (or verify) > $ sudo -E ./tdc.py -f creating-testcases/example.json > Test 1f: simple test to test framework > Test 2f: simple test, no need for verify > Test 3f: simple test, no need for setup or teardown (or verify) > All test results: > > 1..3 > ok 1 1f simple test to test framework > ok 2 2f simple test, no need for verify > ok 3 3f simple test, no need for setup or teardown (or verify) > > $ > > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates
Re: [PATCH net-next] tc-testing: correction to docstring in get_unique_item
On Mon, Oct 30, 2017 at 8:37 PM, Brenda J. Butler wrote: > The docstring says the function returns a "set" but it returns > a "list". These are both python data types, so we should refer to > the right one that is being returned. > > Signed-off-by: Brenda J. Butler Acked-by: Lucas Bates
Re: [Patch net 00/16] net_sched: fix races with RCU callbacks
On Mon, Oct 30, 2017 at 7:12 PM, Cong Wang wrote: > On Mon, Oct 30, 2017 at 3:39 PM, Lucas Bates wrote: >> e.On Thu, Oct 26, 2017 at 9:24 PM, Cong Wang >> wrote: >>> Recently, the RCU callbacks used in TC filters and TC actions keep >>> drawing my attention, they introduce at least 4 race condition bugs: >> >>> As suggested by Paul, we could defer the work to a workqueue and >>> gain the permission of holding RTNL again without any performance >>> impact, however, in tcf_block_put() we could have a deadlock when >>> flushing workqueue while hodling RTNL lock, the trick here is to >>> defer the work itself in workqueue and make it queued after all >>> other works so that we keep the same ordering to avoid any >>> use-after-free. Please see the first patch for details. >> >> Cong, I don't believe the problem's been resolved just yet I have >> a new kernel, compiled just today and I'm still tripping over a kernel >> bug in this scenario when I run Chris' new test case. >> > > Without a stack trace, I can't do anything. "a kernel bug" could > be anything, why do you believe it is caused by this patchset? > (Sending again due to email format issues) The stack trace I saw touched on the same code that was affected by the patchset. I've sent a photo of the trace to you directly - I'm sorry, I should have sent it earlier. I also apologize for having to send the photo but I was doing the testing on a small device lacking any kind of serial console access. >> I'm doing this on a machine where I don't have a spare device to use >> on the run. Instead I created a veth pair that will have one end >> migrated into the container. >> >> The bug isn't consistent. I'm running into it anywhere between one and >> four runs of the d052 test case. >> >> Steps to reproduce: >> >> sudo ip li add foo type veth >> sudo ./tdc.py -d foo -c flower >> [repeat until kernel bug encountered] > > I tested with basic and u32 filter, since it is not specific to any type > of filters, all filters had the same problem. I will try flower filter too > before you provide a stack trace or a detailed report. If you need more information or need me to try something else, I'll be able to tomorrow.
Re: [Patch net 00/16] net_sched: fix races with RCU callbacks
e.On Thu, Oct 26, 2017 at 9:24 PM, Cong Wang wrote: > Recently, the RCU callbacks used in TC filters and TC actions keep > drawing my attention, they introduce at least 4 race condition bugs: > As suggested by Paul, we could defer the work to a workqueue and > gain the permission of holding RTNL again without any performance > impact, however, in tcf_block_put() we could have a deadlock when > flushing workqueue while hodling RTNL lock, the trick here is to > defer the work itself in workqueue and make it queued after all > other works so that we keep the same ordering to avoid any > use-after-free. Please see the first patch for details. Cong, I don't believe the problem's been resolved just yet I have a new kernel, compiled just today and I'm still tripping over a kernel bug in this scenario when I run Chris' new test case. I'm doing this on a machine where I don't have a spare device to use on the run. Instead I created a veth pair that will have one end migrated into the container. The bug isn't consistent. I'm running into it anywhere between one and four runs of the d052 test case. Steps to reproduce: sudo ip li add foo type veth sudo ./tdc.py -d foo -c flower [repeat until kernel bug encountered]
Re: [patch net v2 4/4] selftests: Introduce a new test case to tc testsuite
On Mon, Oct 16, 2017 at 7:18 AM, Chris Mi wrote:
> In this patchset, we fixed a tc bug. This patch adds the test case
> that reproduces the bug. To run this test case, user should specify
> an existing NIC device:
> # sudo ./tdc.py -d enp4s0f0
>
> This test case belongs to category "flower". If user doesn't specify
> a NIC device, the test cases belong to "flower" will not be run.
>
> In this test case, we create 1M filters and all filters share the same
> action. When destroying all filters, kernel should not panic. It takes
> about 18s to run it.
>
> Signed-off-by: Chris Mi
> Acked-by: Jamal Hadi Salim
I'm a little wary about adding changes like these into tdc.py
directly; I don't think it's going to be sustainable in the long run.
Even the namespace creation I put in to the original version is too
specific and limiting.
There are some upcoming changes to tdc to help address these
particular issues. I'll ack this for now, thanks.
Acked-by: Lucas Bates
> ---
> .../tc-testing/tc-tests/filters/tests.json | 23
> +-
> tools/testing/selftests/tc-testing/tdc.py | 20 +++
> tools/testing/selftests/tc-testing/tdc_config.py | 2 ++
> 3 files changed, 40 insertions(+), 5 deletions(-)
>
> diff --git a/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json
> b/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json
> index c727b96..5fa02d8 100644
> --- a/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json
> +++ b/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json
> @@ -17,5 +17,26 @@
> "teardown": [
> "$TC qdisc del dev $DEV1 ingress"
> ]
> +},
> +{
> +"id": "d052",
> +"name": "Add 1M filters with the same action",
> +"category": [
> +"filter",
> +"flower"
> +],
> +"setup": [
> +"$TC qdisc add dev $DEV2 ingress",
> +"./tdc_batch.py $DEV2 $BATCH_FILE --share_action -n 100"
> +],
> +"cmdUnderTest": "$TC -b $BATCH_FILE",
> +"expExitCode": "0",
> +"verifyCmd": "$TC actions list action gact",
> +"matchPattern": "action order 0: gact action drop.*index 1 ref
> 100 bind 100",
> +"matchCount": "1",
> +"teardown": [
> +"$TC qdisc del dev $DEV2 ingress",
> +"/bin/rm $BATCH_FILE"
> +]
> }
> -]
> \ No newline at end of file
> +]
> diff --git a/tools/testing/selftests/tc-testing/tdc.py
> b/tools/testing/selftests/tc-testing/tdc.py
> index cd61b78..5f11f5d 100755
> --- a/tools/testing/selftests/tc-testing/tdc.py
> +++ b/tools/testing/selftests/tc-testing/tdc.py
> @@ -88,7 +88,7 @@ def prepare_env(cmdlist):
> exit(1)
>
>
> -def test_runner(filtered_tests):
> +def test_runner(filtered_tests, args):
> """
> Driver function for the unit tests.
>
> @@ -105,6 +105,8 @@ def test_runner(filtered_tests):
> for tidx in testlist:
> result = True
> tresult = ""
> +if "flower" in tidx["category"] and args.device == None:
> +continue
> print("Test " + tidx["id"] + ": " + tidx["name"])
> prepare_env(tidx["setup"])
> (p, procout) = exec_cmd(tidx["cmdUnderTest"])
> @@ -152,6 +154,10 @@ def ns_create():
> exec_cmd(cmd, False)
> cmd = 'ip -s $NS link set $DEV1 up'
> exec_cmd(cmd, False)
> +cmd = 'ip link set $DEV2 netns $NS'
> +exec_cmd(cmd, False)
> +cmd = 'ip -s $NS link set $DEV2 up'
> +exec_cmd(cmd, False)
>
>
> def ns_destroy():
> @@ -211,7 +217,8 @@ def set_args(parser):
> help='Execute the single test case with specified
> ID')
> parser.add_argument('-i', '--id', action='store_true', dest='gen_id',
> help='Generate ID numbers for new test cases')
> -return parser
> +parser.add_argument('-d', '--device',
> +help='Execute the test case in flower category')
> return parser
>
>
> @@ -225,6 +232,8 @@ def check_default_settings(args):
>
Re: [patch net v2 3/4] selftests: Introduce a new script to generate tc batch file
On Mon, Oct 16, 2017 at 7:18 AM, Chris Mi wrote:
> # ./tdc_batch.py -h
> usage: tdc_batch.py [-h] [-n NUMBER] [-o] [-s] [-p] device file
>
> TC batch file generator
>
> positional arguments:
> devicedevice name
> file batch file name
>
> optional arguments:
> -h, --helpshow this help message and exit
> -n NUMBER, --number NUMBER
> how many lines in batch file
> -o, --skip_sw skip_sw (offload), by default skip_hw
> -s, --share_actionall filters share the same action
> -p, --prioall filters have different prio
>
> Signed-off-by: Chris Mi
> Acked-by: Jamal Hadi Salim
Acked-by: Lucas Bates
> ---
> tools/testing/selftests/tc-testing/tdc_batch.py | 62
> +
> 1 file changed, 62 insertions(+)
> create mode 100755 tools/testing/selftests/tc-testing/tdc_batch.py
>
> diff --git a/tools/testing/selftests/tc-testing/tdc_batch.py
> b/tools/testing/selftests/tc-testing/tdc_batch.py
> new file mode 100755
> index 000..707c6bf
> --- /dev/null
> +++ b/tools/testing/selftests/tc-testing/tdc_batch.py
> @@ -0,0 +1,62 @@
> +#!/usr/bin/python3
> +
> +"""
> +tdc_batch.py - a script to generate TC batch file
> +
> +Copyright (C) 2017 Chris Mi
> +"""
> +
> +import argparse
> +
> +parser = argparse.ArgumentParser(description='TC batch file generator')
> +parser.add_argument("device", help="device name")
> +parser.add_argument("file", help="batch file name")
> +parser.add_argument("-n", "--number", type=int,
> +help="how many lines in batch file")
> +parser.add_argument("-o", "--skip_sw",
> +help="skip_sw (offload), by default skip_hw",
> +action="store_true")
> +parser.add_argument("-s", "--share_action",
> +help="all filters share the same action",
> +action="store_true")
> +parser.add_argument("-p", "--prio",
> +help="all filters have different prio",
> +action="store_true")
> +args = parser.parse_args()
> +
> +device = args.device
> +file = open(args.file, 'w')
> +
> +number = 1
> +if args.number:
> +number = args.number
> +
> +skip = "skip_hw"
> +if args.skip_sw:
> +skip = "skip_sw"
> +
> +share_action = ""
> +if args.share_action:
> +share_action = "index 1"
> +
> +prio = "prio 1"
> +if args.prio:
> +prio = ""
> +if number > 0x4000:
> +number = 0x4000
> +
> +index = 0
> +for i in range(0x100):
> +for j in range(0x100):
> +for k in range(0x100):
> +mac = ("%02x:%02x:%02x" % (i, j, k))
> +src_mac = "e4:11:00:" + mac
> +dst_mac = "e4:12:00:" + mac
> +cmd = ("filter add dev %s %s protocol ip parent : flower %s "
> + "src_mac %s dst_mac %s action drop %s" %
> + (device, prio, skip, src_mac, dst_mac, share_action))
> +file.write("%s\n" % cmd)
> +index += 1
> +if index >= number:
> +file.close()
> +exit(0)
> --
> 1.8.3.1
>
[PATCH net-next v2 0/4] tc-testing: Test suite updates
This patch series is a roundup of changes to the tc-testing suite: - Add test cases for police and mirred modules and some coverage in already-submitted test categories - Break the test case files down into more user-friendly sizes - Bug fix to the tdc.py script's handling of the -l argument v2: fix the lack of final newlines in two new files (thanks David) Lucas Bates (4): tc-testing: Add test cases for flushing actions tc-testing: Split test case files into smaller chunks tc-testing: Add test cases for police and skbmod tc-testing: fix the -l argument bug in tdc.py .../tc-testing/tc-tests/actions/gact.json | 469 .../selftests/tc-testing/tc-tests/actions/ife.json | 52 + .../tc-testing/tc-tests/actions/mirred.json| 223 .../tc-testing/tc-tests/actions/police.json| 527 + .../tc-testing/tc-tests/actions/simple.json| 130 +++ .../tc-testing/tc-tests/actions/skbedit.json | 320 ++ .../tc-testing/tc-tests/actions/skbmod.json| 372 +++ .../tc-testing/tc-tests/actions/tests.json | 1165 tools/testing/selftests/tc-testing/tdc.py |8 +- 9 files changed, 2097 insertions(+), 1169 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/gact.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/ife.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/mirred.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/police.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/simple.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/skbedit.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/skbmod.json delete mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/tests.json -- 2.7.4
[PATCH net-next v2 4/4] tc-testing: fix the -l argument bug in tdc.py
This patch fixes a bug in the tdc script, where executing tdc
with the -l argument would cause the tests to start running
as opposed to listing all the known test cases.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
tools/testing/selftests/tc-testing/tdc.py | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index cd61b78..d2391df 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -49,7 +49,7 @@ def exec_cmd(command, nsonly=True):
stderr=subprocess.PIPE)
(rawout, serr) = proc.communicate()
-if proc.returncode != 0:
+if proc.returncode != 0 and len(serr) > 0:
foutput = serr.decode("utf-8")
else:
foutput = rawout.decode("utf-8")
@@ -203,7 +203,7 @@ def set_args(parser):
help='Run tests only from the specified category, or
if no category is specified, list known categories.')
parser.add_argument('-f', '--file', type=str,
help='Run tests from the specified file')
-parser.add_argument('-l', '--list', type=str, nargs='?', const="",
metavar='CATEGORY',
+parser.add_argument('-l', '--list', type=str, nargs='?', const="++",
metavar='CATEGORY',
help='List all test cases, or those only within the
specified category')
parser.add_argument('-s', '--show', type=str, nargs=1, metavar='ID',
dest='showID',
help='Display the test case with specified id')
@@ -357,10 +357,10 @@ def set_operation_mode(args):
testcases = get_categorized_testlist(alltests, ucat)
if args.list:
-if (len(args.list) == 0):
+if (args.list == "++"):
list_test_cases(alltests)
exit(0)
-elif(len(args.list > 0)):
+elif(len(args.list) > 0):
if (args.list not in ucat):
print("Unknown category " + args.list)
print("Available categories:")
--
2.7.4
[PATCH net-next v2 2/4] tc-testing: Split test case files into smaller chunks
The original submission had the test cases stored in one
monolithic file. This can be unwieldy to edit, especially as more
test cases are added. This patch removes the original tests.json
file in favour of individual ones broken down by category.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
.../tc-testing/tc-tests/actions/gact.json | 469
.../selftests/tc-testing/tc-tests/actions/ife.json | 52 +
.../tc-testing/tc-tests/actions/mirred.json| 223
.../tc-testing/tc-tests/actions/simple.json| 130 +++
.../tc-testing/tc-tests/actions/skbedit.json | 320 ++
.../tc-testing/tc-tests/actions/tests.json | 1212
6 files changed, 1194 insertions(+), 1212 deletions(-)
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/ife.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/mirred.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/simple.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/skbedit.json
delete mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
new file mode 100644
index 000..e2187b6
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
@@ -0,0 +1,469 @@
+[
+{
+"id": "e89a",
+"name": "Add valid pass action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action pass index 8",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action pass.*index 8 ref",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action gact"
+]
+},
+{
+"id": "a02c",
+"name": "Add valid pipe action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action pipe index 6",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action pipe.*index 6 ref",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action gact"
+]
+},
+{
+"id": "feef",
+"name": "Add valid reclassify action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action reclassify index 5",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action reclassify.*index 5
ref",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action gact"
+]
+},
+{
+"id": "8a7a",
+"name": "Add valid drop action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action drop index 30",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action drop.*index 30 ref",
+"matchCount": "1",
+"
[PATCH net-next v2 3/4] tc-testing: Add test cases for police and skbmod
Add basic unit tests for police and skbmod actions in tc.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
.../tc-testing/tc-tests/actions/police.json| 527 +
.../tc-testing/tc-tests/actions/skbmod.json| 372 +++
2 files changed, 899 insertions(+)
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/police.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/skbmod.json
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/police.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/police.json
new file mode 100644
index 000..0e602a3
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/police.json
@@ -0,0 +1,527 @@
+[
+{
+"id": "49aa",
+"name": "Add valid basic police action",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action police rate 1kbit burst 10k
index 1",
+"expExitCode": "0",
+"verifyCmd": "$TC actions ls action police",
+"matchPattern": "action order [0-9]*: police 0x1 rate 1Kbit burst
10Kb",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "3abe",
+"name": "Add police action with duplicate index",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+],
+"$TC actions add action police rate 4Mbit burst 120k index 9"
+],
+"cmdUnderTest": "$TC actions add action police rate 8kbit burst 24k
index 9",
+"expExitCode": "255",
+"verifyCmd": "$TC actions ls action police",
+"matchPattern": "action order [0-9]*: police 0x9",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "49fa",
+"name": "Add valid police action with mtu",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action police rate 90kbit burst 10k
mtu 1k index 98",
+"expExitCode": "0",
+"verifyCmd": "$TC actions get action police index 98",
+"matchPattern": "action order [0-9]*: police 0x62 rate 90Kbit burst
10Kb mtu 1Kb",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "7943",
+"name": "Add valid police action with peakrate",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action police rate 90kbit burst 10k
mtu 2kb peakrate 100kbit index 3",
+"expExitCode": "0",
+"verifyCmd": "$TC actions ls action police",
+"matchPattern": "action order [0-9]*: police 0x3 rate 90Kbit burst
10Kb mtu 2Kb peakrate 100Kbit",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "055e",
+"name": "Add police action with peakrate and no mtu",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+
[PATCH net-next v2 1/4] tc-testing: Add test cases for flushing actions
Tests for flushing gact and mirred were missing. This patch
adds test cases to explicitly test the flush of any installed
gact/mirred actions.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
.../tc-testing/tc-tests/actions/tests.json | 49 +-
1 file changed, 48 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
index 6973bdc..2ea0065 100644
--- a/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
@@ -246,6 +246,27 @@
]
},
{
+"id": "3edf",
+"name": "Flush gact actions",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+"$TC actions add action reclassify index 101",
+"$TC actions add action reclassify index 102",
+"$TC actions add action reclassify index 103",
+"$TC actions add action reclassify index 104",
+"$TC actions add action reclassify index 105"
+],
+"cmdUnderTest": "$TC actions flush action gact",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action reclassify",
+"matchCount": "0",
+"teardown": []
+},
+{
"id": "63ec",
"name": "Delete pass action",
"category": [
@@ -469,6 +490,32 @@
]
},
{
+"id": "58c3",
+"name": "Flush mirred actions",
+"category": [
+"actions",
+"mirred"
+],
+"setup": [
+[
+"$TC actions flush action mirred",
+0,
+1,
+255
+],
+"$TC actions add action mirred egress mirror index 1 dev lo",
+"$TC actions add action mirred egress redirect index 2 dev lo"
+],
+"cmdUnderTest": "$TC actions show action mirred",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action mirred",
+"matchPattern": "[Mirror|Redirect] to device lo",
+"matchCount": "0",
+"teardown": [
+"$TC actions flush action mirred"
+]
+},
+{
"id": "d7c0",
"name": "Add invalid mirred direction",
"category": [
@@ -1162,4 +1209,4 @@
"$TC actions flush action ife"
]
}
-]
\ No newline at end of file
+]
--
2.7.4
[PATCH net-next 0/4] tc-testing: Test suite updates
This patch series is a roundup of changes to the tc-testing suite: - Add test cases for police and mirred modules and some coverage in already-submitted test categories - Break the test case files down into more user-friendly sizes - Bug fix to the tdc.py script's handling of the -l argument Lucas Bates (4): tc-testing: Add test cases for flushing actions tc-testing: Split test case files into smaller chunks tc-testing: Add test cases for police and skbmod tc-testing: fix the -l argument bug in tdc.py script .../tc-testing/tc-tests/actions/gact.json | 469 .../selftests/tc-testing/tc-tests/actions/ife.json | 52 + .../tc-testing/tc-tests/actions/mirred.json| 223 .../tc-testing/tc-tests/actions/police.json| 527 + .../tc-testing/tc-tests/actions/simple.json| 130 +++ .../tc-testing/tc-tests/actions/skbedit.json | 320 ++ .../tc-testing/tc-tests/actions/skbmod.json| 372 +++ .../tc-testing/tc-tests/actions/tests.json | 1165 tools/testing/selftests/tc-testing/tdc.py |8 +- 9 files changed, 2097 insertions(+), 1169 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/gact.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/ife.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/mirred.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/police.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/simple.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/skbedit.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/skbmod.json delete mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/tests.json -- 2.7.4
[PATCH net-next 1/4] tc-testing: Add test cases for flushing actions
Tests for flushing gact and mirred were missing. This patch
adds test cases to explicitly test the flush of any installed
gact/mirred actions.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
.../tc-testing/tc-tests/actions/tests.json | 49 +-
1 file changed, 48 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
index 6973bdc..2ea0065 100644
--- a/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
@@ -246,6 +246,27 @@
]
},
{
+"id": "3edf",
+"name": "Flush gact actions",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+"$TC actions add action reclassify index 101",
+"$TC actions add action reclassify index 102",
+"$TC actions add action reclassify index 103",
+"$TC actions add action reclassify index 104",
+"$TC actions add action reclassify index 105"
+],
+"cmdUnderTest": "$TC actions flush action gact",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action reclassify",
+"matchCount": "0",
+"teardown": []
+},
+{
"id": "63ec",
"name": "Delete pass action",
"category": [
@@ -469,6 +490,32 @@
]
},
{
+"id": "58c3",
+"name": "Flush mirred actions",
+"category": [
+"actions",
+"mirred"
+],
+"setup": [
+[
+"$TC actions flush action mirred",
+0,
+1,
+255
+],
+"$TC actions add action mirred egress mirror index 1 dev lo",
+"$TC actions add action mirred egress redirect index 2 dev lo"
+],
+"cmdUnderTest": "$TC actions show action mirred",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action mirred",
+"matchPattern": "[Mirror|Redirect] to device lo",
+"matchCount": "0",
+"teardown": [
+"$TC actions flush action mirred"
+]
+},
+{
"id": "d7c0",
"name": "Add invalid mirred direction",
"category": [
@@ -1162,4 +1209,4 @@
"$TC actions flush action ife"
]
}
-]
\ No newline at end of file
+]
--
2.7.4
[PATCH net-next 2/4] tc-testing: Split test case files into smaller chunks
The original submission had the test cases stored in one
monolithic file. This can be unwieldy to edit, especially as more
test cases are added. This patch removes the original tests.json
file in favour of individual ones broken down by category.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
.../tc-testing/tc-tests/actions/gact.json | 469
.../selftests/tc-testing/tc-tests/actions/ife.json | 52 +
.../tc-testing/tc-tests/actions/mirred.json| 223
.../tc-testing/tc-tests/actions/simple.json| 130 +++
.../tc-testing/tc-tests/actions/skbedit.json | 320 ++
.../tc-testing/tc-tests/actions/tests.json | 1212
6 files changed, 1194 insertions(+), 1212 deletions(-)
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/ife.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/mirred.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/simple.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/skbedit.json
delete mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/tests.json
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
new file mode 100644
index 000..e2187b6
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/gact.json
@@ -0,0 +1,469 @@
+[
+{
+"id": "e89a",
+"name": "Add valid pass action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action pass index 8",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action pass.*index 8 ref",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action gact"
+]
+},
+{
+"id": "a02c",
+"name": "Add valid pipe action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action pipe index 6",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action pipe.*index 6 ref",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action gact"
+]
+},
+{
+"id": "feef",
+"name": "Add valid reclassify action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action reclassify index 5",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action reclassify.*index 5
ref",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action gact"
+]
+},
+{
+"id": "8a7a",
+"name": "Add valid drop action",
+"category": [
+"actions",
+"gact"
+],
+"setup": [
+[
+"$TC actions flush action gact",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action drop index 30",
+"expExitCode": "0",
+"verifyCmd": "$TC actions list action gact",
+"matchPattern": "action order [0-9]*: gact action drop.*index 30 ref",
+"matchCount": "1",
+"
[PATCH net-next 3/4] tc-testing: Add test cases for police and skbmod
Add basic unit tests for police and skbmod actions in tc.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
.../tc-testing/tc-tests/actions/police.json| 527 +
.../tc-testing/tc-tests/actions/skbmod.json| 372 +++
2 files changed, 899 insertions(+)
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/police.json
create mode 100644
tools/testing/selftests/tc-testing/tc-tests/actions/skbmod.json
diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/police.json
b/tools/testing/selftests/tc-testing/tc-tests/actions/police.json
new file mode 100644
index 000..82d0990
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/tc-tests/actions/police.json
@@ -0,0 +1,527 @@
+[
+{
+"id": "49aa",
+"name": "Add valid basic police action",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action police rate 1kbit burst 10k
index 1",
+"expExitCode": "0",
+"verifyCmd": "$TC actions ls action police",
+"matchPattern": "action order [0-9]*: police 0x1 rate 1Kbit burst
10Kb",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "3abe",
+"name": "Add police action with duplicate index",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+],
+"$TC actions add action police rate 4Mbit burst 120k index 9"
+],
+"cmdUnderTest": "$TC actions add action police rate 8kbit burst 24k
index 9",
+"expExitCode": "255",
+"verifyCmd": "$TC actions ls action police",
+"matchPattern": "action order [0-9]*: police 0x9",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "49fa",
+"name": "Add valid police action with mtu",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action police rate 90kbit burst 10k
mtu 1k index 98",
+"expExitCode": "0",
+"verifyCmd": "$TC actions get action police index 98",
+"matchPattern": "action order [0-9]*: police 0x62 rate 90Kbit burst
10Kb mtu 1Kb",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "7943",
+"name": "Add valid police action with peakrate",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+1,
+255
+]
+],
+"cmdUnderTest": "$TC actions add action police rate 90kbit burst 10k
mtu 2kb peakrate 100kbit index 3",
+"expExitCode": "0",
+"verifyCmd": "$TC actions ls action police",
+"matchPattern": "action order [0-9]*: police 0x3 rate 90Kbit burst
10Kb mtu 2Kb peakrate 100Kbit",
+"matchCount": "1",
+"teardown": [
+"$TC actions flush action police"
+]
+},
+{
+"id": "055e",
+"name": "Add police action with peakrate and no mtu",
+"category": [
+"actions",
+"police"
+],
+"setup": [
+[
+"$TC actions flush action police",
+0,
+
[PATCH net-next 4/4] tc-testing: fix the -l argument bug in tdc.py script
This patch fixes a bug in the tdc script, where executing tdc
with the -l argument would cause the tests to start running
as opposed to listing all the known test cases.
Signed-off-by: Lucas Bates
Acked-by: Jamal Hadi Salim
---
tools/testing/selftests/tc-testing/tdc.py | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py
b/tools/testing/selftests/tc-testing/tdc.py
index cd61b78..d2391df 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -49,7 +49,7 @@ def exec_cmd(command, nsonly=True):
stderr=subprocess.PIPE)
(rawout, serr) = proc.communicate()
-if proc.returncode != 0:
+if proc.returncode != 0 and len(serr) > 0:
foutput = serr.decode("utf-8")
else:
foutput = rawout.decode("utf-8")
@@ -203,7 +203,7 @@ def set_args(parser):
help='Run tests only from the specified category, or
if no category is specified, list known categories.')
parser.add_argument('-f', '--file', type=str,
help='Run tests from the specified file')
-parser.add_argument('-l', '--list', type=str, nargs='?', const="",
metavar='CATEGORY',
+parser.add_argument('-l', '--list', type=str, nargs='?', const="++",
metavar='CATEGORY',
help='List all test cases, or those only within the
specified category')
parser.add_argument('-s', '--show', type=str, nargs=1, metavar='ID',
dest='showID',
help='Display the test case with specified id')
@@ -357,10 +357,10 @@ def set_operation_mode(args):
testcases = get_categorized_testlist(alltests, ucat)
if args.list:
-if (len(args.list) == 0):
+if (args.list == "++"):
list_test_cases(alltests)
exit(0)
-elif(len(args.list > 0)):
+elif(len(args.list) > 0):
if (args.list not in ucat):
print("Unknown category " + args.list)
print("Available categories:")
--
2.7.4
[PATCH iproute2 1/1] Add new man page for tc actions.
This page is to highlight all operations and options that are
applicable to all tc actions.
Signed-off-by: Lucas Bates
Signed-off-by: Jamal Hadi Salim
---
man/man8/tc-actions.8 | 221 ++
1 file changed, 221 insertions(+)
create mode 100644 man/man8/tc-actions.8
diff --git a/man/man8/tc-actions.8 b/man/man8/tc-actions.8
new file mode 100644
index 000..51f43af
--- /dev/null
+++ b/man/man8/tc-actions.8
@@ -0,0 +1,221 @@
+.TH "actions in tc" 8 "4 Jul 2017" "iproute2" "Linux"
+
+.SH NAME
+actions \- independently defined actions in tc
+.SH SYNOPSIS
+.B tc
+[
+.I TC_OPTIONS
+]
+.B actions
+.BR add " | " change " | " replace
+.I ACTSPEC
+
+.B tc
+[
+.I TC_OPTIONS
+]
+.B actions
+.BR get " | " delete
+.I ACTISPEC
+
+.B tc
+[
+.I TC_OPTIONS
+]
+.B actions flush
+.I ACTNAMESPEC
+
+.B tc
+[
+.I TC_OPTIONS
+]
+.B actions
+.BR ls " | " list
+.I ACTNAMESPEC
+
+.in +8
+.I ACTSPEC
+:=
+.B action
+.I ACTDETAIL
+[
+.I INDEXSPEC
+] [
+.I COOKIESPEC
+] [
+.I CONTROL
+]
+
+.I ACTISPEC
+:=
+.I ACTNAMESPEC INDEXSPEC
+
+.I ACTNAMESPEC
+:=
+.B action
+ACTNAME
+
+.I INDEXSPEC
+:=
+.BI index " INDEX"
+
+.I COOKIESPEC
+:=
+.BI cookie " COOKIE"
+
+.I ACTDETAIL
+:=
+.I ACTNAME ACTPARAMS
+
+.I ACTNAME
+may be any valid action type: gact, mirred, bpf, connmark, csum, police, etc.
+
+.I ACTPARAMS
+are the action-specific parameters; see the man page for the specific action
+type to be used for details.
+
+.I CONTROL
+:= {
+.IR reclassify " | " pipe " | " drop " | " continue " | " ok
+}
+
+.I TC_OPTIONS
+These are the options that are specific to
+.B tc
+and not only the options. Refer to
+.BR tc(8)
+for more information.
+.in
+
+.SH DESCRIPTION
+
+The
+.B actions
+object in
+.B tc
+allows a user to define actions independently of a classifier (filter). These
+actions can then be assigned to one or more filters, with any
+packets matching the classifier's criteria having that action performed
+on them.
+
+Each action type (mirred, police, etc.) will have its own table to store
+all created actions.
+
+.SH OPERATIONS
+.TP
+.B add
+Create a new action in that action's table.
+
+.TP
+.B change
+.TQ
+.B replace
+Make modifications to an existing action.
+.TP
+.B get
+Display the action with the specified index value. When combined with the
+.B -s
+option for
+.BR tc ","
+display the statistics for that action.
+.TP
+.B delete
+Delete the action with the specified index value. If the action is already
+associated with a classifier, it does not delete the classifier.
+.TP
+.B ls
+.TQ
+.B list
+List all the actions in the specified table. When combined with the
+.B -s
+option for
+.BR tc ","
+display the statistics for all actions in the specified table.
+.TP
+.B flush
+Delete all actions stored in the specified table.
+
+.SH ACTION OPTIONS
+Note that these options are available to all action types.
+.TP
+.BI index " INDEX"
+Specify the table index value of an action.
+.I INDEX
+is a 32-bit value that is unique to the specific type of action referenced.
+
+.RS
+For
+.BR add ", " change ", and"
+.B replace
+operations, the index is
+.BR optional.
+When adding a new action,
+specifying an index value will assign the action to that index unless that
+index value has already been assigned. Omitting the index value for an add
+operation will cause the kernel to assign a value to the new action.
+.RE
+
+.RS
+For
+.BR get " and " delete
+operations, the index is
+.B required
+to identify the specific action to be displayed or deleted.
+.RE
+
+.TP
+.BI cookie " COOKIE"
+In addition to the specific action, mark the matching packet with the value
+specified by
+.IR COOKIE "."
+The
+.I COOKIE
+is a 128-bit value that will not be interpreted by the kernel whatsoever.
+As such, it can be used as a correlating value for maintaining user state.
+The value to be stored is completely arbitrary and does not require a specific
+format. It is stored inside the action structure itself.
+
+.TP
+.I CONTROL
+The
+.I CONTROL
+indicates how
+.B tc
+should proceed after executing the action. Any of the following are valid:
+.RS
+.TP
+.B reclassify
+Restart the classifiction by jumping back to the first filter attached to
+the action's parent.
+.TP
+.B pipe
+Continue with the next action. This is the default control.
+.TP
+.B drop
+Drop the packed without running any further actions.
+.TP
+.B continue
+Continue the classification with the next filter.
+.TP
+.B pass
+Return to the calling qdisc for packet processing, and end classification of
+this packet.
+.RE
+
+.SH SEE ALSO
+.BR tc (8),
+.BR tc-bpf (8),
+.BR tc-connmark (8),
+.BR tc-csum (8),
+.BR tc-ife (8),
+.BR tc-mirred (8),
+.BR tc-nat (8),
+.BR tc-pedit (8),
+.BR tc-police (8),
+.BR tc-simple (8),
+.BR tc-skbedit (8),
+.BR tc-skbmod (8),
+.BR tc-tunnel_key (8),
+.BR tc-vlan (8),
+.BR tc-xt (8)
--
2.7.4
[PATCH net-next 1/1] selftests: Introduce tc testsuite
Add the beginnings of a testsuite for tc functionality in the kernel. These are a series of unit tests that use the tc executable and verify the success of those commands by checking both the exit codes and the output from tc's 'show' operation. To run the tests: # cd tools/testing/selftests/tc-testing # sudo ./tdc.py You can specify the tc executable to use with the -p argument on the command line or editing the 'TC' variable in tdc_config.py. Refer to the README for full details on how to run. The initial complement of test cases are limited mostly to tc actions. Test cases are most welcome; see the creating-testcases subdirectory for help in creating them. Signed-off-by: Lucas Bates Signed-off-by: Jamal Hadi Salim --- tools/testing/selftests/tc-testing/.gitignore |1 + tools/testing/selftests/tc-testing/README | 102 ++ tools/testing/selftests/tc-testing/TODO.txt| 10 + .../creating-testcases/AddingTestCases.txt | 69 ++ .../tc-testing/creating-testcases/template.json| 40 + .../tc-testing/tc-tests/actions/tests.json | 1115 .../tc-testing/tc-tests/filters/tests.json | 21 + tools/testing/selftests/tc-testing/tdc.py | 413 tools/testing/selftests/tc-testing/tdc_config.py | 17 + tools/testing/selftests/tc-testing/tdc_helper.py | 75 ++ 10 files changed, 1863 insertions(+) create mode 100644 tools/testing/selftests/tc-testing/.gitignore create mode 100644 tools/testing/selftests/tc-testing/README create mode 100644 tools/testing/selftests/tc-testing/TODO.txt create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/template.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/tests.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/filters/tests.json create mode 100755 tools/testing/selftests/tc-testing/tdc.py create mode 100644 tools/testing/selftests/tc-testing/tdc_config.py create mode 100644 tools/testing/selftests/tc-testing/tdc_helper.py diff --git a/tools/testing/selftests/tc-testing/.gitignore b/tools/testing/selftests/tc-testing/.gitignore new file mode 100644 index 000..c18dd8d --- /dev/null +++ b/tools/testing/selftests/tc-testing/.gitignore @@ -0,0 +1 @@ +__pycache__/ diff --git a/tools/testing/selftests/tc-testing/README b/tools/testing/selftests/tc-testing/README new file mode 100644 index 000..970ff29 --- /dev/null +++ b/tools/testing/selftests/tc-testing/README @@ -0,0 +1,102 @@ +tdc - Linux Traffic Control (tc) unit testing suite + +Author: Lucas Bates - luc...@mojatatu.com + +tdc is a Python script to load tc unit tests from a separate JSON file and +execute them inside a network namespace dedicated to the task. + + +REQUIREMENTS + + +* Minimum Python version of 3.4. Earlier 3.X versions may work but are not + guaranteed. + +* The kernel must have network namespace support + +* The kernel must have veth support available, as a veth pair is created + prior to running the tests. + +* All tc-related features must be built in or available as modules. + To check what is required in current setup run: + ./tdc.py -c + + Note: + In the current release, tdc run will abort due to a failure in setup or + teardown commands - which includes not being able to run a test simply + because the kernel did not support a specific feature. (This will be + handled in a future version - the current workaround is to run the tests + on specific test categories that your kernel supports) + + +BEFORE YOU RUN +-- + +The path to the tc executable that will be most commonly tested can be defined +in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and +define the path. + +If you need to test a different tc executable on the fly, you can do so by +using the -p option when running tdc: + ./tdc.py -p /path/to/tc + + +RUNNING TDC +--- + +To use tdc, root privileges are required. tdc will not run otherwise. + +All tests are executed inside a network namespace to prevent conflicts +within the host. + +Running tdc without any arguments will run all tests. Refer to the section +on command line arguments for more information, or run: + ./tdc.py -h + +tdc will list the test names as they are being run, and print a summary in +TAP (Test Anything Protocol) format when they are done. If tests fail, +output captured from the failing test will be printed immediately following +the failed test in the TAP output. + + +USER-DEFINED CONSTANTS +-- + +The tdc_config.py file contains multiple values that can be altered to suit +your needs. Any value in the NAMES dictionary can be altered without affecting +the tests to be run. These values are used in the tc commands that will be +executed as part of t
[PATCH net-next 0/1] Introduction of the tc tests
Apologies for sending this as one big patch. I've been sitting on this a little too long, but it's ready and I wanted to get it out. There are a limited number of tests to start - I plan to add more on a regular basis. Lucas Bates (1): selftests: Introduce tc testsuite tools/testing/selftests/tc-testing/.gitignore |1 + tools/testing/selftests/tc-testing/README | 102 ++ tools/testing/selftests/tc-testing/TODO.txt| 10 + .../creating-testcases/AddingTestCases.txt | 69 ++ .../tc-testing/creating-testcases/template.json| 40 + .../tc-testing/tc-tests/actions/tests.json | 1115 .../tc-testing/tc-tests/filters/tests.json | 21 + tools/testing/selftests/tc-testing/tdc.py | 413 tools/testing/selftests/tc-testing/tdc_config.py | 17 + tools/testing/selftests/tc-testing/tdc_helper.py | 75 ++ 10 files changed, 1863 insertions(+) create mode 100644 tools/testing/selftests/tc-testing/.gitignore create mode 100644 tools/testing/selftests/tc-testing/README create mode 100644 tools/testing/selftests/tc-testing/TODO.txt create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/AddingTestCases.txt create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/template.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/actions/tests.json create mode 100644 tools/testing/selftests/tc-testing/tc-tests/filters/tests.json create mode 100755 tools/testing/selftests/tc-testing/tdc.py create mode 100644 tools/testing/selftests/tc-testing/tdc_config.py create mode 100644 tools/testing/selftests/tc-testing/tdc_helper.py -- 2.7.4
[PATCH iproute2 1/1] man page: add page for skbmod action
Signed-off-by: Lucas Bates
Signed-off-by: Jamal Hadi Salim
Signed-off-by: Roman Mashak
---
man/man8/Makefile| 2 +-
man/man8/tc-skbmod.8 | 137 +++
2 files changed, 138 insertions(+), 1 deletion(-)
create mode 100644 man/man8/tc-skbmod.8
diff --git a/man/man8/Makefile b/man/man8/Makefile
index bc2fc81..1bd2f02 100644
--- a/man/man8/Makefile
+++ b/man/man8/Makefile
@@ -16,7 +16,7 @@ MAN8PAGES = $(TARGETS) ip.8 arpd.8 lnstat.8 routel.8 rtacct.8
rtmon.8 rtpr.8 ss.
tc-basic.8 tc-cgroup.8 tc-flow.8 tc-flower.8 tc-fw.8 tc-route.8 \
tc-tcindex.8 tc-u32.8 tc-matchall.8 \
tc-connmark.8 tc-csum.8 tc-mirred.8 tc-nat.8 tc-pedit.8 tc-police.8 \
- tc-simple.8 tc-skbedit.8 tc-vlan.8 tc-xt.8 tc-ife.8 \
+ tc-simple.8 tc-skbedit.8 tc-vlan.8 tc-xt.8 tc-skbmod.8 tc-ife.8 \
tc-tunnel_key.8 \
devlink.8 devlink-dev.8 devlink-monitor.8 devlink-port.8 devlink-sb.8 \
ifstat.8
diff --git a/man/man8/tc-skbmod.8 b/man/man8/tc-skbmod.8
new file mode 100644
index 000..46418b6
--- /dev/null
+++ b/man/man8/tc-skbmod.8
@@ -0,0 +1,137 @@
+.TH "skbmod action in tc" 8 "21 Sep 2016" "iproute2" "Linux"
+
+.SH NAME
+skbmod - user-friendly packet editor action
+.SH SYNOPSIS
+.in +8
+.ti -8
+.BR tc " ... " "action skbmod " "{ [ " "set "
+.IR SETTABLE " ] [ "
+.BI swap " SWAPPABLE"
+.RI " ] [ " CONTROL " ] [ "
+.BI index " INDEX "
+] }
+
+.ti -8
+.IR SETTABLE " := "
+.RB " [ " dmac
+.IR DMAC " ] "
+.RB " [ " smac
+.IR SMAC " ] "
+.RB " [ " etype
+.IR ETYPE " ] "
+
+.ti -8
+.IR SWAPPABLE " := "
+.B mac
+.ti -8
+.IR CONTROL " := {"
+.BR reclassify " | " pipe " | " drop " | " shot " | " continue " | " pass " }"
+.SH DESCRIPTION
+The
+.B skbmod
+action is intended as a usability upgrade to the existing
+.B pedit
+action. Instead of having to manually edit 8-, 16-, or 32-bit chunks of an
+ethernet header,
+.B skbmod
+allows complete substitution of supported elements.
+.SH OPTIONS
+.TP
+.BI dmac " DMAC"
+Change the destination mac to the specified address.
+.TP
+.BI smac " SMAC"
+Change the source mac to the specified address.
+.TP
+.BI etype " ETYPE"
+Change the ethertype to the specified value.
+.TP
+.BI mac
+Used to swap mac addresses. The
+.B swap mac
+directive is performed
+after any outstanding D/SMAC changes.
+.TP
+.I CONTROL
+The following keywords allow to control how the tree of qdisc, classes,
+filters and actions is further traversed after this action.
+.RS
+.TP
+.B reclassify
+Restart with the first filter in the current list.
+.TP
+.B pipe
+Continue with the next action attached to the same filter.
+.TP
+.B drop
+.TQ
+.B shot
+Drop the packet.
+.TP
+.B continue
+Continue classification with the next filter in line.
+.TP
+.B pass
+Finish classification process and return to calling qdisc for further packet
+processing. This is the default.
+.SH EXAMPLES
+To start, observe the following filter with a pedit action:
+
+.RS
+.EX
+tc filter add dev eth1 parent 1: protocol ip prio 10 \\
+ u32 match ip protocol 1 0xff flowid 1:2 \\
+ action pedit munge offset -14 u8 set 0x02 \\
+ munge offset -13 u8 set 0x15 \\
+ munge offset -12 u8 set 0x15 \\
+ munge offset -11 u8 set 0x15 \\
+ munge offset -10 u16 set 0x1515 \\
+ pipe
+.EE
+.RE
+
+Using the skbmod action, this command can be simplified to:
+
+.RS
+.EX
+tc filter add dev eth1 parent 1: protocol ip prio 10 \\
+ u32 match ip protocol 1 0xff flowid 1:2 \\
+ action skbmod set dmac 02:15:15:15:15:15 \\
+ pipe
+.EE
+.RE
+
+Complexity will increase if source mac and ethertype are also being edited
+as part of the action. If all three fields are to be changed with skbmod:
+
+.RS
+.EX
+tc filter add dev eth5 parent 1: protocol ip prio 10 \\
+ u32 match ip protocol 1 0xff flowid 1:2 \\
+ action skbmod \\
+ set etype 0xBEEF \\
+ set dmac 02:12:13:14:15:16 \\
+ set smac 02:22:23:24:25:26
+.EE
+.RE
+
+Finally, swap the destination and source mac addresses in the header:
+
+.RS
+.EX
+tc filter add dev eth3 parent 1: protocol ip prio 10 \\
+ u32 match ip protocol 1 0xff flowid 1:2 \\
+ action skbmod \\
+ swap mac
+.EE
+.RE
+
+As mentioned above, the swap action will occur after any
+.B " smac/dmac "
+substitutions are executed, if they are present.
+
+.SH SEE ALSO
+.BR tc (8),
+.BR tc-u32 (8),
+.BR tc-pedit (8)
--
2.7.4
