Re: [ovs-dev] [RFC PATCH 07/10] python: ovs: flowviz: Add OpenFlow logical view.

2024-02-02 Thread Adrian Moreno 



On 1/30/24 16:54, Eelco Chaudron wrote:

On 1 Dec 2023, at 20:14, Adrian Moreno wrote:


This view is interesting for debugging the logical pipeline. It arranges
the flows in "logical" groups (not to be confused with OVN's
Logical_Flows). A logical group of flows is a set of flows that:
- Have the same table number and priority
- Match on the same fields (regardless of the value they match against)
- Have the same actions, regardless of the arguments for tose actions,
   except for output and recirc, for which arguments do care.

Optionally, the cookie can also be force to be unique for the logical
group. By doing so, we can extend the information we show by querying an
external OVN database and running "ovn-detrace" on each cookie. The
result is a compact list of flow groups with interlieved OVN
information.

Furthermore, if connected to an OVN database, we can apply an OVN
regexp filter.

Examples:
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h
$ export OVN_NB_DB=...
$ export OVN_SB_DB=...
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d
--ovn-filter="acl.*icmp4"


When trying some random trace I got the following:

$ ovs-flowviz -i ~/ofctl.txt -l "drop" --style=dark openflow logic -h

╭─╮
│   
    Filename: 
/home/echaudron/ofctl.txt  
 │
╰─╯
Traceback (most recent call last):
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/bin/ovs-flowviz", line 
20, in 
 main.main()
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/main.py",
 line 196, in main
 maincli()
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1157, in __call__
 return self.main(*args, **kwargs)
^^
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1078, in main
 rv = self.invoke(ctx)
  
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1688, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1688, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1434, in invoke
 return ctx.invoke(self.callback, **ctx.params)
^^^
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 783, in invoke
 return __callback(*args, **kwargs)
^^^
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/decorators.py",
 line 45, in new_func
 return f(get_current_context().obj, *args, **kwargs)
^
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/cli.py",
 line 182, in logic
 processor.print(show_flows)
   File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/logic.py",
 line 203, in print
 self.console.style.set_value_style(
 
AttributeError: 'LogicFlowProcessor' object has no attribute 'console'

This gets solved if I apply the next patch, see the comment in the next patch.



Thanks for doing per-patch testing. I missed that and, since the patch 
organization is artificial (meaning this was developed over many patches in my 
private repo and then artificially squashed) I

Re: [ovs-dev] [RFC PATCH 07/10] python: ovs: flowviz: Add OpenFlow logical view.

2024-01-30 Thread Eelco Chaudron 
On 1 Dec 2023, at 20:14, Adrian Moreno wrote:

> This view is interesting for debugging the logical pipeline. It arranges
> the flows in "logical" groups (not to be confused with OVN's
> Logical_Flows). A logical group of flows is a set of flows that:
> - Have the same table number and priority
> - Match on the same fields (regardless of the value they match against)
> - Have the same actions, regardless of the arguments for tose actions,
>   except for output and recirc, for which arguments do care.
>
> Optionally, the cookie can also be force to be unique for the logical
> group. By doing so, we can extend the information we show by querying an
> external OVN database and running "ovn-detrace" on each cookie. The
> result is a compact list of flow groups with interlieved OVN
> information.
>
> Furthermore, if connected to an OVN database, we can apply an OVN
> regexp filter.
>
> Examples:
> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic
> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h
> $ export OVN_NB_DB=...
> $ export OVN_SB_DB=...
> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d
> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d
> --ovn-filter="acl.*icmp4"

When trying some random trace I got the following:

$ ovs-flowviz -i ~/ofctl.txt -l "drop" --style=dark openflow logic -h

╭─╮
│   
    Filename: 
/home/echaudron/ofctl.txt  
 │
╰─╯
Traceback (most recent call last):
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/bin/ovs-flowviz",
 line 20, in 
main.main()
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/main.py",
 line 196, in main
maincli()
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1157, in __call__
return self.main(*args, **kwargs)
   ^^
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1078, in main
rv = self.invoke(ctx)
 
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
   ^^^
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
   ^^^
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
   ^^^
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py",
 line 783, in invoke
return __callback(*args, **kwargs)
   ^^^
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/decorators.py",
 line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
   ^
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/cli.py",
 line 182, in logic
processor.print(show_flows)
  File 
"/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/logic.py",
 line 203, in print
self.console.style.set_value_style(

AttributeError: 'LogicFlowProcessor' object has no attribute 'console'

This gets solved if I apply the next patch, see the comment in the next patch.

The code itself looks good, I did not test with the actual ovs-detrace tool.

> Signed-off-by: Adrian Moreno 
> ---
>  python/automake.mk  |   4 +-
>  python/ovs/flowviz/ofp/cli.py   | 113 
>

Re: [ovs-dev] [RFC PATCH 07/10] python: ovs: flowviz: Add OpenFlow logical view.

2024-01-30 Thread Eelco Chaudron 



On 30 Jan 2024, at 16:52, Eelco Chaudron wrote:

> On 1 Dec 2023, at 20:14, Adrian Moreno wrote:
>
>> Datapath flows can be arranged into a "tree"-like structure based on
>> recirculation ids, e.g:
>>
>>  recirc(0),eth(...),ipv4(...) actions=ct,recirc(0x42)
>>\-> recirc(42),ct_state(0/0),eth(...),ipv4(...) actions=1
>>\-> recirc(42),ct_state(1/0),eth(...),ipv4(...) actions=userspace(...)
>>
>> This patch adds support for building such logical datapath trees in a
>> format-agnostic way and adds support for console-based formatting
>> supporting:
>> - head-maps formatting of statistics
>> - hash-based pallete of recirculation ids: each recirculation id is
>>   assigned a unique color to easily follow the sequence of related
>>   actions.
>> - full-tree filtering: if a user specifies a filter, an entire subtree
>>   is filtered out if none of its branches satisfy it.
>>
>> Signed-off-by: Adrian Moreno 
>
> This patch looks good to me. One small nit on a comment not ending with a dot.
>
> Acked-by: Eelco Chaudron 
>

Ignore this email, it was supposed to be a reply to patch 6 :(

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [RFC PATCH 07/10] python: ovs: flowviz: Add OpenFlow logical view.

2024-01-30 Thread Eelco Chaudron 
On 1 Dec 2023, at 20:14, Adrian Moreno wrote:

> Datapath flows can be arranged into a "tree"-like structure based on
> recirculation ids, e.g:
>
>  recirc(0),eth(...),ipv4(...) actions=ct,recirc(0x42)
>\-> recirc(42),ct_state(0/0),eth(...),ipv4(...) actions=1
>\-> recirc(42),ct_state(1/0),eth(...),ipv4(...) actions=userspace(...)
>
> This patch adds support for building such logical datapath trees in a
> format-agnostic way and adds support for console-based formatting
> supporting:
> - head-maps formatting of statistics
> - hash-based pallete of recirculation ids: each recirculation id is
>   assigned a unique color to easily follow the sequence of related
>   actions.
> - full-tree filtering: if a user specifies a filter, an entire subtree
>   is filtered out if none of its branches satisfy it.
>
> Signed-off-by: Adrian Moreno 

This patch looks good to me. One small nit on a comment not ending with a dot.

Acked-by: Eelco Chaudron 

> ---
>  python/automake.mk |   1 +
>  python/ovs/flowviz/console.py  |  22 +++
>  python/ovs/flowviz/odp/cli.py  |  21 ++-
>  python/ovs/flowviz/odp/tree.py | 290 +
>  4 files changed, 332 insertions(+), 2 deletions(-)
>  create mode 100644 python/ovs/flowviz/odp/tree.py
>
> diff --git a/python/automake.mk b/python/automake.mk
> index b4c1f84be..5050089e9 100644
> --- a/python/automake.mk
> +++ b/python/automake.mk
> @@ -71,6 +71,7 @@ ovs_flowviz = \
>   python/ovs/flowviz/main.py \
>   python/ovs/flowviz/odp/__init__.py \
>   python/ovs/flowviz/odp/cli.py \
> + python/ovs/flowviz/odp/tree.py \
>   python/ovs/flowviz/ofp/__init__.py \
>   python/ovs/flowviz/ofp/cli.py \
>   python/ovs/flowviz/ofp/html.py \
> diff --git a/python/ovs/flowviz/console.py b/python/ovs/flowviz/console.py
> index 5b4b047c2..2d65f9bb6 100644
> --- a/python/ovs/flowviz/console.py
> +++ b/python/ovs/flowviz/console.py
> @@ -13,6 +13,9 @@
>  # limitations under the License.
>
>  import colorsys
> +import itertools
> +import zlib
> +
>  from rich.console import Console
>  from rich.text import Text
>  from rich.style import Style
> @@ -169,6 +172,25 @@ def heat_pallete(min_value, max_value):
>  return heat
>
>
> +def hash_pallete(hue, saturation, value):
> +"""Generates a color pallete with the cartesian product
> +of the hsv values provided and returns a callable that assigns a color 
> for
> +each value hash
> +"""
> +HSV_tuples = itertools.product(hue, saturation, value)
> +RGB_tuples = map(lambda x: colorsys.hsv_to_rgb(*x), HSV_tuples)
> +styles = [
> +Style(color=Color.from_rgb(r * 255, g * 255, b * 255))
> +for r, g, b in RGB_tuples
> +]
> +
> +def get_style(string):
> +hash_val = zlib.crc32(bytes(str(string), "utf-8"))
> +return styles[hash_val % len(styles)]
> +
> +return get_style
> +
> +
>  def default_highlight():
>  """Generates a default style for highlights."""
>  return Style(underline=True)
> diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py
> index 78f5cfff4..4740e753e 100644
> --- a/python/ovs/flowviz/odp/cli.py
> +++ b/python/ovs/flowviz/odp/cli.py
> @@ -13,12 +13,12 @@
>  # limitations under the License.
>
>  import click
> -
>  from ovs.flowviz.main import maincli
> +from ovs.flowviz.odp.tree import ConsoleTreeProcessor
>  from ovs.flowviz.process import (
>  DatapathFactory,
> -JSONProcessor,
>  ConsoleProcessor,
> +JSONProcessor,
>  )
>
>
> @@ -65,3 +65,20 @@ def console(opts, heat_map):
>  )
>  proc.process()
>  proc.print()
> +
> +
> +@datapath.command()
> +@click.option(
> +"-h",
> +"--heat-map",
> +is_flag=True,
> +default=False,
> +show_default=True,
> +help="Create heat-map with packet and byte counters",
> +)
> +@click.pass_obj
> +def tree(opts, heat_map):
> +"""Print the flows in a tree based on the 'recirc_id'."""
> +processor = ConsoleTreeProcessor(opts)
> +processor.process()
> +processor.print(heat_map)
> diff --git a/python/ovs/flowviz/odp/tree.py b/python/ovs/flowviz/odp/tree.py
> new file mode 100644
> index 0..cfddb162e
> --- /dev/null
> +++ b/python/ovs/flowviz/odp/tree.py
> @@ -0,0 +1,290 @@
> +# Copyright (c) 2023 Red Hat, Inc.
> +#
> +# Licensed under the Apache License, Version 2.0 (the "License");
> +# you may not use this file except in compliance with the License.
> +# You may obtain a copy of the License at:
> +#
> +# http://www.apache.org/licenses/LICENSE-2.0
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS,
> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +# See the License for the specific language governing permissions and
> +# limitations under the License.
> +
> +from rich.style import Style
> +from rich.text import Text
> +from rich.tree import Tree
> +
>

[ovs-dev] [RFC PATCH 07/10] python: ovs: flowviz: Add OpenFlow logical view.

2023-12-01 Thread Adrian Moreno 
This view is interesting for debugging the logical pipeline. It arranges
the flows in "logical" groups (not to be confused with OVN's
Logical_Flows). A logical group of flows is a set of flows that:
- Have the same table number and priority
- Match on the same fields (regardless of the value they match against)
- Have the same actions, regardless of the arguments for those actions,
  except for output and recirc, for which arguments do care.

Optionally, the cookie can also be force to be unique for the logical
group. By doing so, we can extend the information we show by querying an
external OVN database and running "ovn-detrace" on each cookie. The
result is a compact list of flow groups with interlieved OVN
information.

Furthermore, if connected to an OVN database, we can apply an OVN
regexp filter.

Examples:
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h
$ export OVN_NB_DB=...
$ export OVN_SB_DB=...
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d
$ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d
--ovn-filter="acl.*icmp4"

Signed-off-by: Adrian Moreno 
---
 python/automake.mk  |   4 +-
 python/ovs/flowviz/ofp/cli.py   | 113 
 python/ovs/flowviz/ofp/logic.py | 303 
 3 files changed, 418 insertions(+), 2 deletions(-)
 create mode 100644 python/ovs/flowviz/ofp/logic.py

diff --git a/python/automake.mk b/python/automake.mk
index 5050089e9..fdffafbc5 100644
--- a/python/automake.mk
+++ b/python/automake.mk
@@ -74,12 +74,12 @@ ovs_flowviz = \
python/ovs/flowviz/odp/tree.py \
python/ovs/flowviz/ofp/__init__.py \
python/ovs/flowviz/ofp/cli.py \
+   python/ovs/flowviz/ofp/logic.py \
python/ovs/flowviz/ofp/html.py \
python/ovs/flowviz/ovs-flowviz \
python/ovs/flowviz/process.py
 
-# These python files are used at build time but not runtime,
-# so they are not installed.
+# These python files are used at build time but not runtime, so they are not 
installed.
 EXTRA_DIST += \
python/ovs_build_helpers/__init__.py \
python/ovs_build_helpers/extract_ofp_fields.py \
diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py
index 5917a6bf0..6b1435ea1 100644
--- a/python/ovs/flowviz/ofp/cli.py
+++ b/python/ovs/flowviz/ofp/cli.py
@@ -12,10 +12,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import os
+
 import click
 
 from ovs.flowviz.main import maincli
 from ovs.flowviz.ofp.html import HTMLProcessor
+from ovs.flowviz.ofp.logic import LogicFlowProcessor
 from ovs.flowviz.process import (
 OpenFlowFactory,
 JSONProcessor,
@@ -69,6 +72,116 @@ def console(opts, heat_map):
 proc.print()
 
 
+def ovn_detrace_callback(ctx, param, value):
+"""click callback to add detrace information to config object and
+set general ovn-detrace flag to True
+"""
+ctx.obj[param.name] = value
+if value != param.default:
+ctx.obj["ovn_detrace_flag"] = True
+return value
+
+
+@openflow.command()
+@click.option(
+"-d",
+"--ovn-detrace",
+"ovn_detrace_flag",
+is_flag=True,
+show_default=True,
+help="Use ovn-detrace to extract cookie information (implies '-c')",
+)
+@click.option(
+"--ovn-detrace-path",
+default="/usr/bin",
+type=click.Path(),
+help="Use an alternative path to where ovn_detrace.py is located. "
+"Instead of using this option you can just set PYTHONPATH accordingly.",
+show_default=True,
+callback=ovn_detrace_callback,
+)
+@click.option(
+"--ovnnb-db",
+default=os.getenv("OVN_NB_DB") or "unix:/var/run/ovn/ovnnb_db.sock",
+help="Specify the OVN NB database string (implies -d). "
+"If the OVN_NB_DB environment variable is set, it's used as default. "
+"Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock",
+callback=ovn_detrace_callback,
+)
+@click.option(
+"--ovnsb-db",
+default=os.getenv("OVN_SB_DB") or "unix:/var/run/ovn/ovnsb_db.sock",
+help="Specify the OVN NB database string (implies -d). "
+"If the OVN_NB_DB environment variable is set, it's used as default. "
+"Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock",
+callback=ovn_detrace_callback,
+)
+@click.option(
+"-o",
+"--ovn-filter",
+help="Specify a filter to be run on ovn-detrace information (implied -d). "
+"Format: python regular expression "
+"(see https://docs.python.org/3/library/re.html)",
+callback=ovn_detrace_callback,
+)
+@click.option(
+"-s",
+"--show-flows",
+is_flag=True,
+default=False,
+show_default=True,
+help="Show the full flows under each logical flow",
+)
+@click.option(
+"-c",
+"--cookie",
+"cookie_flag",
+is_flag=True,
+default=False,
+show_default=True,
+help="Consider the cookie in the logical flow",
+)