Re: [RFC PATCH v2 0/3] virtio-net: graceful drop of vhost for TAP

[Jason Wang]

在 2021/3/22 下午8:24, Yuri Benditovich 写道:

Allow fallback to userspace only upon migration, only for specific features
and only if 'vhostforce' is not requested.

Changes from v1:
Patch 1 dropeed (will be submitted in another series)
Added device callback in case the migration should fail due to missing features



Hi Yuri:

Have a quick glance at the series. A questions is why we need to do the 
fallback only during load?


I think we should do it in the device initializating. E.g when the vhost 
features can not satisfy, we should disable vhost since there.


Thanks




Yuri Benditovich (3):
   net: add ability to hide (disable) vhost_net
   virtio: introduce 'missing_features_migrated' device callback
   virtio-net: implement missing_features_migrated callback

  hw/net/vhost_net.c |  4 ++-
  hw/net/virtio-net.c| 51 ++
  hw/virtio/virtio.c |  8 ++
  include/hw/virtio/virtio.h |  8 ++
  include/net/net.h  |  1 +
  5 files changed, 71 insertions(+), 1 deletion(-)

Re: [PATCH 10/28] qapi: Rework name checking in preparation of stricter checking

[Markus Armbruster]

John Snow  writes:

> On 3/24/21 1:57 AM, Markus Armbruster wrote:
>> John Snow  writes:
>> 
>>> On 3/23/21 5:40 AM, Markus Armbruster wrote:
 Naming rules differ for the various kinds of names.  To prepare
 enforcing them, define functions to check them: check_name_upper(),
 check_name_lower(), and check_name_camel().  For now, these merely
 wrap around check_name_str(), but that will change shortly.  Replace
 the other uses of check_name_str() by appropriate uses of the
 wrappers.  No change in behavior just yet.
 Signed-off-by: Markus Armbruster 
 ---
scripts/qapi/expr.py | 51 +++-
1 file changed, 36 insertions(+), 15 deletions(-)
 diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
 index e00467636c..30285fe334 100644
 --- a/scripts/qapi/expr.py
 +++ b/scripts/qapi/expr.py
 @@ -21,11 +21,12 @@
from .error import QAPISemError
  -# Names must be letters, numbers, -, and _.  They must start
 with letter,
 -# except for downstream extensions which must start with __RFQDN_.
 -# Dots are only valid in the downstream extension prefix.
 -valid_name = re.compile(r'^(__[a-zA-Z0-9.-]+_)?'
 -'[a-zA-Z][a-zA-Z0-9_-]*$')
 +# Names consist of letters, digits, -, and _, starting with a letter.
 +# An experimental name is prefixed with x-.  A name of a downstream
 +# extension is prefixed with __RFQDN_.  The latter prefix goes first.
 +valid_name = re.compile(r'(__[a-z0-9.-]+_)?'
 +r'(x-)?'
 +r'([a-z][a-z0-9_-]*)$', re.IGNORECASE)
 def check_name_is_str(name, info, source):
 @@ -37,16 +38,38 @@ def check_name_str(name, info, source,
   permit_upper=False):
# Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
# and 'q_obj_*' implicit type names.
 -if not valid_name.match(name) or \
 -   c_name(name, False).startswith('q_'):
 +match = valid_name.match(name)
 +if not match or c_name(name, False).startswith('q_'):
raise QAPISemError(info, "%s has an invalid name" % source)
if not permit_upper and name.lower() != name:
raise QAPISemError(
info, "%s uses uppercase in name" % source)
 +return match.group(3)
 +
 +
 +def check_name_upper(name, info, source):
 +stem = check_name_str(name, info, source, permit_upper=True)
 +# TODO reject '[a-z-]' in @stem
 +
>>>
>>> Creates (presumably) temporary errors in flake8 for the dead
>>> assignment here and below.
>> 
>> All gone by the end of the series.
>> 
>> "make check" and checkpatch were content.  Anything else you'd like me
>> to run?
>
> Eventually it'll be part of CI, with targets to run locally.
>
> I never expected the process to take this long, so I did not invest my
> time in developing an interim solution.
>
> I use a hastily written script to do my own testing, which I run for
> every commit that touches QAPI:
>
> #!/usr/bin/env bash
> set -e
>
> if [[ -f qapi/.flake8 ]]; then
> echo "flake8 --config=qapi/.flake8 qapi/"
> flake8 --config=qapi/.flake8 qapi/
> fi
> if [[ -f qapi/pylintrc ]]; then
> echo "pylint --rcfile=qapi/pylintrc qapi/"
> pylint --rcfile=qapi/pylintrc qapi/
> fi
> if [[ -f qapi/mypy.ini ]]; then
> echo "mypy --config-file=qapi/mypy.ini qapi/"
> mypy --config-file=qapi/mypy.ini qapi/
> fi
>
> if [[ -f qapi/.isort.cfg ]]; then
> pushd qapi
> echo "isort -c ."
> isort -c .
> popd
> fi
>
> pushd ../bin/git
> make -j9
> make check-qapi-schema
> popd

Thanks for sharing this!

Apropos qapi-gen testing scripts.  I have scripts to show me how the
generated code changes along the way in a branch.  They evolved over a
long time, and try to cope with changes in the tree that are hardly
relevant anymore.  By now, they could quite possibly make Frankenstein
recoil in horror.

As a secondary purpose, the scripts show me how output of pycodestyle-3
and pylint change.  This would be uninteresting if the code in master
was clean against a useful configuration of these tools.  Your work has
been making it less interesting.

Re: [PATCH 3/5] tools/vhost-user-i2c: Add backend driver

[Viresh Kumar]

On 25-03-21, 14:17, Jie Deng wrote:
> i2c->adapter_num is set here, but used in vi2c_remove_adapters.
> when you goto out from while {...}, i2c->adapter_num is always 0,
> May be a bug ?

It certainly is, this should fix it:

diff --git a/tools/vhost-user-i2c/main.c b/tools/vhost-user-i2c/main.c
index 071493cbd5c5..65d27ef04d42 100644
--- a/tools/vhost-user-i2c/main.c
+++ b/tools/vhost-user-i2c/main.c
@@ -202,7 +202,7 @@ static void vi2c_remove_adapters(VuI2c *i2c)
 VI2cAdapter *adapter;
 int32_t i;
 
-for (i = 0; i < i2c->adapter_num; i++) {
+for (i = 0; i < MAX_I2C_ADAPTER; i++) {
 adapter = i2c->adapter[i];
 if (!adapter) {
 break;

-- 
viresh

Re: [PATCH 3/5] tools/vhost-user-i2c: Add backend driver

[Jie Deng]

On 2021/3/24 15:33, Viresh Kumar wrote:

+static int vi2c_parse(VuI2c *i2c)
+{
+uint16_t client_addr[MAX_I2C_VDEV];
+int32_t n_adapter = 0, n_client;
+int64_t addr, bus;
+const char *cp, *t;
+
+while (device_list) {
+/* Read :[:] entries one by one */
+cp = strsep(&device_list, ",");
+
+if (!cp || *cp =='\0') {
+break;
+}
+
+if (n_adapter == MAX_I2C_ADAPTER) {
+g_printerr("too many adapter (%d), only support %d \n", n_adapter,
+   MAX_I2C_ADAPTER);
+goto out;
+}
+
+if (qemu_strtol(cp, &t, 10, &bus) || bus < 0) {
+g_printerr("Invalid bus number %s\n", cp);
+goto out;
+}
+
+cp = t;
+n_client = 0;
+
+/* Parse clients [:] entries one by one */
+while (cp != NULL && *cp !='\0') {
+if (*cp == ':')
+cp++;
+
+if (n_client == MAX_I2C_VDEV) {
+g_printerr("too many devices (%d), only support %d \n",
+   n_client, MAX_I2C_VDEV);
+goto out;
+}
+
+if (qemu_strtol(cp, &t, 16, &addr) || addr < 0 || addr > 
MAX_I2C_VDEV) {
+g_printerr("Invalid address %s : %lx\n", cp, addr);
+goto out;
+}
+
+client_addr[n_client++] = addr;
+cp = t;
+if (verbose) {
+g_print("i2c adapter %ld:0x%lx\n", bus, addr);
+}
+}
+
+i2c->adapter[n_adapter] = vi2c_create_adapter(bus, client_addr, 
n_client);
+if (!i2c->adapter[n_adapter])
+goto out;
+n_adapter++;
+}
+
+if (!n_adapter) {
+g_printerr("Failed to add any adapters\n");
+return -1;
+}
+
+i2c->adapter_num = n_adapter;



i2c->adapter_num is set here, but used in vi2c_remove_adapters.
when you goto out from while {...}, i2c->adapter_num is always 0,
May be a bug ?



+
+if (!vi2c_map_adapters(i2c)) {
+return 0;
+}
+
+out:
+vi2c_remove_adapters(i2c);
+return -1;
+}
+

Re: [PATCH v4 4/4] virtio-pci: add support for configure interrupt

[Cindy Lu]

On Wed, Mar 24, 2021 at 2:34 PM Jason Wang  wrote:
>
>
> 在 2021/3/23 上午9:56, Cindy Lu 写道:
> > Add support for configure interrupt, use kvm_irqfd_assign and set the
> > gsi to kernel. When the configure notifier was eventfd_signal by host
> > kernel, this will finally inject an msix interrupt to guest
> >
> > Signed-off-by: Cindy Lu 
> > ---
> >   hw/virtio/virtio-pci.c | 171 +
> >   1 file changed, 137 insertions(+), 34 deletions(-)
> >
> > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > index 36524a5728..b0c190caba 100644
> > --- a/hw/virtio/virtio-pci.c
> > +++ b/hw/virtio/virtio-pci.c
> > @@ -664,7 +664,6 @@ static uint32_t virtio_read_config(PCIDevice *pci_dev,
> >   }
> >
> >   static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,
> > -unsigned int queue_no,
> >   unsigned int vector)
>
>
> Let's use a separated patch for decoupling queue_no from those irqfd
> helpers.
>
> Thanks
>
sure will split this
>
> >   {
> >   VirtIOIRQFD *irqfd = &proxy->vector_irqfd[vector];
> > @@ -691,23 +690,17 @@ static void 
> > kvm_virtio_pci_vq_vector_release(VirtIOPCIProxy *proxy,
> >   }
> >
> >   static int kvm_virtio_pci_irqfd_use(VirtIOPCIProxy *proxy,
> > - unsigned int queue_no,
> > + EventNotifier *n,
> >unsigned int vector)
> >   {
> >   VirtIOIRQFD *irqfd = &proxy->vector_irqfd[vector];
> > -VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
> > -VirtQueue *vq = virtio_get_queue(vdev, queue_no);
> > -EventNotifier *n = virtio_queue_get_guest_notifier(vq);
> >   return kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, n, NULL, 
> > irqfd->virq);
> >   }
> >
> >   static void kvm_virtio_pci_irqfd_release(VirtIOPCIProxy *proxy,
> > -  unsigned int queue_no,
> > +  EventNotifier *n ,
> > unsigned int vector)
> >   {
> > -VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
> > -VirtQueue *vq = virtio_get_queue(vdev, queue_no);
> > -EventNotifier *n = virtio_queue_get_guest_notifier(vq);
> >   VirtIOIRQFD *irqfd = &proxy->vector_irqfd[vector];
> >   int ret;
> >
> > @@ -722,7 +715,8 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy 
> > *proxy, int nvqs)
> >   VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
> >   unsigned int vector;
> >   int ret, queue_no;
> > -
> > +VirtQueue *vq;
> > +EventNotifier *n;
> >   for (queue_no = 0; queue_no < nvqs; queue_no++) {
> >   if (!virtio_queue_get_num(vdev, queue_no)) {
> >   break;
> > @@ -731,7 +725,7 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy 
> > *proxy, int nvqs)
> >   if (vector >= msix_nr_vectors_allocated(dev)) {
> >   continue;
> >   }
> > -ret = kvm_virtio_pci_vq_vector_use(proxy, queue_no, vector);
> > +ret = kvm_virtio_pci_vq_vector_use(proxy,  vector);
> >   if (ret < 0) {
> >   goto undo;
> >   }
> > @@ -739,7 +733,9 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy 
> > *proxy, int nvqs)
> >* Otherwise, delay until unmasked in the frontend.
> >*/
> >   if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
> > -ret = kvm_virtio_pci_irqfd_use(proxy, queue_no, vector);
> > +vq = virtio_get_queue(vdev, queue_no);
> > +n = virtio_queue_get_guest_notifier(vq);
> > +ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
> >   if (ret < 0) {
> >   kvm_virtio_pci_vq_vector_release(proxy, vector);
> >   goto undo;
> > @@ -755,13 +751,69 @@ undo:
> >   continue;
> >   }
> >   if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
> > -kvm_virtio_pci_irqfd_release(proxy, queue_no, vector);
> > +vq = virtio_get_queue(vdev, queue_no);
> > +n = virtio_queue_get_guest_notifier(vq);
> > +kvm_virtio_pci_irqfd_release(proxy, n, vector);
> >   }
> >   kvm_virtio_pci_vq_vector_release(proxy, vector);
> >   }
> >   return ret;
> >   }
> >
> > +static int kvm_virtio_pci_vector_config_use(VirtIOPCIProxy *proxy)
> > +{
> > +
> > +VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
> > +unsigned int vector;
> > +int ret;
> > +EventNotifier *n = virtio_get_config_notifier(vdev);
> > +
> > +vector = vdev->config_vector ;
> > +ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
> > +if (ret < 0) {
> > +goto undo;
> > +}
> > +ret = kvm_virtio_pci_irqfd_use(proxy,  n, vector);
> > +if (ret < 0) {
> > +goto undo;
> > +}
> > +return 0;
> > +undo:
> >

[PATCH v4 17/19] qapi/expr.py: Use tuples instead of lists for static data

[John Snow]

It is -- maybe -- possibly -- three nanoseconds faster.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 

---

This can be dropped if desired; it has no real functional impact I could
defend in code review court. I just happened to write it this way.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index adc5b903bc..b11c11b965 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -210,11 +210,11 @@ def check_flags(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 :param expr: Expression to validate.
 :param info: QAPI source file information.
 """
-for key in ['gen', 'success-response']:
+for key in ('gen', 'success-response'):
 if key in expr and expr[key] is not False:
 raise QAPISemError(
 info, "flag '%s' may only use false value" % key)
-for key in ['boxed', 'allow-oob', 'allow-preconfig', 'coroutine']:
+for key in ('boxed', 'allow-oob', 'allow-preconfig', 'coroutine'):
 if key in expr and expr[key] is not True:
 raise QAPISemError(
 info, "flag '%s' may only use true value" % key)
-- 
2.30.2

[PATCH v4 15/19] qapi/expr.py: enable pylint checks

[John Snow]

Signed-off-by: John Snow 
Tested-by: Eduardo Habkost 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
---
 scripts/qapi/pylintrc | 1 -
 1 file changed, 1 deletion(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index b9e077a164..fb0386d529 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -3,7 +3,6 @@
 # Add files or directories matching the regex patterns to the ignore list.
 # The regex matches against base names, not paths.
 ignore-patterns=error.py,
-expr.py,
 parser.py,
 schema.py,
 
-- 
2.30.2

Re: [PATCH 13/28] qapi: Enforce event naming rules

[Markus Armbruster]

John Snow  writes:

> On 3/24/21 2:22 AM, Markus Armbruster wrote:
>> John Snow  writes:
>> 
>>> On 3/23/21 5:40 AM, Markus Armbruster wrote:
 Event names should be ALL_CAPS with words separated by underscore.
 Enforce this.  The only offenders are in tests/.  Fix them.  Existing
 test event-case covers the new error.
 Signed-off-by: Markus Armbruster 
 ---
tests/unit/test-qmp-event.c   |  6 +++---
scripts/qapi/expr.py  |  4 +++-
tests/qapi-schema/doc-good.json   |  4 ++--
tests/qapi-schema/doc-good.out|  4 ++--
tests/qapi-schema/doc-good.txt|  2 +-
tests/qapi-schema/doc-invalid-return.json |  4 ++--
tests/qapi-schema/event-case.err  |  2 ++
tests/qapi-schema/event-case.json |  2 --
tests/qapi-schema/event-case.out  | 14 --
tests/qapi-schema/qapi-schema-test.json   |  6 +++---
tests/qapi-schema/qapi-schema-test.out|  8 
11 files changed, 22 insertions(+), 34 deletions(-)
 diff --git a/tests/unit/test-qmp-event.c
 b/tests/unit/test-qmp-event.c
 index 047f44ff9a..d58c3b78f2 100644
 --- a/tests/unit/test-qmp-event.c
 +++ b/tests/unit/test-qmp-event.c
 @@ -143,7 +143,7 @@ static void test_event_d(TestEventData *data,
  static void test_event_deprecated(TestEventData *data, const
 void *unused)
{
 -data->expect = qdict_from_jsonf_nofail("{ 'event': 
 'TEST-EVENT-FEATURES1' }");
 +data->expect = qdict_from_jsonf_nofail("{ 'event': 
 'TEST_EVENT_FEATURES1' }");
  memset(&compat_policy, 0, sizeof(compat_policy));
@@ -163,7 +163,7 @@ static void
 test_event_deprecated_data(TestEventData *data, const void *unused)
{
memset(&compat_policy, 0, sizeof(compat_policy));
-data->expect = qdict_from_jsonf_nofail("{ 'event':
 'TEST-EVENT-FEATURES0',"
 +data->expect = qdict_from_jsonf_nofail("{ 'event': 
 'TEST_EVENT_FEATURES0',"
   " 'data': { 'foo': 42 } }");
qapi_event_send_test_event_features0(42);
g_assert(data->emitted);
 @@ -172,7 +172,7 @@ static void test_event_deprecated_data(TestEventData 
 *data, const void *unused)
  compat_policy.has_deprecated_output = true;
compat_policy.deprecated_output = COMPAT_POLICY_OUTPUT_HIDE;
 -data->expect = qdict_from_jsonf_nofail("{ 'event': 
 'TEST-EVENT-FEATURES0' }");
 +data->expect = qdict_from_jsonf_nofail("{ 'event': 
 'TEST_EVENT_FEATURES0' }");
qapi_event_send_test_event_features0(42);
g_assert(data->emitted);
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
 index b5fb0be48b..c065505b27 100644
 --- a/scripts/qapi/expr.py
 +++ b/scripts/qapi/expr.py
 @@ -45,7 +45,9 @@ def check_name_str(name, info, source):
  def check_name_upper(name, info, source):
stem = check_name_str(name, info, source)
 -# TODO reject '[a-z-]' in @stem
 +if re.search(r'[a-z-]', stem):
 +raise QAPISemError(
 +info, "name of %s must not use lowercase or '-'" % source)

>>>
>>> Does a little bit more than check_name_upper. Is this only used for
>>> event names? I guess so. Should it be inlined into check_defn_name_str
>>> instead in this case, or nah?
>> 
>> I'd prefer not to inline.  I'm open to better function names.
>> 
>> We have three name styles.  qapi-code-gen.txt:
>> 
>>  [Type] definitions should always use CamelCase for
>>  user-defined type names, while built-in types are lowercase.
>> 
>>  [...]
>> 
>>  Command names, and member names within a type, should be all lower
>>  case with words separated by a hyphen.  [...]
>> 
>>  Event names should be ALL_CAPS with words separated by underscore.
>> 
>> I define three functions for them: check_name_camel(),
>> check_name_lower(), and check_name_upper().
>> 
>> The functions factor out the naming rule aspect, and they let us keep
>> the naming rule aspect together.  That's why I'd prefer not to inline.
>> 
>> We could name them after their purpose instead:
>> check_name_user_defined_type(), check_name_command_or_member(),
>> check_name_event().  The first two are rather long.  Shorter:
>> check_name_type(), check_name_other(), check_name_event().
>> 
>> Thoughts?
>> 
>
> The long names are nice and descriptive.

Then I should give them a try to see whether the result feels neat or
ugly.

[PATCH v4 19/19] qapi/expr.py: Use an expression checker dispatch table

[John Snow]

This enforces a type signature against all of the top-level expression
check routines without necessarily needing to create some
overcomplicated class hierarchy for them.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 63 +++-
 1 file changed, 33 insertions(+), 30 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index aabbc255d2..c42d061e68 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -31,8 +31,10 @@
 structures and contextual semantic validation.
 """
 
+from enum import Enum
 import re
 from typing import (
+Callable,
 Collection,
 Dict,
 Iterable,
@@ -572,6 +574,29 @@ def check_event(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 check_type(args, info, "'data'", allow_dict=not boxed)
 
 
+class ExpressionType(str, Enum):
+INCLUDE = 'include'
+ENUM = 'enum'
+UNION = 'union'
+ALTERNATE = 'alternate'
+STRUCT = 'struct'
+COMMAND = 'command'
+EVENT = 'event'
+
+def __str__(self) -> str:
+return str(self.value)
+
+
+_CHECK_FN: Dict[str, Callable[[_JSONObject, QAPISourceInfo], None]] = {
+'enum': check_enum,
+'union': check_union,
+'alternate': check_alternate,
+'struct': check_struct,
+'command': check_command,
+'event': check_event,
+}
+
+
 def check_exprs(exprs: List[_JSONObject]) -> List[_JSONObject]:
 """
 Validate and normalize a list of parsed QAPI schema expressions.
@@ -598,24 +623,16 @@ def check_exprs(exprs: List[_JSONObject]) -> 
List[_JSONObject]:
 assert tmp is None or isinstance(tmp, QAPIDoc)
 doc: Optional[QAPIDoc] = tmp
 
-if 'include' in expr:
-continue
-
-if 'enum' in expr:
-meta = 'enum'
-elif 'union' in expr:
-meta = 'union'
-elif 'alternate' in expr:
-meta = 'alternate'
-elif 'struct' in expr:
-meta = 'struct'
-elif 'command' in expr:
-meta = 'command'
-elif 'event' in expr:
-meta = 'event'
+for kind in ExpressionType:
+if kind in expr:
+meta = kind
+break
 else:
 raise QAPISemError(info, "expression is missing metatype")
 
+if meta == ExpressionType.INCLUDE:
+continue
+
 check_name_is_str(expr[meta], info, "'%s'" % meta)
 name = cast(str, expr[meta])
 info.set_defn(meta, name)
@@ -630,21 +647,7 @@ def check_exprs(exprs: List[_JSONObject]) -> 
List[_JSONObject]:
 raise QAPISemError(info,
"documentation comment required")
 
-if meta == 'enum':
-check_enum(expr, info)
-elif meta == 'union':
-check_union(expr, info)
-elif meta == 'alternate':
-check_alternate(expr, info)
-elif meta == 'struct':
-check_struct(expr, info)
-elif meta == 'command':
-check_command(expr, info)
-elif meta == 'event':
-check_event(expr, info)
-else:
-assert False, 'unexpected meta type'
-
+_CHECK_FN[meta](expr, info)
 check_if(expr, info, meta)
 check_features(expr.get('features'), info)
 check_flags(expr, info)
-- 
2.30.2

[PATCH v4 12/19] qapi/expr.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py  | 68 +++
 scripts/qapi/mypy.ini |  5 
 2 files changed, 43 insertions(+), 30 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 7e22723b50..ea9d39fcf2 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -15,7 +15,15 @@
 # See the COPYING file in the top-level directory.
 
 import re
-from typing import Dict, Optional, cast
+from typing import (
+Collection,
+Dict,
+Iterable,
+List,
+Optional,
+Union,
+cast,
+)
 
 from .common import c_name
 from .error import QAPISemError
@@ -37,12 +45,14 @@
 r'([a-z][a-z0-9_-]*)$', re.IGNORECASE)
 
 
-def check_name_is_str(name, info, source):
+def check_name_is_str(name: object,
+  info: QAPISourceInfo,
+  source: str) -> None:
 if not isinstance(name, str):
 raise QAPISemError(info, "%s requires a string name" % source)
 
 
-def check_name_str(name, info, source):
+def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str:
 # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
 # and 'q_obj_*' implicit type names.
 match = valid_name.match(name)
@@ -51,16 +61,16 @@ def check_name_str(name, info, source):
 return match.group(3)
 
 
-def check_name_upper(name, info, source):
+def check_name_upper(name: str, info: QAPISourceInfo, source: str) -> None:
 stem = check_name_str(name, info, source)
 if re.search(r'[a-z-]', stem):
 raise QAPISemError(
 info, "name of %s must not use lowercase or '-'" % source)
 
 
-def check_name_lower(name, info, source,
- permit_upper=False,
- permit_underscore=False):
+def check_name_lower(name: str, info: QAPISourceInfo, source: str,
+ permit_upper: bool = False,
+ permit_underscore: bool = False) -> None:
 stem = check_name_str(name, info, source)
 if ((not permit_upper and re.search(r'[A-Z]', stem))
 or (not permit_underscore and '_' in stem)):
@@ -68,13 +78,13 @@ def check_name_lower(name, info, source,
 info, "name of %s must not use uppercase or '_'" % source)
 
 
-def check_name_camel(name, info, source):
+def check_name_camel(name: str, info: QAPISourceInfo, source: str) -> None:
 stem = check_name_str(name, info, source)
 if not re.match(r'[A-Z][A-Za-z0-9]*[a-z][A-Za-z0-9]*$', stem):
 raise QAPISemError(info, "name of %s must use CamelCase" % source)
 
 
-def check_defn_name_str(name, info, meta):
+def check_defn_name_str(name: str, info: QAPISourceInfo, meta: str) -> None:
 if meta == 'event':
 check_name_upper(name, info, meta)
 elif meta == 'command':
@@ -88,9 +98,13 @@ def check_defn_name_str(name, info, meta):
 info, "%s name should not end in '%s'" % (meta, name[-4:]))
 
 
-def check_keys(value, info, source, required, optional):
+def check_keys(value: _JSONObject,
+   info: QAPISourceInfo,
+   source: str,
+   required: Collection[str],
+   optional: Collection[str]) -> None:
 
-def pprint(elems):
+def pprint(elems: Iterable[str]) -> str:
 return ', '.join("'" + e + "'" for e in sorted(elems))
 
 missing = set(required) - set(value)
@@ -110,7 +124,7 @@ def pprint(elems):
pprint(unknown), pprint(allowed)))
 
 
-def check_flags(expr, info):
+def check_flags(expr: _JSONObject, info: QAPISourceInfo) -> None:
 for key in ['gen', 'success-response']:
 if key in expr and expr[key] is not False:
 raise QAPISemError(
@@ -128,9 +142,9 @@ def check_flags(expr, info):
  "are incompatible")
 
 
-def check_if(expr, info, source):
+def check_if(expr: _JSONObject, info: QAPISourceInfo, source: str) -> None:
 
-def check_if_str(ifcond):
+def check_if_str(ifcond: object) -> None:
 if not isinstance(ifcond, str):
 raise QAPISemError(
 info,
@@ -156,7 +170,7 @@ def check_if_str(ifcond):
 expr['if'] = [ifcond]
 
 
-def normalize_members(members):
+def normalize_members(members: object) -> None:
 if isinstance(members, dict):
 for key, arg in members.items():
 if isinstance(arg, dict):
@@ -164,8 +178,11 @@ def normalize_members(members):
 members[key] = {'type': arg}
 
 
-def check_type(value, info, source,
-   allow_array=False, allow_dict=False):
+def check_type(value: Optional[object],
+   info: QAPISourceInfo,
+   source: str,
+   allow_array: bool = False,
+   allow_dict: Union[bool, str] = False) -> None:
 if value is None:
 return
 
@@ -212,7 +229,8 @@ def check_type(value, info, source,
 check_type(a

[PATCH v4 16/19] qapi/expr.py: Add docstrings

[John Snow]

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 213 ++-
 1 file changed, 208 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 1869ddf815..adc5b903bc 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
 #
-# Check (context-free) QAPI schema expression structure
-#
 # Copyright IBM, Corp. 2011
 # Copyright (c) 2013-2019 Red Hat Inc.
 #
@@ -14,6 +12,25 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+"""
+Normalize and validate (context-free) QAPI schema expression structures.
+
+After QAPI expressions are parsed from disk, they are stored in
+recursively nested Python data structures using Dict, List, str, bool,
+and int. This module ensures that those nested structures have the
+correct type(s) and key(s) where appropriate for the QAPI context-free
+grammar.
+
+The QAPI schema expression language allows for syntactic sugar; this
+module also handles the normalization process of these nested
+structures.
+
+See `check_exprs` for the main entry point.
+
+See `schema.QAPISchema` for processing into native Python data
+structures and contextual semantic validation.
+"""
+
 import re
 from typing import (
 Collection,
@@ -31,9 +48,10 @@
 from .source import QAPISourceInfo
 
 
-# Deserialized JSON objects as returned by the parser;
-# The values of this mapping are not necessary to exhaustively type
-# here, because the purpose of this module is to interrogate that type.
+#: Deserialized JSON objects as returned by the parser.
+#:
+#: The values of this mapping are not necessary to exhaustively type
+#: here, because the purpose of this module is to interrogate that type.
 _JSONObject = Dict[str, object]
 
 
@@ -48,11 +66,29 @@
 def check_name_is_str(name: object,
   info: QAPISourceInfo,
   source: str) -> None:
+"""Ensures that ``name`` is a string."""
 if not isinstance(name, str):
 raise QAPISemError(info, "%s requires a string name" % source)
 
 
 def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str:
+"""
+Ensures a string is a legal name.
+
+A legal name consists of ascii letters, digits, ``-``, and ``_``,
+starting with a letter. The names of downstream extensions are
+prefixed with an __com.example_ style prefix, allowing ``.`` and
+``-``.  An experimental name is prefixed with ``x-``, following the
+RFQDN if present.
+
+A legal name cannot start with ``q_``, which is reserved.
+
+:param name:   Name to check.
+:param info:   QAPI source file information.
+:param source: Human-readable str describing "what" this name is.
+
+:return: The stem of the valid name, with no prefixes.
+"""
 # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
 # and 'q_obj_*' implicit type names.
 match = valid_name.match(name)
@@ -62,6 +98,12 @@ def check_name_str(name: str, info: QAPISourceInfo, source: 
str) -> str:
 
 
 def check_name_upper(name: str, info: QAPISourceInfo, source: str) -> None:
+"""
+Ensures a string is a legal event name.
+
+Checks the same criteria as `check_name_str`, but requires uppercase
+and prohibits ``-``.
+"""
 stem = check_name_str(name, info, source)
 if re.search(r'[a-z-]', stem):
 raise QAPISemError(
@@ -71,6 +113,15 @@ def check_name_upper(name: str, info: QAPISourceInfo, 
source: str) -> None:
 def check_name_lower(name: str, info: QAPISourceInfo, source: str,
  permit_upper: bool = False,
  permit_underscore: bool = False) -> None:
+"""
+Ensures a string is a legal user defined type name.
+
+Checks the same criteria as `check_name_str`, but may impose
+additional constraints.
+
+:param permit_upper: Prohibits uppercase when false.
+:param permit_underscore: Prohibits underscores when false.
+"""
 stem = check_name_str(name, info, source)
 if ((not permit_upper and re.search(r'[A-Z]', stem))
 or (not permit_underscore and '_' in stem)):
@@ -79,12 +130,31 @@ def check_name_lower(name: str, info: QAPISourceInfo, 
source: str,
 
 
 def check_name_camel(name: str, info: QAPISourceInfo, source: str) -> None:
+"""
+Ensures a string is a legal CamelCase name.
+
+Checks the same criteria as `check_name_str`,
+but additionally imposes a CamelCase constraint.
+"""
 stem = check_name_str(name, info, source)
 if not re.match(r'[A-Z][A-Za-z0-9]*[a-z][A-Za-z0-9]*$', stem):
 raise QAPISemError(info, "name of %s must use CamelCase" % source)
 
 
 def check_defn_name_str(name: str, info: QAPISourceInfo, meta: str) -> None:
+"""
+Ensures a name is a legal definition name.
+
+- 'event' names adhere to `check_name_upper`.
+- 'command' names adhere to `check_name_lower`.

[PATCH v4 14/19] qapi/expr.py: Remove single-letter variable

[John Snow]

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 5921fa34ab..1869ddf815 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -233,14 +233,14 @@ def check_features(features: Optional[object],
 raise QAPISemError(info, "'features' must be an array")
 features[:] = [f if isinstance(f, dict) else {'name': f}
for f in features]
-for f in features:
+for feat in features:
 source = "'features' member"
-assert isinstance(f, dict)
-check_keys(f, info, source, ['name'], ['if'])
-check_name_is_str(f['name'], info, source)
-source = "%s '%s'" % (source, f['name'])
-check_name_lower(f['name'], info, source)
-check_if(f, info, source)
+assert isinstance(feat, dict)
+check_keys(feat, info, source, ['name'], ['if'])
+check_name_is_str(feat['name'], info, source)
+source = "%s '%s'" % (source, feat['name'])
+check_name_str(feat['name'], info, source)
+check_if(feat, info, source)
 
 
 def check_enum(expr: _JSONObject, info: QAPISourceInfo) -> None:
-- 
2.30.2

[PATCH v4 05/19] qapi/expr.py: constrain incoming expression types

[John Snow]

mypy does not know the types of values stored in Dicts that masquerade
as objects. Help the type checker out by constraining the type.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 26 +++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index b4bbcd54c0..b75c85c160 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -15,9 +15,18 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Dict, Optional
 
 from .common import c_name
 from .error import QAPISemError
+from .parser import QAPIDoc
+from .source import QAPISourceInfo
+
+
+# Deserialized JSON objects as returned by the parser;
+# The values of this mapping are not necessary to exhaustively type
+# here, because the purpose of this module is to interrogate that type.
+_JSONObject = Dict[str, object]
 
 
 # Names consist of letters, digits, -, and _, starting with a letter.
@@ -315,9 +324,20 @@ def check_event(expr, info):
 
 def check_exprs(exprs):
 for expr_elem in exprs:
-expr = expr_elem['expr']
-info = expr_elem['info']
-doc = expr_elem.get('doc')
+# Expression
+assert isinstance(expr_elem['expr'], dict)
+for key in expr_elem['expr'].keys():
+assert isinstance(key, str)
+expr: _JSONObject = expr_elem['expr']
+
+# QAPISourceInfo
+assert isinstance(expr_elem['info'], QAPISourceInfo)
+info: QAPISourceInfo = expr_elem['info']
+
+# Optional[QAPIDoc]
+tmp = expr_elem.get('doc')
+assert tmp is None or isinstance(tmp, QAPIDoc)
+doc: Optional[QAPIDoc] = tmp
 
 if 'include' in expr:
 continue
-- 
2.30.2

[PATCH v4 10/19] qapi/expr.py: Add casts in a few select cases

[John Snow]

Casts are instructions to the type checker only, they aren't "safe" and
should probably be avoided in general. In this case, when we perform
type checking on a nested structure, the type of each field does not
"stick".

(See PEP 647 for an example of "type narrowing" that does "stick".
 It is available in Python 3.10, so we can't use it yet.)

We don't need to assert that something is a str if we've already checked
or asserted that it is -- use a cast instead for these cases.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index ca5ab7bfda..505e67bd21 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -15,7 +15,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
-from typing import Dict, Optional
+from typing import Dict, Optional, cast
 
 from .common import c_name
 from .error import QAPISemError
@@ -259,7 +259,7 @@ def check_enum(expr, info):
 
 
 def check_struct(expr, info):
-name = expr['struct']
+name = cast(str, expr['struct'])  # Asserted in check_exprs
 members = expr['data']
 
 check_type(members, info, "'data'", allow_dict=name)
@@ -267,7 +267,7 @@ def check_struct(expr, info):
 
 
 def check_union(expr, info):
-name = expr['union']
+name = cast(str, expr['union'])  # Asserted in check_exprs
 base = expr.get('base')
 discriminator = expr.get('discriminator')
 members = expr['data']
@@ -366,8 +366,8 @@ def check_exprs(exprs):
 else:
 raise QAPISemError(info, "expression is missing metatype")
 
-name = expr[meta]
-check_name_is_str(name, info, "'%s'" % meta)
+check_name_is_str(expr[meta], info, "'%s'" % meta)
+name = cast(str, expr[meta])
 info.set_defn(meta, name)
 check_defn_name_str(name, info, meta)
 
-- 
2.30.2

[PATCH v4 13/19] qapi/expr.py: Consolidate check_if_str calls in check_if

[John Snow]

This is a small rewrite to address some minor style nits.

Don't compare against the empty list to check for the empty condition, and
move the normalization forward to unify the check on the now-normalized
structure.

With the check unified, the local nested function isn't needed anymore
and can be brought down into the normal flow of the function. With the
nesting level changed, shuffle the error strings around a bit to get
them to fit in 79 columns.

Note: although ifcond is typed as Sequence[str] elsewhere, we *know* that
the parser will produce real, bona-fide lists. It's okay to check
isinstance(ifcond, list) here.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 32 ++--
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index ea9d39fcf2..5921fa34ab 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -144,30 +144,26 @@ def check_flags(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 
 def check_if(expr: _JSONObject, info: QAPISourceInfo, source: str) -> None:
 
-def check_if_str(ifcond: object) -> None:
-if not isinstance(ifcond, str):
-raise QAPISemError(
-info,
-"'if' condition of %s must be a string or a list of strings"
-% source)
-if ifcond.strip() == '':
-raise QAPISemError(
-info,
-"'if' condition '%s' of %s makes no sense"
-% (ifcond, source))
-
 ifcond = expr.get('if')
 if ifcond is None:
 return
+
 if isinstance(ifcond, list):
-if ifcond == []:
+if not ifcond:
 raise QAPISemError(
-info, "'if' condition [] of %s is useless" % source)
-for elt in ifcond:
-check_if_str(elt)
+info, f"'if' condition [] of {source} is useless")
 else:
-check_if_str(ifcond)
-expr['if'] = [ifcond]
+# Normalize to a list
+ifcond = expr['if'] = [ifcond]
+
+for elt in ifcond:
+if not isinstance(elt, str):
+raise QAPISemError(info, (
+f"'if' condition of {source}"
+" must be a string or a list of strings"))
+if not elt.strip():
+raise QAPISemError(
+info, f"'if' condition '{elt}' of {source} makes no sense")
 
 
 def normalize_members(members: object) -> None:
-- 
2.30.2

Re: [PATCH V4] file-posix: allow -EBUSY error during ioctl(fd, BLKZEROOUT, range) on block

[ChangLimin]

>On Wed, Mar 24, 2021 at 4:52 PM Max Reitz  wrote:
>On 22.03.21 10:25, ChangLimin wrote:
>> For Linux 5.10/5.11, qemu write zeros to a multipath device using
>> ioctl(fd, BLKZEROOUT, range) with cache none or directsync return -EBUSY
>> permanently.
>
>So as far as I can track back the discussion, Kevin asked on v1 why we’d 
>set has_write_zeroes to false, i.e. whether the EBUSY might not go away 
>at some point, and if it did, whether we shouldn’t retry BLKZEROOUT then.
>You haven’t explicitly replied to that question (as far as I can see), 
>so it kind of still stands.
>
>Implicitly, there are two conflicting answers in this patch: On one 
>hand, the commit message says “permanently”, and this is what you told 
>Nir as a realistic case where this can occur. 

For Linux 5.10/5.11, the EBUSY is permanently, the reproduce step is below. 
For other Linux version, the EBUSY may be temporary. 
Because  Linux 5.10/5.11 is not used widely, so do not set has_write_zeroes to 
false.

>I'm afraid ChangLimin did not answer my question. I'm looking for real
>world used case when qemu cannot write zeros to multipath device, when
>nobody else is using the device.
>
>I tried to reproduce this on Fedora (kernel 5.10) with qemu-img convert,
>once with a multipath device, and once with logical volume on a vg created
>on the multipath device, and I could not reproduce this issue.

The following is steps to reproduct the issue on Fedora 34.

# uname -a
Linux fedora-34 5.11.3-300.fc34.x86_64 #1 SMP Thu Mar 4 19:03:18 UTC 2021 
x86_64 x86_64 x86_64 GNU/Linux

# qemu-img -V
qemu-img version 5.2.0 (qemu-5.2.0-5.fc34.1)

1.  Login in an ISCSI LUN created using targetcli on ubuntu 20.04
# iscsiadm -m discovery -t st -p 192.169.1.109
192.169.1.109:3260,1 iqn.2003-01.org.linux-iscsi:lio-lv100

# iscsiadm -m node -l -T iqn.2003-01.org.linux-iscsi:lio-lv100
# iscsiadm -m session
tcp: [1] 192.169.1.109:3260,1 iqn.2003-01.org.linux-iscsi:lio-lv100 (non-flash)

2. start multipathd service
# mpathconf --enable
# systemctl start multipathd

3.  add multipath path
# multipath -a `/lib/udev/scsi_id -g /dev/sdb`   # sdb means the ISCSI LUN
wwid '36001405b76856e4816b48b99c6a77de3' added

# multipathd add path /dev/sdb
ok

# multipath -ll # /dev/dm-1 is the multipath device based on /dev/sdb
mpatha (36001405bebfc3a0522541cda30220db9) dm-1 LIO-ORG,lv102
size=1.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  `- 5:0:0:0  sdd  8:48   active ready running

4. qemu-img return EBUSY both to dm-1 and sdb
# wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
# qemu-img convert -O raw -t none cirros-0.4.0-x86_64-disk.img /dev/dm-1
qemu-img: error while writing at byte 0: Device or resource busy

# qemu-img convert -O raw -t none cirros-0.4.0-x86_64-disk.img /dev/sdb
qemu-img: error while writing at byte 0: Device or resource busy

5. blkdiscard also return EBUSY  both to dm-1 and sdb
# blkdiscard -o 0 -l 4096 /dev/dm-1
blkdiscard: cannot open /dev/dm-1: Device or resource busy

# blkdiscard -o 0 -l 4096 /dev/sdb
blkdiscard: cannot open /dev/sdb: No such file or directory

6. dd write zero is good, because it does not use blkdiscard
# dd if=/dev/zero of=/dev/dm-1 bs=1M count=100 oflag=direct 
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.33623 s, 44.9 MB/s

7. The LUN should support blkdiscard feature, otherwise it will not write zero 
with  ioctl(fd, BLKZEROOUT, range) 

>If I understand the kernel change correctly, this can happen when there is
>a mounted file system on top of the multipath device. I don't think we have 
>a use case when qemu accesses a multipath device when the device is used
>by a file system, but maybe I missed something.
> 
>So that to me implies 
>that we actually should not retry BLKZEROOUT, because the EBUSY will 
>remain, and that condition won’t change while the block device is in use 
>by qemu.
>
>On the other hand, in the code, you have decided not to reset 
>has_write_zeroes to false, so the implementation will retry.
>
>EBUSY is usually a temporary error, so retrying makes sense. The question
>is if we really can write zeroes manually in this case?
> 
>So I don’t quite understand.  Should we keep trying BLKZEROOUT or is 
>there no chance of it working after it has at one point failed with 
>EBUSY?  (Are there other cases besides what’s described in this commit 
>message where EBUSY might be returned and it is only temporary?)
>
>> Fallback to pwritev instead of exit for -EBUSY error.
>> 
>> The issue was introduced in Linux 5.10:
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=384d87ef2c954fc58e6c5fd8253e4a1984f5fe02
>> 
>> Fixed in Linux 5.12:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=56887cffe946bb0a90c74429fa94d6110a73119d
>> 
>> Signed-off-by: ChangLimin 
>> ---
>>   block/file-posix.c | 8 ++--
>>   1 file changed, 6 insertions(+), 2 deletions(-

[PATCH v4 08/19] qapi: add tests for invalid 'data' field type

[John Snow]

It needs to be an object (dict), not anything else.

Signed-off-by: John Snow 

---

Note: this actually doesn't ... work, but on-list, we discussed wanting
tests first, then the fix. That can't happen here, because QAPI crashes
at runtime. So uh, just squash this into the following patch, I guess?

I tried.

Signed-off-by: John Snow 
---
 tests/qapi-schema/alternate-invalid-data-type.err  |  0
 tests/qapi-schema/alternate-invalid-data-type.json |  4 
 tests/qapi-schema/alternate-invalid-data-type.out  |  0
 tests/qapi-schema/meson.build  |  2 ++
 tests/qapi-schema/union-invalid-data-type.err  |  0
 tests/qapi-schema/union-invalid-data-type.json | 13 +
 tests/qapi-schema/union-invalid-data-type.out  |  0
 7 files changed, 19 insertions(+)
 create mode 100644 tests/qapi-schema/alternate-invalid-data-type.err
 create mode 100644 tests/qapi-schema/alternate-invalid-data-type.json
 create mode 100644 tests/qapi-schema/alternate-invalid-data-type.out
 create mode 100644 tests/qapi-schema/union-invalid-data-type.err
 create mode 100644 tests/qapi-schema/union-invalid-data-type.json
 create mode 100644 tests/qapi-schema/union-invalid-data-type.out

diff --git a/tests/qapi-schema/alternate-invalid-data-type.err 
b/tests/qapi-schema/alternate-invalid-data-type.err
new file mode 100644
index 00..e69de29bb2
diff --git a/tests/qapi-schema/alternate-invalid-data-type.json 
b/tests/qapi-schema/alternate-invalid-data-type.json
new file mode 100644
index 00..7d5d905581
--- /dev/null
+++ b/tests/qapi-schema/alternate-invalid-data-type.json
@@ -0,0 +1,4 @@
+# Alternate type requires an object for 'data'
+{ 'alternate': 'Alt',
+  'data': ['rubbish', 'nonsense']
+}
diff --git a/tests/qapi-schema/alternate-invalid-data-type.out 
b/tests/qapi-schema/alternate-invalid-data-type.out
new file mode 100644
index 00..e69de29bb2
diff --git a/tests/qapi-schema/meson.build b/tests/qapi-schema/meson.build
index 8ba6917132..cc5b136cfb 100644
--- a/tests/qapi-schema/meson.build
+++ b/tests/qapi-schema/meson.build
@@ -15,6 +15,7 @@ schemas = [
   'alternate-conflict-bool-string.json',
   'alternate-conflict-num-string.json',
   'alternate-empty.json',
+  'alternate-invalid-data-type.json',
   'alternate-invalid-dict.json',
   'alternate-nested.json',
   'alternate-unknown.json',
@@ -192,6 +193,7 @@ schemas = [
   'union-clash-branches.json',
   'union-empty.json',
   'union-invalid-base.json',
+  'union-invalid-data-type.json',
   'union-optional-branch.json',
   'union-unknown.json',
   'unknown-escape.json',
diff --git a/tests/qapi-schema/union-invalid-data-type.err 
b/tests/qapi-schema/union-invalid-data-type.err
new file mode 100644
index 00..e69de29bb2
diff --git a/tests/qapi-schema/union-invalid-data-type.json 
b/tests/qapi-schema/union-invalid-data-type.json
new file mode 100644
index 00..5a32d267bf
--- /dev/null
+++ b/tests/qapi-schema/union-invalid-data-type.json
@@ -0,0 +1,13 @@
+# the union data type must be an object.
+
+{ 'struct': 'TestTypeA',
+  'data': { 'string': 'str' } }
+
+{ 'struct': 'TestTypeB',
+  'data': { 'integer': 'int' } }
+
+{ 'union': 'TestUnion',
+  'base': 'int',
+  'discriminator': 'int',
+  'data': ['TestTypeA', 'TestTypeB']
+}
diff --git a/tests/qapi-schema/union-invalid-data-type.out 
b/tests/qapi-schema/union-invalid-data-type.out
new file mode 100644
index 00..e69de29bb2
-- 
2.30.2

[PATCH v4 03/19] qapi/expr.py: Remove 'info' argument from nested check_if_str

[John Snow]

The function can just use the argument from the scope above. Otherwise,
we get shadowed argument errors because the parameter name clashes with
the name of a variable already in-scope.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
---
 scripts/qapi/expr.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index c207481f7e..3fda5d5082 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -122,7 +122,7 @@ def check_flags(expr, info):
 
 def check_if(expr, info, source):
 
-def check_if_str(ifcond, info):
+def check_if_str(ifcond):
 if not isinstance(ifcond, str):
 raise QAPISemError(
 info,
@@ -142,9 +142,9 @@ def check_if_str(ifcond, info):
 raise QAPISemError(
 info, "'if' condition [] of %s is useless" % source)
 for elt in ifcond:
-check_if_str(elt, info)
+check_if_str(elt)
 else:
-check_if_str(ifcond, info)
+check_if_str(ifcond)
 expr['if'] = [ifcond]
 
 
-- 
2.30.2

[PATCH v4 09/19] qapi/expr.py: Check type of 'data' member

[John Snow]

We don't actually check, so the user can get some unpleasant stacktraces.
Formalize it.

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py  | 7 +++
 tests/qapi-schema/alternate-invalid-data-type.err | 2 ++
 tests/qapi-schema/union-invalid-data-type.err | 2 ++
 3 files changed, 11 insertions(+)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 73e7d8cb0d..ca5ab7bfda 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -281,6 +281,9 @@ def check_union(expr, info):
 raise QAPISemError(info, "'discriminator' requires 'base'")
 check_name_is_str(discriminator, info, "'discriminator'")
 
+if not isinstance(members, dict):
+raise QAPISemError(info, "'data' must be an object")
+
 for (key, value) in members.items():
 source = "'data' member '%s'" % key
 if discriminator is None:
@@ -296,6 +299,10 @@ def check_alternate(expr, info):
 
 if not members:
 raise QAPISemError(info, "'data' must not be empty")
+
+if not isinstance(members, dict):
+raise QAPISemError(info, "'data' must be an object")
+
 for (key, value) in members.items():
 source = "'data' member '%s'" % key
 check_name_lower(key, info, source)
diff --git a/tests/qapi-schema/alternate-invalid-data-type.err 
b/tests/qapi-schema/alternate-invalid-data-type.err
index e69de29bb2..c7301ccb00 100644
--- a/tests/qapi-schema/alternate-invalid-data-type.err
+++ b/tests/qapi-schema/alternate-invalid-data-type.err
@@ -0,0 +1,2 @@
+alternate-invalid-data-type.json: In alternate 'Alt':
+alternate-invalid-data-type.json:2: 'data' must be an object
diff --git a/tests/qapi-schema/union-invalid-data-type.err 
b/tests/qapi-schema/union-invalid-data-type.err
index e69de29bb2..b71c3400c5 100644
--- a/tests/qapi-schema/union-invalid-data-type.err
+++ b/tests/qapi-schema/union-invalid-data-type.err
@@ -0,0 +1,2 @@
+union-invalid-data-type.json: In union 'TestUnion':
+union-invalid-data-type.json:9: 'data' must be an object
-- 
2.30.2

[PATCH v4 11/19] qapi/expr.py: Modify check_keys to accept any Collection

[John Snow]

This is a minor adjustment that allows the 'required' and 'optional'
keys fields to take a default value of an empty, immutable sequence (the
empty tuple).

This reveals a quirk of this function, which is that "a + b" is
list-specific behavior. We can accept a wider variety of types if we
avoid that behavior. Using Collection allows us to accept things like
lists, tuples, sets, and so on.

(Iterable would also have worked, but Iterable also includes things like
generator expressions which are consumed upon iteration, which would
require a rewrite to make sure that each input was only traversed once.)

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 505e67bd21..7e22723b50 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -100,7 +100,7 @@ def pprint(elems):
 "%s misses key%s %s"
 % (source, 's' if len(missing) > 1 else '',
pprint(missing)))
-allowed = set(required + optional)
+allowed = set(required) | set(optional)
 unknown = set(value) - allowed
 if unknown:
 raise QAPISemError(
-- 
2.30.2

[PATCH v4 06/19] qapi/expr.py: Add assertion for union type 'check_dict'

[John Snow]

mypy isn't fond of allowing you to check for bool membership in a
collection of str elements. Guard this lookup for precisely when we were
given a name.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
---
 scripts/qapi/expr.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index b75c85c160..2a2cf7064f 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -192,7 +192,9 @@ def check_type(value, info, source,
 raise QAPISemError(info,
"%s should be an object or type name" % source)
 
-permissive = allow_dict in info.pragma.member_name_exceptions
+permissive = False
+if isinstance(allow_dict, str):
+permissive = allow_dict in info.pragma.member_name_exceptions
 
 # value is a dictionary, check that each member is okay
 for (key, arg) in value.items():
-- 
2.30.2

[PATCH v4 02/19] flake8: Enforce shorter line length for comments and docstrings

[John Snow]

PEP8's BDFL writes: "For flowing long blocks of text with fewer
structural restrictions (docstrings or comments), the line length should
be limited to 72 characters."

I do not like this patch. I have included it explicitly to recommend we
do not pay any further heed to the 72 column limit.

Signed-off-by: John Snow 
---
 scripts/qapi/.flake8   |  1 +
 scripts/qapi/common.py |  8 +---
 scripts/qapi/events.py |  9 +
 scripts/qapi/gen.py|  8 
 scripts/qapi/introspect.py |  8 +---
 scripts/qapi/main.py   |  4 ++--
 scripts/qapi/parser.py | 15 ---
 scripts/qapi/schema.py | 23 +--
 scripts/qapi/types.py  |  7 ---
 9 files changed, 47 insertions(+), 36 deletions(-)

diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
index 6b158c68b8..4f00455290 100644
--- a/scripts/qapi/.flake8
+++ b/scripts/qapi/.flake8
@@ -1,2 +1,3 @@
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
+max-doc-length = 72
\ No newline at end of file
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index cbd3fd81d3..6e3d9b8ecd 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -41,7 +41,8 @@ def camel_to_upper(value: str) -> str:
 length = len(c_fun_str)
 for i in range(length):
 char = c_fun_str[i]
-# When char is upper case and no '_' appears before, do more checks
+# When char is upper case and no '_' appears before,
+# do more checks
 if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
 if i < length - 1 and c_fun_str[i + 1].islower():
 new_name += '_'
@@ -78,8 +79,9 @@ def c_name(name: str, protect: bool = True) -> str:
 protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 
 :param name: The name to map.
-:param protect: If true, avoid returning certain ticklish identifiers
-(like C keywords) by prepending ``q_``.
+:param protect: If true, avoid returning certain ticklish
+identifiers (like C keywords) by prepending
+``q_``.
 """
 # ANSI X3J11/88-090, 3.1.1
 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index fee8c671e7..210b56974f 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -48,7 +48,8 @@ def gen_param_var(typ: QAPISchemaObjectType) -> str:
 """
 Generate a struct variable holding the event parameters.
 
-Initialize it with the function arguments defined in `gen_event_send`.
+Initialize it with the function arguments defined in
+`gen_event_send`.
 """
 assert not typ.variants
 ret = mcgen('''
@@ -86,9 +87,9 @@ def gen_event_send(name: str,
 # FIXME: Our declaration of local variables (and of 'errp' in the
 # parameter list) can collide with exploded members of the event's
 # data type passed in as parameters.  If this collision ever hits in
-# practice, we can rename our local variables with a leading _ prefix,
-# or split the code into a wrapper function that creates a boxed
-# 'param' object then calls another to do the real work.
+# practice, we can rename our local variables with a leading _
+# prefix, or split the code into a wrapper function that creates a
+# boxed 'param' object then calls another to do the real work.
 have_args = boxed or (arg_type and not arg_type.is_empty())
 
 ret = mcgen('''
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 1fa503bdbd..c54980074e 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -63,9 +63,9 @@ def _bottom(self) -> str:
 return ''
 
 def write(self, output_dir: str) -> None:
-# Include paths starting with ../ are used to reuse modules of the main
-# schema in specialised schemas. Don't overwrite the files that are
-# already generated for the main schema.
+# Include paths starting with ../ are used to reuse modules
+# of the main schema in specialised schemas. Don't overwrite
+# the files that are already generated for the main schema.
 if self.fname.startswith('../'):
 return
 pathname = os.path.join(output_dir, self.fname)
@@ -189,7 +189,7 @@ def _bottom(self) -> str:
 @contextmanager
 def ifcontext(ifcond: Sequence[str], *args: QAPIGenCCode) -> Iterator[None]:
 """
-A with-statement context manager that wraps with `start_if()` / `end_if()`.
+A context manager that wraps output with `start_if()` / `end_if()`.
 
 :param ifcond: A sequence of conditionals, passed to `start_if()`.
 :param args: any number of `QAPIGenCCode`.
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 9a348ca2e5..faf00013ad 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -61,8 +61,9 @@
 # With optional annotations, the type 

[PATCH v4 04/19] qapi/expr.py: Check for dict instead of OrderedDict

[John Snow]

OrderedDict is a subtype of dict, so we can check for a more general
form. These functions do not themselves depend on it being any
particular type.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
---
 scripts/qapi/expr.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 3fda5d5082..b4bbcd54c0 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -14,7 +14,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from collections import OrderedDict
 import re
 
 from .common import c_name
@@ -149,7 +148,7 @@ def check_if_str(ifcond):
 
 
 def normalize_members(members):
-if isinstance(members, OrderedDict):
+if isinstance(members, dict):
 for key, arg in members.items():
 if isinstance(arg, dict):
 continue
@@ -180,7 +179,7 @@ def check_type(value, info, source,
 if not allow_dict:
 raise QAPISemError(info, "%s should be a type name" % source)
 
-if not isinstance(value, OrderedDict):
+if not isinstance(value, dict):
 raise QAPISemError(info,
"%s should be an object or type name" % source)
 
-- 
2.30.2

[PATCH v4 18/19] qapi/expr.py: move related checks inside check_xxx functions

[John Snow]

There's not a big obvious difference between the types of checks that
happen in the main function versus the kind that happen in the
functions. Now they're in one place for each of the main types.

As part of the move, spell out the required and optional keywords so
they're obvious at a glance. Use tuples instead of lists for immutable
data, too.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
---
 scripts/qapi/expr.py | 55 ++--
 1 file changed, 33 insertions(+), 22 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index b11c11b965..aabbc255d2 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -398,6 +398,10 @@ def check_enum(expr: _JSONObject, info: QAPISourceInfo) -> 
None:
 
 :return: None, ``expr`` is normalized in-place as needed.
 """
+check_keys(expr, info, 'enum',
+   required=('enum', 'data'),
+   optional=('if', 'features', 'prefix'))
+
 name = expr['enum']
 members = expr['data']
 prefix = expr.get('prefix')
@@ -435,6 +439,11 @@ def check_struct(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 
 :return: None, ``expr`` is normalized in-place as needed.
 """
+check_keys(expr, info, 'struct',
+   required=('struct', 'data'),
+   optional=('base', 'if', 'features'))
+normalize_members(expr['data'])
+
 name = cast(str, expr['struct'])  # Asserted in check_exprs
 members = expr['data']
 
@@ -451,6 +460,13 @@ def check_union(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 
 :return: None, ``expr`` is normalized in-place as needed.
 """
+check_keys(expr, info, 'union',
+   required=('union', 'data'),
+   optional=('base', 'discriminator', 'if', 'features'))
+
+normalize_members(expr.get('base'))
+normalize_members(expr['data'])
+
 name = cast(str, expr['union'])  # Asserted in check_exprs
 base = expr.get('base')
 discriminator = expr.get('discriminator')
@@ -487,6 +503,11 @@ def check_alternate(expr: _JSONObject, info: 
QAPISourceInfo) -> None:
 
 :return: None, ``expr`` is normalized in-place as needed.
 """
+check_keys(expr, info, 'alternate',
+   required=('alternate', 'data'),
+   optional=('if', 'features'))
+normalize_members(expr['data'])
+
 members = expr['data']
 
 if not members:
@@ -512,6 +533,13 @@ def check_command(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 
 :return: None, ``expr`` is normalized in-place as needed.
 """
+check_keys(expr, info, 'command',
+   required=('command',),
+   optional=('data', 'returns', 'boxed', 'if', 'features',
+ 'gen', 'success-response', 'allow-oob',
+ 'allow-preconfig', 'coroutine'))
+normalize_members(expr.get('data'))
+
 args = expr.get('data')
 rets = expr.get('returns')
 boxed = expr.get('boxed', False)
@@ -531,6 +559,11 @@ def check_event(expr: _JSONObject, info: QAPISourceInfo) 
-> None:
 
 :return: None, ``expr`` is normalized in-place as needed.
 """
+check_keys(expr, info, 'event',
+   required=('event',),
+   optional=('data', 'boxed', 'if', 'features'))
+normalize_members(expr.get('data'))
+
 args = expr.get('data')
 boxed = expr.get('boxed', False)
 
@@ -598,38 +631,16 @@ def check_exprs(exprs: List[_JSONObject]) -> 
List[_JSONObject]:
"documentation comment required")
 
 if meta == 'enum':
-check_keys(expr, info, meta,
-   ['enum', 'data'], ['if', 'features', 'prefix'])
 check_enum(expr, info)
 elif meta == 'union':
-check_keys(expr, info, meta,
-   ['union', 'data'],
-   ['base', 'discriminator', 'if', 'features'])
-normalize_members(expr.get('base'))
-normalize_members(expr['data'])
 check_union(expr, info)
 elif meta == 'alternate':
-check_keys(expr, info, meta,
-   ['alternate', 'data'], ['if', 'features'])
-normalize_members(expr['data'])
 check_alternate(expr, info)
 elif meta == 'struct':
-check_keys(expr, info, meta,
-   ['struct', 'data'], ['base', 'if', 'features'])
-normalize_members(expr['data'])
 check_struct(expr, info)
 elif meta == 'command':
-check_keys(expr, info, meta,
-   ['command'],
-   ['data', 'returns', 'boxed', 'if', 'features',
-'gen', 'success-response', 'allow-oob',
-'allow-preconfig', 'coroutine'])
-normalize_members(expr.get('data'))
 check_command(expr, info)
 elif meta == 'event':
-check_keys(expr, info, meta,
-  

[PATCH v4 07/19] qapi/expr.py: move string check upwards in check_type

[John Snow]

For readability purposes only, shimmy the early return upwards to the
top of the function, so cases proceed in order from least to most
complex.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
---
 scripts/qapi/expr.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 2a2cf7064f..73e7d8cb0d 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -169,6 +169,10 @@ def check_type(value, info, source,
 if value is None:
 return
 
+# Type name
+if isinstance(value, str):
+return
+
 # Array type
 if isinstance(value, list):
 if not allow_array:
@@ -179,10 +183,6 @@ def check_type(value, info, source,
source)
 return
 
-# Type name
-if isinstance(value, str):
-return
-
 # Anonymous type
 
 if not allow_dict:
-- 
2.30.2

[PATCH v4 01/19] qapi/expr: Comment cleanup

[John Snow]

Fixes: 0825f62c842

Signed-off-by: John Snow 
---
 scripts/qapi/expr.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 540b3982b1..c207481f7e 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -241,7 +241,7 @@ def check_enum(expr, info):
 source = "%s '%s'" % (source, member_name)
 # Enum members may start with a digit
 if member_name[0].isdigit():
-member_name = 'd' + member_name # Hack: hide the digit
+member_name = 'd' + member_name  # Hack: hide the digit
 check_name_lower(member_name, info, source,
  permit_upper=permissive,
  permit_underscore=permissive)
-- 
2.30.2

[PATCH v4 00/19] qapi: static typing conversion, pt3

[John Snow]

Hi, this series adds static types to the QAPI module.
This is part three, and it focuses on expr.py.

Environment:
- Python >= 3.6, <= 3.8 *
- mypy >= 0.770
- pylint >= 2.6.0
- flake8
- isort

Every commit should pass with (from ./scripts/):
 - flake8 qapi/
 - pylint --rcfile=qapi/pylintrc qapi/
 - mypy --config-file=qapi/mypy.ini qapi/
 - pushd qapi && isort -c . && popd

V4:

Patch 2 is exploratory.
Patch 8 is broken and should be merged into Patch 9.
Patches 17-19 are optional and I'd sooner you drop them than have to respin.

001/19:[down] 'qapi/expr: Comment cleanup'
002/19:[down] 'flake8: Enforce shorter line length for comments and docstrings'
003/19:[] [--] 'qapi/expr.py: Remove 'info' argument from nested 
check_if_str'
004/19:[] [--] 'qapi/expr.py: Check for dict instead of OrderedDict'
005/19:[0011] [FC] 'qapi/expr.py: constrain incoming expression types'
006/19:[0006] [FC] 'qapi/expr.py: Add assertion for union type 'check_dict''
007/19:[] [--] 'qapi/expr.py: move string check upwards in check_type'
008/19:[down] 'qapi: add tests for invalid 'data' field type'
009/19:[0004] [FC] 'qapi/expr.py: Check type of 'data' member'
010/19:[0008] [FC] 'qapi/expr.py: Add casts in a few select cases'
011/19:[0005] [FC] 'qapi/expr.py: Modify check_keys to accept any Collection'
012/19:[0057] [FC] 'qapi/expr.py: add type hint annotations'
013/19:[0032] [FC] 'qapi/expr.py: Consolidate check_if_str calls in check_if'
014/19:[0016] [FC] 'qapi/expr.py: Remove single-letter variable'
015/19:[] [--] 'qapi/expr.py: enable pylint checks'
016/19:[0168] [FC] 'qapi/expr.py: Add docstrings'
017/19:[] [-C] 'qapi/expr.py: Use tuples instead of lists for static data'
018/19:[] [-C] 'qapi/expr.py: move related checks inside check_xxx 
functions'
019/19:[0003] [FC] 'qapi/expr.py: Use an expression checker dispatch table'

- Add test patch to demonstrate 72col docstring enforcement. (Not a fan.)
- Changed MutableMapping type to regular ol' dict.
- Added tests for alternate and union to see what happens when we pass a list
  for 'data' instead. (It crashes.)
- Rewrote a bunch of the docstrings.
- Updated type hints for rc0
- Rebased on latest master, incorporating latest qapi changes.
- Addressed most feedback, some exceptions;
  - Kept isinstance check for dict; it is strictly more convenient to me and it
does not cause breakages. It won't cause breakages.

RFCs/notes:

- I'd be flabbergasted if anyone reads these.

John Snow (19):
  qapi/expr: Comment cleanup
  flake8: Enforce shorter line length for comments and docstrings
  qapi/expr.py: Remove 'info' argument from nested check_if_str
  qapi/expr.py: Check for dict instead of OrderedDict
  qapi/expr.py: constrain incoming expression types
  qapi/expr.py: Add assertion for union type 'check_dict'
  qapi/expr.py: move string check upwards in check_type
  qapi: add tests for invalid 'data' field type
  qapi/expr.py: Check type of 'data' member
  qapi/expr.py: Add casts in a few select cases
  qapi/expr.py: Modify check_keys to accept any Collection
  qapi/expr.py: add type hint annotations
  qapi/expr.py: Consolidate check_if_str calls in check_if
  qapi/expr.py: Remove single-letter variable
  qapi/expr.py: enable pylint checks
  qapi/expr.py: Add docstrings
  qapi/expr.py: Use tuples instead of lists for static data
  qapi/expr.py: move related checks inside check_xxx functions
  qapi/expr.py: Use an expression checker dispatch table

 scripts/qapi/.flake8  |   1 +
 scripts/qapi/common.py|   8 +-
 scripts/qapi/events.py|   9 +-
 scripts/qapi/expr.py  | 499 +-
 scripts/qapi/gen.py   |   8 +-
 scripts/qapi/introspect.py|   8 +-
 scripts/qapi/main.py  |   4 +-
 scripts/qapi/mypy.ini |   5 -
 scripts/qapi/parser.py|  15 +-
 scripts/qapi/pylintrc |   1 -
 scripts/qapi/schema.py|  23 +-
 scripts/qapi/types.py |   7 +-
 .../alternate-invalid-data-type.err   |   2 +
 .../alternate-invalid-data-type.json  |   4 +
 .../alternate-invalid-data-type.out   |   0
 tests/qapi-schema/meson.build |   2 +
 tests/qapi-schema/union-invalid-data-type.err |   2 +
 .../qapi-schema/union-invalid-data-type.json  |  13 +
 tests/qapi-schema/union-invalid-data-type.out |   0
 19 files changed, 449 insertions(+), 162 deletions(-)
 create mode 100644 tests/qapi-schema/alternate-invalid-data-type.err
 create mode 100644 tests/qapi-schema/alternate-invalid-data-type.json
 create mode 100644 tests/qapi-schema/alternate-invalid-data-type.out
 create mode 100644 tests/qapi-schema/union-invalid-data-type.err
 create mode 100644 tests/qapi-schema/union-invalid-data-type.json
 create mode 100644 tests/qapi-schema/union-invalid-data-type.out

-- 
2.30.2

[PATCH V2] target/riscv: Align the data type of reset vector address

[Dylan Jhong]

Signed-off-by: Dylan Jhong 
Signed-off-by: Ruinland ChuanTzu Tsai 
---
 target/riscv/cpu.c | 2 +-
 target/riscv/cpu.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index 7d6ed80f6b..4ac901245a 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -137,7 +137,7 @@ static void set_feature(CPURISCVState *env, int feature)
 env->features |= (1ULL << feature);
 }
 
-static void set_resetvec(CPURISCVState *env, int resetvec)
+static void set_resetvec(CPURISCVState *env, target_ulong resetvec)
 {
 #ifndef CONFIG_USER_ONLY
 env->resetvec = resetvec;
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index 0a33d387ba..d9d7891666 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -303,7 +303,7 @@ struct RISCVCPU {
 uint16_t elen;
 bool mmu;
 bool pmp;
-uint64_t resetvec;
+target_ulong resetvec;
 } cfg;
 };
 
-- 
2.17.1

Re: [PATCH v3 04/16] qapi/expr.py: Add assertion for union type 'check_dict'

[Markus Armbruster]

John Snow  writes:

> On 2/24/21 5:35 AM, Markus Armbruster wrote:
>> John Snow  writes:
>> 
>>> mypy isn't fond of allowing you to check for bool membership in a
>>> collection of str elements. Guard this lookup for precisely when we were
>>> given a name.
>>>
>>> Signed-off-by: John Snow 
>>> Reviewed-by: Eduardo Habkost 
>>> Reviewed-by: Cleber Rosa 
>>> ---
>>>   scripts/qapi/expr.py | 4 +++-
>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
>>> index 783282b53ce..138fab0711f 100644
>>> --- a/scripts/qapi/expr.py
>>> +++ b/scripts/qapi/expr.py
>>> @@ -173,7 +173,9 @@ def check_type(value, info, source,
>>>   raise QAPISemError(info,
>>>  "%s should be an object or type name" % source)
>>>   -permit_upper = allow_dict in info.pragma.name_case_whitelist
>>> +permit_upper = False
>>> +if isinstance(allow_dict, str):
>>> +permit_upper = allow_dict in info.pragma.name_case_whitelist
>>> # value is a dictionary, check that each member is okay
>>>   for (key, arg) in value.items():
>> 
>> Busy-work like this can make me doubt typing is worth the notational
>> overhead.
>> There must a less awkward way to plumb "upper case okay" through
>> check_type() to check_name_is_str().  But we're typing what we have.
>
> Leaving this as-is for now. There's something I'd like to do about it,
> but it has to happen later.
>
> (I think all the pragma checks should happen in schema.py, and not in
> expr.py. They are by their essence not context-free, since they depend 
> on the context of the pragma.)

True.

Pragmas other than doc-required are an ugly consequence of us having
made a a bit of a mess in the schema.  The oldest parts of the schema
were set in stone before we decided on certain rules, and then we kept
failing at manually enforcing these rules.  To get automatic
enforcement, we needed a way to give a pass to existing rule breakers.
Preferably without rearchitecting the frontend.  Pragmas solve that
problem.  The solution is as ugly as the problem.

Without pragmas, the name checks are context-free.  That's why they are
where they are.

Re: gitlab-ci: Only build /staging branch?

[Thomas Huth]

On 24/03/2021 22.58, Philippe Mathieu-Daudé wrote:

On 3/24/21 7:33 PM, Philippe Mathieu-Daudé wrote:

On 3/24/21 7:01 PM, Philippe Mathieu-Daudé wrote:

Hi,

Peter's current workflow is push to /staging and if his
testing succeeds, he pushes the same commit as /master.

IMO there is no point in building /master branch, as it
has already been built earlier as /staging.


Similarly with tags. Although we don't tag often.


Tags are used for pull-requests. So I think we should run the whole CI for 
tags, to make it clear that a pull-request always includes code that builds 
fine.


 Thomas

Re: [PATCH 3/5] tools/vhost-user-i2c: Add backend driver

[Viresh Kumar]

On 25-03-21, 13:09, Jie Deng wrote:
> 
> On 2021/3/24 15:33, Viresh Kumar wrote:
> > +
> > +/* Definitions from virtio-i2c specifications */
> > +#define VHOST_USER_I2C_MAX_QUEUES   1
> > +
> > +/* Status */
> > +#define VIRTIO_I2C_MSG_OK   0
> > +#define VIRTIO_I2C_MSG_ERR  1
> > +
> > +/* The bit 0 of the @virtio_i2c_out_hdr.@flags, used to group the requests 
> > */
> > +#define VIRTIO_I2C_FLAGS_FAIL_NEXT  0x0001
> > +
> > +/**
> > + * struct virtio_i2c_out_hdr - the virtio I2C message OUT header
> > + * @addr: the controlled device's address
> > + * @padding: used to pad to full dword
> > + * @flags: used for feature extensibility
> > + */
> > +struct virtio_i2c_out_hdr {
> > +uint16_t addr;
> > +uint16_t padding;
> > +uint32_t flags;
> > +} __attribute__((packed));
> 
> 
> __le16,  __le32 ?

Maybe, but I didn't do them because of this:

docs/devel/style.rst:

"Don't use Linux kernel internal types like u32, __u32 or __le32."
 
> > +
> > +/**
> > + * struct virtio_i2c_in_hdr - the virtio I2C message IN header
> > + * @status: the processing result from the backend
> > + */
> > +struct virtio_i2c_in_hdr {
> > +uint8_t status;
> > +} __attribute__((packed));
> > +
> 
> I understand these definitions can be removed once the frontend driver is
> merged by the Linux ?

Yes, we would be required to somehow include the uapi header that
kernel is adding and then this won't be required.
 
> > +/* vhost-user-i2c definitions */
> > +
> > +#ifndef container_of
> > +#define container_of(ptr, type, member) ({  \
> > +const typeof(((type *) 0)->member) *__mptr = (ptr); \
> > +(type *) ((char *) __mptr - offsetof(type, member));})
> > +#endif
> 
> 
> This seems to be a general interface.  I see there is a definition in
> qemu/compiler.h.
> 
> Can we reuse it ?

Damn. My bad (maybe not). I picked this part from the RPMB patchset
that Alex sent and didn't bother looking for it.

Though on the other hand, we are looking to make this file independent
of qemu so it can be used by other hypervisors without any (or much)
modifications, and maybe so it was done so.

Alex ?

-- 
viresh

Re: [PATCH v3 15/16] qapi/expr.py: move related checks inside check_xxx functions

[John Snow]

On 2/25/21 10:28 AM, Markus Armbruster wrote:

John Snow  writes:


There's not a big obvious difference between the types of checks that
happen in the main function versus the kind that happen in the
functions. Now they're in one place for each of the main types.

As part of the move, spell out the required and optional keywords so
they're obvious at a glance. Use tuples instead of lists for immutable
data, too.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 


No objection to changing read-only lists to tuples (applies to previous
patch, too).

No objection to turning positional into keyword arguments where that
improves clarity.

I have doubts on the code motion.  Yes, the checks for each type are now
together.  On the other hand, the check_keys() are now separate.  I can
no longer see all the keys at a glance.



I guess it depends on where you wanted to see them; I thought it was 
strange that in check_foobar I couldn't see what foobar's valid keys 
were without scrolling back to the bottom of the file.


Needing to see all the keys for the disparate forms together was not a 
case I ran into, but you can always drop this patch for now if you'd 
like. I had some more adventurous patches that keeps pushing in this 
direction, but I don't know if it's really important. My appetite in 
this area has waned since November.


--js

Re: [PATCH 3/5] tools/vhost-user-i2c: Add backend driver

[Jie Deng]

On 2021/3/24 15:33, Viresh Kumar wrote:

+
+/* Definitions from virtio-i2c specifications */
+#define VHOST_USER_I2C_MAX_QUEUES   1
+
+/* Status */
+#define VIRTIO_I2C_MSG_OK   0
+#define VIRTIO_I2C_MSG_ERR  1
+
+/* The bit 0 of the @virtio_i2c_out_hdr.@flags, used to group the requests */
+#define VIRTIO_I2C_FLAGS_FAIL_NEXT  0x0001
+
+/**
+ * struct virtio_i2c_out_hdr - the virtio I2C message OUT header
+ * @addr: the controlled device's address
+ * @padding: used to pad to full dword
+ * @flags: used for feature extensibility
+ */
+struct virtio_i2c_out_hdr {
+uint16_t addr;
+uint16_t padding;
+uint32_t flags;
+} __attribute__((packed));



__le16,  __le32 ?



+
+/**
+ * struct virtio_i2c_in_hdr - the virtio I2C message IN header
+ * @status: the processing result from the backend
+ */
+struct virtio_i2c_in_hdr {
+uint8_t status;
+} __attribute__((packed));
+



I understand these definitions can be removed once the frontend driver 
is merged by the Linux ?




+/* vhost-user-i2c definitions */
+
+#ifndef container_of
+#define container_of(ptr, type, member) ({  \
+const typeof(((type *) 0)->member) *__mptr = (ptr); \
+(type *) ((char *) __mptr - offsetof(type, member));})
+#endif



This seems to be a general interface.  I see there is a definition in 
qemu/compiler.h.


Can we reuse it ?

[PATCH qemu v17] spapr: Implement Open Firmware client interface

[Alexey Kardashevskiy]

The PAPR platform which describes an OS environment that's presented by
a combination of a hypervisor and firmware. The features it specifies
require collaboration between the firmware and the hypervisor.

Since the beginning, the runtime component of the firmware (RTAS) has
been implemented as a 20 byte shim which simply forwards it to
a hypercall implemented in qemu. The boot time firmware component is
SLOF - but a build that's specific to qemu, and has always needed to be
updated in sync with it. Even though we've managed to limit the amount
of runtime communication we need between qemu and SLOF, there's some,
and it has become increasingly awkward to handle as we've implemented
new features.

This implements a boot time OF client interface (CI) which is
enabled by a new "x-vof" pseries machine option (stands for "Virtual Open
Firmware). When enabled, QEMU implements the custom H_OF_CLIENT hcall
which implements Open Firmware Client Interface (OF CI). This allows
using a smaller stateless firmware which does not have to manage
the device tree.

The new "vof.bin" firmware image is included with source code under
pc-bios/. It also includes RTAS blob.

This implements a handful of CI methods just to get -kernel/-initrd
working. In particular, this implements the device tree fetching and
simple memory allocator - "claim" (an OF CI memory allocator) and updates
"/memory@0/available" to report the client about available memory.

This implements changing some device tree properties which we know how
to deal with, the rest is ignored. To allow changes, this skips
fdt_pack() when x-vof=on as not packing the blob leaves some room for
appending.

In absence of SLOF, this assigns phandles to device tree nodes to make
device tree traversing work.

When x-vof=on, this adds "/chosen" every time QEMU (re)builds a tree.

This adds basic instances support which are managed by a hash map
ihandle -> [phandle].

Before the guest started, the used memory is:
0..e60 - the initial firmware
8000..1 - stack
40.. - kernel
3ea.. - initramdisk

This OF CI does not implement "interpret".

Unlike SLOF, this does not format uninitialized nvram. Instead, this
includes a disk image with pre-formatted nvram.

With this basic support, this can only boot into kernel directly.
However this is just enough for the petitboot kernel and initradmdisk to
boot from any possible source. Note this requires reasonably recent guest
kernel with:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=df5be5be8735

The immediate benefit is much faster booting time which especially
crucial with fully emulated early CPU bring up environments. Also this
may come handy when/if GRUB-in-the-userspace sees light of the day.

This separates VOF and sPAPR in a hope that VOF bits may be reused by
other POWERPC boards which do not support pSeries.

This is coded in assumption that later on we might be adding support for
booting from QEMU backends (blockdev is the first candidate) without
devices/drivers in between as OF1275 does not require that and
it is quite easy to so.

Signed-off-by: Alexey Kardashevskiy 
---

The example command line is:

/home/aik/pbuild/qemu-killslof-localhost-ppc64/qemu-system-ppc64 \
-nodefaults \
-chardev stdio,id=STDIO0,signal=off,mux=on \
-device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
-mon id=MON0,chardev=STDIO0,mode=readline \
-nographic \
-vga none \
-enable-kvm \
-m 2G \
-machine 
pseries,x-vof=on,cap-cfpc=broken,cap-sbbc=broken,cap-ibs=broken,cap-ccf-assist=off
 \
-kernel pbuild/kernel-le-guest/vmlinux \
-initrd pb/rootfs.cpio.xz \
-drive 
id=DRIVE0,if=none,file=./p/qemu-killslof/pc-bios/vof-nvram.bin,format=raw \
-global spapr-nvram.drive=DRIVE0 \
-snapshot \
-smp 8,threads=8 \
-L /home/aik/t/qemu-ppc64-bios/ \
-trace events=qemu_trace_events \
-d guest_errors \
-chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.tmux26 \
-mon chardev=SOCKET0,mode=control

---
Changes:
v17:
* mv hw/ppc/vof.h include/hw/ppc/vof.h
* VofMachineIfClass -> VofMachineClass; it is not VofMachineInterface as
nobody used this scheme, usually "Interface" is dropped, a couple of times
it is "xxxInterfaceClass" or "xxxIfClass", as used the latter as it is
used by include/hw/vmstate-if.h
* added SPDX
* other fixes from v16 review

v16:
* rebased on dwg/ppc-for-6.1
* s/SpaprVofInterface/VofMachineInterface/

v15:
* bugfix: claimed memory for the VOF itself
* ditched OF_STACK_ADDR and allocate one instead, now it starts from 0x8000
because it is aligned to its size (no particular reason though)
* coding style
* moved nvram.bin up one level
* ditched bool in the firmware
* made debugging code conditional using trace_event_get_state() + 
qemu_loglevel_mask()
* renamed the CAS interface to SpaprVofInterface
* added "write" which for now dumps the message and ihandle via
trace point for early debug assistance
* commented on when we allocate of_instances in vof_build_dt()
* store fw_size is SpaprMachine to let spapr_v

Re: [PATCH] target/riscv: Align the data type of reset vector address

[Bin Meng]

On Thu, Mar 25, 2021 at 11:40 AM Bin Meng  wrote:
>
> On Thu, Mar 25, 2021 at 11:32 AM Dylan Jhong  wrote:
> >
> > On Wed, Mar 24, 2021 at 10:59:55PM +0800, Alistair Francis wrote:
> > > On Tue, Mar 23, 2021 at 5:15 AM Dylan Jhong  wrote:
> > > >
> > > > Although the AE350 has not been upstream (preparing for v2),
> > > > the reset vector of the AE350 is known to be at the 2G position,
> > > > so this patch is corrected in advance.
> > > >
> > > > Signed-off-by: Dylan Jhong 
> > > > Signed-off-by: Ruinland ChuanTzu Tsai 
> > > > ---
> > > >  target/riscv/cpu.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> > > > index 2a990f6253..0236abf169 100644
> > > > --- a/target/riscv/cpu.c
> > > > +++ b/target/riscv/cpu.c
> > > > @@ -137,7 +137,7 @@ static void set_feature(CPURISCVState *env, int 
> > > > feature)
> > > >  env->features |= (1ULL << feature);
> > > >  }
> > > >
> > > > -static void set_resetvec(CPURISCVState *env, int resetvec)
> > > > +static void set_resetvec(CPURISCVState *env, uint64_t resetvec)
> > >
> > > resetvec in env is a target_ulong so this should be as well (instead
> > > of a uint64_t).
> > >
> > > Alistair
> > >
> >
> > Hi Alistar,
> >
> > Thanks for your comments.
> >
> > Indeed resetvec should use target_ulong instead of uint64_t.
>
> resetvec being target_ulong means that rv32 cannot have a reset vector
> beyond 4GiB. I don't think the spec disallow this.

Ah, I was wrong. The spec says: "the pc is set to an implementation-de
ned reset vector" and pc is XLEN wide.

So for rv32 the reset vector cannot beyond 4GiB. We should change it
to target_ulong.

Regards,
Bin

Re: [PATCH 2/3] aspeed: Add Scater-Gather support for HACE Hash

[Joel Stanley]

On Wed, 24 Mar 2021 at 22:39, Klaus Heinrich Kiwi
 wrote:
>
> Complement the Aspeed HACE support with Scatter-Gather hash support for
> sha256 and sha512. Scatter-Gather is only supported on AST2600-series.

Please update the documentation at docs/system/arm/aspeed.rst too.

>
> Signed-off-by: Klaus Heinrich Kiwi 
> ---
>  hw/misc/aspeed_hace.c | 127 --
>  include/hw/misc/aspeed_hace.h |   6 ++
>  2 files changed, 127 insertions(+), 6 deletions(-)
>
> diff --git a/hw/misc/aspeed_hace.c b/hw/misc/aspeed_hace.c
> index 93313d2b80..8a37b1d961 100644
> --- a/hw/misc/aspeed_hace.c
> +++ b/hw/misc/aspeed_hace.c
> @@ -57,6 +57,10 @@
>  /* Other cmd bits */
>  #define  HASH_IRQ_ENBIT(9)
>  #define  HASH_SG_EN BIT(18)
> +/* Scatter-gather data list */
> +#define  SG_LIST_LAST   BIT(31)
> +#define  SG_LIST_LEN_MASK   0x7fff
> +#define  SG_LIST_ADDR_MASK  0x7ff8  /* 8-byte aligned */
>
>  static const struct {
>  uint32_t mask;
> @@ -129,6 +133,117 @@ static int do_hash_operation(AspeedHACEState *s, int 
> algo)
>  return 0;
>  }
>
> +static int do_hash_sg_operation(AspeedHACEState *s, int algo)
> +{
> +uint32_t src, dest, reqSize;
> +hwaddr len;
> +const size_t reqLen = sizeof(struct aspeed_sg_list);

It would be more descriptive to use this sizeof where you need it,
instead of assigning to a constant.

> +struct iovec iov[ASPEED_HACE_MAX_SG];
> +unsigned int i = 0;
> +unsigned int isLast = 0;

This sounds like it's a boolean.

> +uint8_t *digestBuf = NULL;
> +size_t digestLen = 0, size = 0;
> +struct aspeed_sg_list *sgList;
> +int rc;

This needs some work to match qemu coding style.

> +
> +reqSize = s->regs[R_HASH_SRC_LEN];
> +dest = s->regs[R_HASH_DEST];
> +
> +while (!isLast && i < ASPEED_HACE_MAX_SG) {
> +src = s->regs[R_HASH_SRC] + (i * reqLen);
> +len = reqLen;
> +sgList = (struct aspeed_sg_list *) address_space_map(&s->dram_as,

You can remove this cast as the function returns a void pointer.

> + src,
> + (hwaddr *) &len,

You can remove this cast as the variable is already a hwaddr type.

> +   false,
> + MEMTXATTRS_UNSPECIFIED);

In the direct access code, we use address_space_map to save copying
the memory contents that is to be hashed. That's not the case for the
scatter gather list.

Instead of creating mappings to read the sg list, you could load the
addr, len pairs using address_space_ldl_le. This would give you the
pointer to create mappings for.

> +if (!sgList) {
> +qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: failed to map dram for SG Array entry '%u' for address 
> '0x%0x'\n",
> + __func__, i, src);
> +rc = -EACCES;
> +goto cleanup;
> +}
> +if (len != reqLen)
> +qemu_log_mask(LOG_GUEST_ERROR,
> + "%s:  Warning: dram map for SG array entry '%u' requested size 
> '%lu' != mapped size '%lu'\n",
> + __func__, i, reqLen, len);
> +
> +isLast = sgList->len & SG_LIST_LAST;

You could drop the isLast variable, and perform this test at the
bottom of the while loop:

if (sgList->len & SG_LIST_LAST)
   break;

> +
> +iov[i].iov_len = (hwaddr) (sgList->len & SG_LIST_LEN_MASK);
> +iov[i].iov_base = address_space_map(&s->dram_as,
> +sgList->phy_addr & SG_LIST_ADDR_MASK,
> +&iov[i].iov_len, false,
> +MEMTXATTRS_UNSPECIFIED);
> +if (!iov[i].iov_base) {
> +qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: failed to map dram for SG array entry '%u' for region 
> '0x%x', len '%u'\n",
> + __func__, i, sgList->phy_addr & SG_LIST_ADDR_MASK,
> + sgList->len & SG_LIST_LEN_MASK);
> +rc = -EACCES;
> +goto cleanup;
> +}
> +if (iov[i].iov_len != (sgList->len & SG_LIST_LEN_MASK))
> +qemu_log_mask(LOG_GUEST_ERROR,
> + "%s:  Warning: dram map for SG region entry %u requested size 
> %u != mapped size %lu\n",
> + __func__, i, (sgList->len & SG_LIST_LEN_MASK), iov[i].iov_len);
> +
> +
> +address_space_unmap(&s->dram_as, (void *) sgList, len, false,
> +len);
> +size += iov[i].iov_len;
> +i++;
> +}
> +
> +if (!isLast) {
> +qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: Error: Exhausted maximum of '%u' SG array 
> entries\n",
> + __func__, ASPEED_HACE_MAX_SG);
> +rc = -ENOTSUP;
> +goto cleanup;
> +}
> +
> +if (size != reqSize)
> +

Re: [PATCH] target/riscv: Align the data type of reset vector address

[Bin Meng]

On Thu, Mar 25, 2021 at 11:32 AM Dylan Jhong  wrote:
>
> On Wed, Mar 24, 2021 at 10:59:55PM +0800, Alistair Francis wrote:
> > On Tue, Mar 23, 2021 at 5:15 AM Dylan Jhong  wrote:
> > >
> > > Although the AE350 has not been upstream (preparing for v2),
> > > the reset vector of the AE350 is known to be at the 2G position,
> > > so this patch is corrected in advance.
> > >
> > > Signed-off-by: Dylan Jhong 
> > > Signed-off-by: Ruinland ChuanTzu Tsai 
> > > ---
> > >  target/riscv/cpu.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> > > index 2a990f6253..0236abf169 100644
> > > --- a/target/riscv/cpu.c
> > > +++ b/target/riscv/cpu.c
> > > @@ -137,7 +137,7 @@ static void set_feature(CPURISCVState *env, int 
> > > feature)
> > >  env->features |= (1ULL << feature);
> > >  }
> > >
> > > -static void set_resetvec(CPURISCVState *env, int resetvec)
> > > +static void set_resetvec(CPURISCVState *env, uint64_t resetvec)
> >
> > resetvec in env is a target_ulong so this should be as well (instead
> > of a uint64_t).
> >
> > Alistair
> >
>
> Hi Alistar,
>
> Thanks for your comments.
>
> Indeed resetvec should use target_ulong instead of uint64_t.

resetvec being target_ulong means that rv32 cannot have a reset vector
beyond 4GiB. I don't think the spec disallow this.

> But in target/riscv/cpu.h:306, there is also a resetvec in struct RISCVCPU 
> but it is defined as uint64_t.
> Do you think I should change it to target_ulong together?
>
> ref:
> commit 9b4c9b2b2a50fe4eb90d0ac2d8723b46ecb42511
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg730077.html

Regards,
Bin

Re: [PATCH] target/riscv: Align the data type of reset vector address

[Dylan Jhong]

On Wed, Mar 24, 2021 at 10:59:55PM +0800, Alistair Francis wrote:
> On Tue, Mar 23, 2021 at 5:15 AM Dylan Jhong  wrote:
> >
> > Although the AE350 has not been upstream (preparing for v2),
> > the reset vector of the AE350 is known to be at the 2G position,
> > so this patch is corrected in advance.
> >
> > Signed-off-by: Dylan Jhong 
> > Signed-off-by: Ruinland ChuanTzu Tsai 
> > ---
> >  target/riscv/cpu.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> > index 2a990f6253..0236abf169 100644
> > --- a/target/riscv/cpu.c
> > +++ b/target/riscv/cpu.c
> > @@ -137,7 +137,7 @@ static void set_feature(CPURISCVState *env, int feature)
> >  env->features |= (1ULL << feature);
> >  }
> >
> > -static void set_resetvec(CPURISCVState *env, int resetvec)
> > +static void set_resetvec(CPURISCVState *env, uint64_t resetvec)
> 
> resetvec in env is a target_ulong so this should be as well (instead
> of a uint64_t).
> 
> Alistair
>

Hi Alistar,

Thanks for your comments.

Indeed resetvec should use target_ulong instead of uint64_t.
But in target/riscv/cpu.h:306, there is also a resetvec in struct RISCVCPU but 
it is defined as uint64_t.
Do you think I should change it to target_ulong together?

ref: 
commit 9b4c9b2b2a50fe4eb90d0ac2d8723b46ecb42511
https://www.mail-archive.com/qemu-devel@nongnu.org/msg730077.html

> >  {
> >  #ifndef CONFIG_USER_ONLY
> >  env->resetvec = resetvec;
> > --
> > 2.17.1
> >
> >

Re: [PATCH qemu v16] spapr: Implement Open Firmware client interface

[Alexey Kardashevskiy]

On 25/03/2021 13:52, David Gibson wrote:

On Tue, Mar 23, 2021 at 01:58:30PM +1100, Alexey Kardashevskiy wrote:

The PAPR platform which describes an OS environment that's presented by
a combination of a hypervisor and firmware. The features it specifies
require collaboration between the firmware and the hypervisor.

Since the beginning, the runtime component of the firmware (RTAS) has
been implemented as a 20 byte shim which simply forwards it to
a hypercall implemented in qemu. The boot time firmware component is
SLOF - but a build that's specific to qemu, and has always needed to be
updated in sync with it. Even though we've managed to limit the amount
of runtime communication we need between qemu and SLOF, there's some,
and it has become increasingly awkward to handle as we've implemented
new features.

This implements a boot time OF client interface (CI) which is
enabled by a new "x-vof" pseries machine option (stands for "Virtual Open
Firmware). When enabled, QEMU implements the custom H_OF_CLIENT hcall
which implements Open Firmware Client Interface (OF CI). This allows
using a smaller stateless firmware which does not have to manage
the device tree.

The new "vof.bin" firmware image is included with source code under
pc-bios/. It also includes RTAS blob.

This implements a handful of CI methods just to get -kernel/-initrd
working. In particular, this implements the device tree fetching and
simple memory allocator - "claim" (an OF CI memory allocator) and updates
"/memory@0/available" to report the client about available memory.

This implements changing some device tree properties which we know how
to deal with, the rest is ignored. To allow changes, this skips
fdt_pack() when x-vof=on as not packing the blob leaves some room for
appending.

In absence of SLOF, this assigns phandles to device tree nodes to make
device tree traversing work.

When x-vof=on, this adds "/chosen" every time QEMU (re)builds a tree.

This adds basic instances support which are managed by a hash map
ihandle -> [phandle].

Before the guest started, the used memory is:
0..e60 - the initial firmware
8000..1 - stack
40.. - kernel
3ea.. - initramdisk

This OF CI does not implement "interpret".

Unlike SLOF, this does not format uninitialized nvram. Instead, this
includes a disk image with pre-formatted nvram.

With this basic support, this can only boot into kernel directly.
However this is just enough for the petitboot kernel and initradmdisk to
boot from any possible source. Note this requires reasonably recent guest
kernel with:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=df5be5be8735

The immediate benefit is much faster booting time which especially
crucial with fully emulated early CPU bring up environments. Also this
may come handy when/if GRUB-in-the-userspace sees light of the day.

This separates VOF and sPAPR in a hope that VOF bits may be reused by
other POWERPC boards which do not support pSeries.

This is coded in assumption that later on we might be adding support for
booting from QEMU backends (blockdev is the first candidate) without
devices/drivers in between as OF1275 does not require that and
it is quite easy to so.

Signed-off-by: Alexey Kardashevskiy 


I have some comments below, but they're basically all trivial at this
point.  We've missed qemu-6.0 obviously, but I'm hoping I can merge
the next spin to my ppc-for-6.1 tree.


---

The example command line is:

/home/aik/pbuild/qemu-killslof-localhost-ppc64/qemu-system-ppc64 \
-nodefaults \
-chardev stdio,id=STDIO0,signal=off,mux=on \
-device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
-mon id=MON0,chardev=STDIO0,mode=readline \
-nographic \
-vga none \
-enable-kvm \
-m 2G \
-machine 
pseries,x-vof=on,cap-cfpc=broken,cap-sbbc=broken,cap-ibs=broken,cap-ccf-assist=off
 \
-kernel pbuild/kernel-le-guest/vmlinux \
-initrd pb/rootfs.cpio.xz \
-drive 
id=DRIVE0,if=none,file=./p/qemu-killslof/pc-bios/vof-nvram.bin,format=raw \


Removing the need for a prebuild NVRAM image is something I'd like to
see as a followup.



We do not _need_ NVRAM in the VM to begin with, or is this a 
requirement? The whole VOF thing is more like a hack and I do not recall 
myself on doing anything useful with NVRAM.


If we really need it, then when to format it - in QEMU or VOF.bin? This 
alone will trigger a (lengthy) discussion :)




-global spapr-nvram.drive=DRIVE0 \
-snapshot \
-smp 8,threads=8 \
-L /home/aik/t/qemu-ppc64-bios/ \
-trace events=qemu_trace_events \
-d guest_errors \
-chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.tmux26 \
-mon chardev=SOCKET0,mode=control

---
Changes:
v16:
* rebased on dwg/ppc-for-6.1
* s/SpaprVofInterface/VofMachineInterface/

v15:
* bugfix: claimed memory for the VOF itself
* ditched OF_STACK_ADDR and allocate one instead, now it starts from 0x8000
because it is aligned to its size (no particular reason though)
* coding style
* moved nvram.bin up one level
* ditched bool in the fi

Re: [PATCH v3] i386/cpu_dump: support AVX512 ZMM regs dump

[Robert Hoo]

On Wed, 2021-03-24 at 07:44 -0600, Richard Henderson wrote:
> On 3/24/21 2:00 AM, Robert Hoo wrote:
> > +if ((env->xcr0 & XFEATURE_AVX512) == XFEATURE_AVX512) {
> > +/* XSAVE enabled AVX512 */
> > +nb = (env->hflags & HF_CS64_MASK) ? 32 : 8;
> > +for (i = 0; i < nb; i++) {
> > +qemu_fprintf(f, "ZMM%02d=0x%016lx %016lx %016lx
> > %016lx %016lx "
> > +"%016lx %016lx %016lx\n",
> > + i,
> > + env->xmm_regs[i].ZMM_Q(7),
> > + env->xmm_regs[i].ZMM_Q(6),
> > + env->xmm_regs[i].ZMM_Q(5),
> > + env->xmm_regs[i].ZMM_Q(4),
> > + env->xmm_regs[i].ZMM_Q(3),
> > + env->xmm_regs[i].ZMM_Q(2),
> > + env->xmm_regs[i].ZMM_Q(1),
> > + env->xmm_regs[i].ZMM_Q(0));
> > +}
> 
> Dump opmask regs?

OK
> 
> > +} else if (env->xcr0 & XFEATURE_AVX) {
> 
> This is normally a 2-bit test.

I beg your pardon. What 2 bits?
> 
> > +/* XSAVE enabled AVX */
> > +nb = env->hflags & HF_CS64_MASK ? 16 : 8;
> > +for (i = 0; i < nb; i++) {
> > +qemu_fprintf(f, "YMM%02d=0x%016lx %016lx %016lx
> > %016lx\n",
> > + i,
> > + env->xmm_regs[i].ZMM_Q(3),
> > + env->xmm_regs[i].ZMM_Q(2),
> > + env->xmm_regs[i].ZMM_Q(1),
> > + env->xmm_regs[i].ZMM_Q(0));
> > +}
> > +} else { /* SSE and below cases */
> > +nb = env->hflags & HF_CS64_MASK ? 16 : 8;
> > +for (i = 0; i < nb; i++) {
> > +qemu_fprintf(f, "XMM%02d=0x%016lx %016lx",
> > + i,
> > + env->xmm_regs[i].ZMM_Q(1),
> > + env->xmm_regs[i].ZMM_Q(0));
> > +if ((i & 1) == 1)
> > +qemu_fprintf(f, "\n");
> > +else
> > +qemu_fprintf(f, " ");
> 
> I'd be tempted to merge that second printf into the first, with "%s"
> and (i & 1 
> ? "\n" : " ").  Otherwise you'll need to add braces to that IF to
> satisfy 
> checkpatch.

Sure. I just retained previous code.
BTW, checkpatch didn't warn me on this. It escaped.:)
> 
> > +#define XFEATURE_X87(1UL << 0)
> > +#define XFEATURE_SSE(1UL << 1)
> > +#define XFEATURE_AVX(1UL << 2)
> > +#define XFEATURE_AVX512_OPMASK  (1UL << 5)
> > +#define XFEATURE_AVX512_ZMM_Hi256   (1UL << 6)
> > +#define XFEATURE_AVX512_Hi16_ZMM(1UL << 7)
> > +#define XFEATURE_AVX512 (XFEATURE_AVX512_OPMASK | \
> > + XFEATURE_AVX512_ZMM_Hi256 | \
> > + XFEATURE_AVX512_Hi16_ZMM)
> 
> Except for the last, these already exist under the name
> XSTATE_*_MASK.

Ah, my poor eye sight. They even exist in the same file. Thanks
pointing out.
> 
> I think you can just as well declare local variables to hold the 3
> bits for the 
> avx512 test and the 2 bits for the avx test.
> 
Sure.
> 
> r~

Re: [PATCH 1/2] spapr: number of SMP sockets must be equal to NUMA nodes

[David Gibson]

On Tue, Mar 23, 2021 at 02:21:33PM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 3/22/21 10:03 PM, David Gibson wrote:
> > On Fri, Mar 19, 2021 at 03:34:52PM -0300, Daniel Henrique Barboza wrote:
> > > Kernel commit 4bce545903fa ("powerpc/topology: Update
> > > topology_core_cpumask") cause a regression in the pseries machine when
> > > defining certain SMP topologies [1]. The reasoning behind the change is
> > > explained in kernel commit 4ca234a9cbd7 ("powerpc/smp: Stop updating
> > > cpu_core_mask"). In short, cpu_core_mask logic was causing troubles with
> > > large VMs with lots of CPUs and was changed by cpu_cpu_mask because, as
> > > far as the kernel understanding of SMP topologies goes, both masks are
> > > equivalent.
> > > 
> > > Further discussions in the kernel mailing list [2] shown that the
> > > powerpc kernel always considered that the number of sockets were equal
> > > to the number of NUMA nodes. The claim is that it doesn't make sense,
> > > for Power hardware at least, 2+ sockets being in the same NUMA node. The
> > > immediate conclusion is that all SMP topologies the pseries machine were
> > > supplying to the kernel, with more than one socket in the same NUMA node
> > > as in [1], happened to be correctly represented in the kernel by
> > > accident during all these years.
> > > 
> > > There's a case to be made for virtual topologies being detached from
> > > hardware constraints, allowing maximum flexibility to users. At the same
> > > time, this freedom can't result in unrealistic hardware representations
> > > being emulated. If the real hardware and the pseries kernel don't
> > > support multiple chips/sockets in the same NUMA node, neither should we.
> > > 
> > > Starting in 6.0.0, all sockets must match an unique NUMA node in the
> > > pseries machine. qtest changes were made to adapt to this new
> > > condition.
> > 
> > Oof.  I really don't like this idea.  It means a bunch of fiddly work
> > for users to match these up, for no real gain.  I'm also concerned
> > that this will require follow on changes in libvirt to not make this a
> > really cryptic and irritating point of failure.
> 
> Haven't though about required Libvirt changes, although I can say that there
> will be some amount to be mande and it will probably annoy existing users
> (everyone that has a multiple socket per NUMA node topology).
> 
> There is not much we can do from the QEMU layer aside from what I've proposed
> here. The other alternative is to keep interacting with the kernel folks to
> see if there is a way to keep our use case untouched.

Right.  Well.. not necessarily untouched, but I'm hoping for more
replies from Cédric to my objections and mpe's.  Even with sockets
being a kinda meaningless concept in PAPR, I don't think tying it to
NUMA nodes makes sense.

> This also means that
> 'ibm,chip-id' will probably remain in use since it's the only place where
> we inform cores per socket information to the kernel.

Well.. unless we can find some other sensible way to convey that
information.  I haven't given up hope for that yet.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

[PATCH 10/15] Hexagon (target/hexagon) instructions with multiple definitions

[Taylor Simpson]

Instructions with multiple definitions require special handling
because the generator wants to create a helper, but helpers can
only return a single result.  Therefore, we must override the
generated code.

The following instructions are added
A4_addp_cRdd32 = add(Rss32, Rtt32, Px4):carry
 Add with carry
A4_subp_cRdd32 = sub(Rss32, Rtt32, Px4):carry
 Sub with carry
A5_ACS   Rxx32,Pe4 = vacsh(Rss32, Rtt32)
 Add compare and select elements of two vectors
A6_vminub_RdPRdd32,Pe4 = vminub(Rtt32, Rss32)
 Vector min of bytes
F2_invsqrta  Rd32,Pe4 = sfinvsqrta(Rs32)
 Square root approx
F2_sfrecipa  Rd32,Pe4 = sfrecipa(Rs32, Rt32)
 Recripocal approx

Test cases in tests/tcg/hexagon/multi_result.c
FP exception tests added to tests/tcg/hexagon/fpstuff.c

Signed-off-by: Taylor Simpson 
---
 target/hexagon/arch.c |  57 +++-
 target/hexagon/arch.h |   2 +
 target/hexagon/gen_tcg.h  | 107 +++
 target/hexagon/genptr.c   | 111 +++
 target/hexagon/helper.h   |   6 +
 target/hexagon/imported/alu.idef  |  44 ++
 target/hexagon/imported/encode_pp.def |   6 +
 target/hexagon/imported/float.idef|  32 +
 target/hexagon/macros.h   |  41 ++
 target/hexagon/op_helper.c| 130 ++
 tests/tcg/hexagon/Makefile.target |   1 +
 tests/tcg/hexagon/fpstuff.c   |  97 +
 tests/tcg/hexagon/multi_result.c  | 249 ++
 13 files changed, 879 insertions(+), 4 deletions(-)
 create mode 100644 tests/tcg/hexagon/multi_result.c

diff --git a/target/hexagon/arch.c b/target/hexagon/arch.c
index 40b6e3d..d756dfa 100644
--- a/target/hexagon/arch.c
+++ b/target/hexagon/arch.c
@@ -181,12 +181,13 @@ int arch_sf_recip_common(float32 *Rs, float32 *Rt, 
float32 *Rd, int *adjust,
 /* or put Inf in num fixup? */
 uint8_t RsV_sign = float32_is_neg(RsV);
 uint8_t RtV_sign = float32_is_neg(RtV);
+/* Check that RsV is NOT infinite before we overwrite it */
+if (!float32_is_infinity(RsV)) {
+float_raise(float_flag_divbyzero, fp_status);
+}
 RsV = infinite_float32(RsV_sign ^ RtV_sign);
 RtV = float32_one;
 RdV = float32_one;
-if (float32_is_infinity(RsV)) {
-float_raise(float_flag_divbyzero, fp_status);
-}
 } else if (float32_is_infinity(RtV)) {
 RsV = make_float32(0x8000 & (RsV ^ RtV));
 RtV = float32_one;
@@ -246,7 +247,7 @@ int arch_sf_invsqrt_common(float32 *Rs, float32 *Rd, int 
*adjust,
 int r_exp;
 int ret = 0;
 RsV = *Rs;
-if (float32_is_infinity(RsV)) {
+if (float32_is_any_nan(RsV)) {
 if (extract32(RsV, 22, 1) == 0) {
 float_raise(float_flag_invalid, fp_status);
 }
@@ -279,3 +280,51 @@ int arch_sf_invsqrt_common(float32 *Rs, float32 *Rd, int 
*adjust,
 *adjust = PeV;
 return ret;
 }
+
+int arch_recip_lookup(int index)
+{
+index &= 0x7f;
+unsigned const int roundrom[128] = {
+0x0fe, 0x0fa, 0x0f6, 0x0f2, 0x0ef, 0x0eb, 0x0e7, 0x0e4,
+0x0e0, 0x0dd, 0x0d9, 0x0d6, 0x0d2, 0x0cf, 0x0cc, 0x0c9,
+0x0c6, 0x0c2, 0x0bf, 0x0bc, 0x0b9, 0x0b6, 0x0b3, 0x0b1,
+0x0ae, 0x0ab, 0x0a8, 0x0a5, 0x0a3, 0x0a0, 0x09d, 0x09b,
+0x098, 0x096, 0x093, 0x091, 0x08e, 0x08c, 0x08a, 0x087,
+0x085, 0x083, 0x080, 0x07e, 0x07c, 0x07a, 0x078, 0x075,
+0x073, 0x071, 0x06f, 0x06d, 0x06b, 0x069, 0x067, 0x065,
+0x063, 0x061, 0x05f, 0x05e, 0x05c, 0x05a, 0x058, 0x056,
+0x054, 0x053, 0x051, 0x04f, 0x04e, 0x04c, 0x04a, 0x049,
+0x047, 0x045, 0x044, 0x042, 0x040, 0x03f, 0x03d, 0x03c,
+0x03a, 0x039, 0x037, 0x036, 0x034, 0x033, 0x032, 0x030,
+0x02f, 0x02d, 0x02c, 0x02b, 0x029, 0x028, 0x027, 0x025,
+0x024, 0x023, 0x021, 0x020, 0x01f, 0x01e, 0x01c, 0x01b,
+0x01a, 0x019, 0x017, 0x016, 0x015, 0x014, 0x013, 0x012,
+0x011, 0x00f, 0x00e, 0x00d, 0x00c, 0x00b, 0x00a, 0x009,
+0x008, 0x007, 0x006, 0x005, 0x004, 0x003, 0x002, 0x000,
+};
+return roundrom[index];
+};
+
+int arch_invsqrt_lookup(int index)
+{
+index &= 0x7f;
+unsigned const int roundrom[128] = {
+0x069, 0x066, 0x063, 0x061, 0x05e, 0x05b, 0x059, 0x057,
+0x054, 0x052, 0x050, 0x04d, 0x04b, 0x049, 0x047, 0x045,
+0x043, 0x041, 0x03f, 0x03d, 0x03b, 0x039, 0x037, 0x036,
+0x034, 0x032, 0x030, 0x02f, 0x02d, 0x02c, 0x02a, 0x028,
+0x027, 0x025, 0x024, 0x022, 0x021, 0x01f, 0x01e, 0x01d,
+0x01b, 0x01a, 0x019, 0x017, 0x016, 0x015, 0x014, 0x012,
+0x011, 0x010, 0x00f, 0x00d, 0x00c, 0x00b, 0x00a, 0x009,
+0x008, 0x007, 0x006, 0x005, 0x004, 0x003, 0x002, 0x001,
+0x

[PATCH 01/15] Hexagon (target/hexagon) TCG generation cleanup

[Taylor Simpson]

Simplify TCG generation of hex_reg_written

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/genptr.c | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/target/hexagon/genptr.c b/target/hexagon/genptr.c
index 7481f4c..349b949 100644
--- a/target/hexagon/genptr.c
+++ b/target/hexagon/genptr.c
@@ -43,9 +43,15 @@ static inline void gen_log_predicated_reg_write(int rnum, 
TCGv val, int slot)
 tcg_gen_movcond_tl(TCG_COND_EQ, hex_new_value[rnum], slot_mask, zero,
val, hex_new_value[rnum]);
 #if HEX_DEBUG
-/* Do this so HELPER(debug_commit_end) will know */
-tcg_gen_movcond_tl(TCG_COND_EQ, hex_reg_written[rnum], slot_mask, zero,
-   one, hex_reg_written[rnum]);
+/*
+ * Do this so HELPER(debug_commit_end) will know
+ *
+ * Note that slot_mask indicates the value is not written
+ * (i.e., slot was cancelled), so we create a true/false value before
+ * or'ing with hex_reg_written[rnum].
+ */
+tcg_gen_setcond_tl(TCG_COND_EQ, slot_mask, slot_mask, zero);
+tcg_gen_or_tl(hex_reg_written[rnum], hex_reg_written[rnum], slot_mask);
 #endif
 
 tcg_temp_free(one);
-- 
2.7.4

[PATCH 02/15] Hexagon (target/hexagon) remove unnecessary inline directives

[Taylor Simpson]

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 linux-user/hexagon/cpu_loop.c |  2 +-
 target/hexagon/cpu.c  |  9 -
 target/hexagon/cpu.h  |  6 ++
 target/hexagon/decode.c   |  6 +++---
 target/hexagon/fma_emu.c  | 39 ---
 target/hexagon/op_helper.c| 39 +++
 target/hexagon/translate.c|  2 +-
 7 files changed, 50 insertions(+), 53 deletions(-)

diff --git a/linux-user/hexagon/cpu_loop.c b/linux-user/hexagon/cpu_loop.c
index 9a68ca0..a752a0a 100644
--- a/linux-user/hexagon/cpu_loop.c
+++ b/linux-user/hexagon/cpu_loop.c
@@ -25,7 +25,7 @@
 
 void cpu_loop(CPUHexagonState *env)
 {
-CPUState *cs = CPU(hexagon_env_get_cpu(env));
+CPUState *cs = CPUSTATE_FROM_ENV(env);
 int trapnr, signum, sigcode;
 target_ulong sigaddr;
 target_ulong syscallnum;
diff --git a/target/hexagon/cpu.c b/target/hexagon/cpu.c
index b0b3040..002275a 100644
--- a/target/hexagon/cpu.c
+++ b/target/hexagon/cpu.c
@@ -69,10 +69,9 @@ const char * const hexagon_regnames[TOTAL_PER_THREAD_REGS] = 
{
  * stacks at different locations.  This is used to compensate so the diff is
  * cleaner.
  */
-static inline target_ulong adjust_stack_ptrs(CPUHexagonState *env,
- target_ulong addr)
+static target_ulong adjust_stack_ptrs(CPUHexagonState *env, target_ulong addr)
 {
-HexagonCPU *cpu = container_of(env, HexagonCPU, env);
+HexagonCPU *cpu = HEXAGONCPU_FROM_ENV(env);
 target_ulong stack_adjust = cpu->lldb_stack_adjust;
 target_ulong stack_start = env->stack_start;
 target_ulong stack_size = 0x1;
@@ -88,7 +87,7 @@ static inline target_ulong adjust_stack_ptrs(CPUHexagonState 
*env,
 }
 
 /* HEX_REG_P3_0 (aka C4) is an alias for the predicate registers */
-static inline target_ulong read_p3_0(CPUHexagonState *env)
+static target_ulong read_p3_0(CPUHexagonState *env)
 {
 int32_t control_reg = 0;
 int i;
@@ -116,7 +115,7 @@ static void print_reg(FILE *f, CPUHexagonState *env, int 
regnum)
 
 static void hexagon_dump(CPUHexagonState *env, FILE *f)
 {
-HexagonCPU *cpu = container_of(env, HexagonCPU, env);
+HexagonCPU *cpu = HEXAGONCPU_FROM_ENV(env);
 
 if (cpu->lldb_compat) {
 /*
diff --git a/target/hexagon/cpu.h b/target/hexagon/cpu.h
index e04eac5..433f420 100644
--- a/target/hexagon/cpu.h
+++ b/target/hexagon/cpu.h
@@ -127,10 +127,8 @@ typedef struct HexagonCPU {
 target_ulong lldb_stack_adjust;
 } HexagonCPU;
 
-static inline HexagonCPU *hexagon_env_get_cpu(CPUHexagonState *env)
-{
-return container_of(env, HexagonCPU, env);
-}
+#define HEXAGONCPU_FROM_ENV(env)container_of((env), HexagonCPU, env)
+#define CPUSTATE_FROM_ENV(env)  CPU(HEXAGONCPU_FROM_ENV(env))
 
 #include "cpu_bits.h"
 
diff --git a/target/hexagon/decode.c b/target/hexagon/decode.c
index 1c9c074..65d97ce 100644
--- a/target/hexagon/decode.c
+++ b/target/hexagon/decode.c
@@ -354,7 +354,7 @@ static void decode_split_cmpjump(Packet *pkt)
 }
 }
 
-static inline int decode_opcode_can_jump(int opcode)
+static int decode_opcode_can_jump(int opcode)
 {
 if ((GET_ATTRIB(opcode, A_JUMP)) ||
 (GET_ATTRIB(opcode, A_CALL)) ||
@@ -370,7 +370,7 @@ static inline int decode_opcode_can_jump(int opcode)
 return 0;
 }
 
-static inline int decode_opcode_ends_loop(int opcode)
+static int decode_opcode_ends_loop(int opcode)
 {
 return GET_ATTRIB(opcode, A_HWLOOP0_END) ||
GET_ATTRIB(opcode, A_HWLOOP1_END);
@@ -764,7 +764,7 @@ static void decode_add_endloop_insn(Insn *insn, int loopnum)
 }
 }
 
-static inline int decode_parsebits_is_loopend(uint32_t encoding32)
+static int decode_parsebits_is_loopend(uint32_t encoding32)
 {
 uint32_t bits = parse_bits(encoding32);
 return bits == 0x2;
diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c
index 842d903..f324b83 100644
--- a/target/hexagon/fma_emu.c
+++ b/target/hexagon/fma_emu.c
@@ -64,7 +64,7 @@ typedef union {
 };
 } Float;
 
-static inline uint64_t float64_getmant(float64 f64)
+static uint64_t float64_getmant(float64 f64)
 {
 Double a = { .i = f64 };
 if (float64_is_normal(f64)) {
@@ -91,7 +91,7 @@ int32_t float64_getexp(float64 f64)
 return -1;
 }
 
-static inline uint64_t float32_getmant(float32 f32)
+static uint64_t float32_getmant(float32 f32)
 {
 Float a = { .i = f32 };
 if (float32_is_normal(f32)) {
@@ -118,17 +118,17 @@ int32_t float32_getexp(float32 f32)
 return -1;
 }
 
-static inline uint32_t int128_getw0(Int128 x)
+static uint32_t int128_getw0(Int128 x)
 {
 return int128_getlo(x);
 }
 
-static inline uint32_t int128_getw1(Int128 x)
+static uint32_t int128_getw1(Int128 x)
 {
 return int128_getlo(x) >> 32;
 }
 
-static inline Int128 int128_mul_6464(uint64_t ai, uint64_t bi)
+static Int128 int128_mul_6464(uint64_t ai, uint64_t bi)
 {
 Int128 a, b;
 uint64_t pp0, pp1a, pp1b, pp1s, pp2

[PATCH 07/15] Hexagon (target/hexagon) use softfloat default NaN and tininess

[Taylor Simpson]

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 fpu/softfloat-specialize.c.inc |  5 +
 target/hexagon/cpu.c   |  5 +
 target/hexagon/op_helper.c | 47 --
 3 files changed, 10 insertions(+), 47 deletions(-)

diff --git a/fpu/softfloat-specialize.c.inc b/fpu/softfloat-specialize.c.inc
index c2f87ad..c7ab148 100644
--- a/fpu/softfloat-specialize.c.inc
+++ b/fpu/softfloat-specialize.c.inc
@@ -145,6 +145,9 @@ static FloatParts parts_default_nan(float_status *status)
 #elif defined(TARGET_HPPA)
 /* snan_bit_is_one, set msb-1.  */
 frac = 1ULL << (DECOMPOSED_BINARY_POINT - 2);
+#elif defined(TARGET_HEXAGON)
+sign = 1;
+frac = ~0ULL;
 #else
 /* This case is true for Alpha, ARM, MIPS, OpenRISC, PPC, RISC-V,
  * S390, SH4, TriCore, and Xtensa.  I cannot find documentation
@@ -180,6 +183,8 @@ static FloatParts parts_silence_nan(FloatParts a, 
float_status *status)
 #if defined(TARGET_HPPA)
 a.frac &= ~(1ULL << (DECOMPOSED_BINARY_POINT - 1));
 a.frac |= 1ULL << (DECOMPOSED_BINARY_POINT - 2);
+#elif defined(TARGET_HEXAGON)
+return parts_default_nan(status);
 #else
 if (snan_bit_is_one(status)) {
 return parts_default_nan(status);
diff --git a/target/hexagon/cpu.c b/target/hexagon/cpu.c
index 002275a..ed829ce 100644
--- a/target/hexagon/cpu.c
+++ b/target/hexagon/cpu.c
@@ -23,6 +23,7 @@
 #include "exec/exec-all.h"
 #include "qapi/error.h"
 #include "hw/qdev-properties.h"
+#include "fpu/softfloat-helpers.h"
 
 static void hexagon_v67_cpu_init(Object *obj)
 {
@@ -205,8 +206,12 @@ static void hexagon_cpu_reset(DeviceState *dev)
 CPUState *cs = CPU(dev);
 HexagonCPU *cpu = HEXAGON_CPU(cs);
 HexagonCPUClass *mcc = HEXAGON_CPU_GET_CLASS(cpu);
+CPUHexagonState *env = &cpu->env;
 
 mcc->parent_reset(dev);
+
+set_default_nan_mode(1, &env->fp_status);
+set_float_detect_tininess(float_tininess_before_rounding, &env->fp_status);
 }
 
 static void hexagon_cpu_disas_set_info(CPUState *s, disassemble_info *info)
diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c
index 6a364ea..2ceca39 100644
--- a/target/hexagon/op_helper.c
+++ b/target/hexagon/op_helper.c
@@ -297,26 +297,6 @@ int32_t HELPER(fcircadd)(int32_t RxV, int32_t offset, 
int32_t M, int32_t CS)
 }
 
 /*
- * Hexagon FP operations return ~0 instead of NaN
- * The hex_check_sfnan/hex_check_dfnan functions perform this check
- */
-static float32 hex_check_sfnan(float32 x)
-{
-if (float32_is_any_nan(x)) {
-return make_float32(0xU);
-}
-return x;
-}
-
-static float64 hex_check_dfnan(float64 x)
-{
-if (float64_is_any_nan(x)) {
-return make_float64(0xULL);
-}
-return x;
-}
-
-/*
  * mem_noshuf
  * Section 5.5 of the Hexagon V67 Programmer's Reference Manual
  *
@@ -373,7 +353,6 @@ float64 HELPER(conv_sf2df)(CPUHexagonState *env, float32 
RsV)
 float64 out_f64;
 arch_fpop_start(env);
 out_f64 = float32_to_float64(RsV, &env->fp_status);
-out_f64 = hex_check_dfnan(out_f64);
 arch_fpop_end(env);
 return out_f64;
 }
@@ -383,7 +362,6 @@ float32 HELPER(conv_df2sf)(CPUHexagonState *env, float64 
RssV)
 float32 out_f32;
 arch_fpop_start(env);
 out_f32 = float64_to_float32(RssV, &env->fp_status);
-out_f32 = hex_check_sfnan(out_f32);
 arch_fpop_end(env);
 return out_f32;
 }
@@ -393,7 +371,6 @@ float32 HELPER(conv_uw2sf)(CPUHexagonState *env, int32_t 
RsV)
 float32 RdV;
 arch_fpop_start(env);
 RdV = uint32_to_float32(RsV, &env->fp_status);
-RdV = hex_check_sfnan(RdV);
 arch_fpop_end(env);
 return RdV;
 }
@@ -403,7 +380,6 @@ float64 HELPER(conv_uw2df)(CPUHexagonState *env, int32_t 
RsV)
 float64 RddV;
 arch_fpop_start(env);
 RddV = uint32_to_float64(RsV, &env->fp_status);
-RddV = hex_check_dfnan(RddV);
 arch_fpop_end(env);
 return RddV;
 }
@@ -413,7 +389,6 @@ float32 HELPER(conv_w2sf)(CPUHexagonState *env, int32_t RsV)
 float32 RdV;
 arch_fpop_start(env);
 RdV = int32_to_float32(RsV, &env->fp_status);
-RdV = hex_check_sfnan(RdV);
 arch_fpop_end(env);
 return RdV;
 }
@@ -423,7 +398,6 @@ float64 HELPER(conv_w2df)(CPUHexagonState *env, int32_t RsV)
 float64 RddV;
 arch_fpop_start(env);
 RddV = int32_to_float64(RsV, &env->fp_status);
-RddV = hex_check_dfnan(RddV);
 arch_fpop_end(env);
 return RddV;
 }
@@ -433,7 +407,6 @@ float32 HELPER(conv_ud2sf)(CPUHexagonState *env, int64_t 
RssV)
 float32 RdV;
 arch_fpop_start(env);
 RdV = uint64_to_float32(RssV, &env->fp_status);
-RdV = hex_check_sfnan(RdV);
 arch_fpop_end(env);
 return RdV;
 }
@@ -443,7 +416,6 @@ float64 HELPER(conv_ud2df)(CPUHexagonState *env, int64_t 
RssV)
 float64 RddV;
 arch_fpop_start(env);
 RddV = uint64_to_float64(RssV, &env->fp_status);
-RddV = hex_check_dfnan(RddV);
 arch_fpop_end(env);
 return RddV;
 }

[PATCH 13/15] Hexagon (target/hexagon) load and unpack bytes instructions

[Taylor Simpson]

The following instructions are added
L2_loadbzw2_io  Rd32 = memubh(Rs32+#s11:1)
L2_loadbzw4_io  Rdd32 = memubh(Rs32+#s11:1)
L2_loadbsw2_io  Rd32 = membh(Rs32+#s11:1)
L2_loadbsw4_io  Rdd32 = membh(Rs32+#s11:1)

L4_loadbzw2_ur  Rd32 = memubh(Rt32<<#u2+#U6)
L4_loadbzw4_ur  Rdd32 = memubh(Rt32<<#u2+#U6)
L4_loadbsw2_ur  Rd32 = membh(Rt32<<#u2+#U6)
L4_loadbsw4_ur  Rdd32 = membh(Rt32<<#u2+#U6)

L4_loadbzw2_ap  Rd32 = memubh(Re32=#U6)
L4_loadbzw4_ap  Rdd32 = memubh(Re32=#U6)
L4_loadbsw2_ap  Rd32 = membh(Re32=#U6)
L4_loadbsw4_ap  Rdd32 = membh(Re32=#U6)

L2_loadbzw2_pr  Rd32 = memubh(Rx32++Mu2)
L2_loadbzw4_pr  Rdd32 = memubh(Rx32++Mu2)
L2_loadbsw2_pr  Rd32 = membh(Rx32++Mu2)
L2_loadbsw4_pr  Rdd32 = membh(Rx32++Mu2)

L2_loadbzw2_pbr Rd32 = memubh(Rx32++Mu2:brev)
L2_loadbzw4_pbr Rdd32 = memubh(Rx32++Mu2:brev)
L2_loadbsw2_pbr Rd32 = membh(Rx32++Mu2:brev)
L2_loadbsw4_pbr Rdd32 = membh(Rx32++Mu2:brev)

L2_loadbzw2_pi  Rd32 = memubh(Rx32++#s4:1)
L2_loadbzw4_pi  Rdd32 = memubh(Rx32++#s4:1)
L2_loadbsw2_pi  Rd32 = membh(Rx32++#s4:1)
L2_loadbsw4_pi  Rdd32 = membh(Rx32++#s4:1)

L2_loadbzw2_pci Rd32 = memubh(Rx32++#s4:1:circ(Mu2))
L2_loadbzw4_pci Rdd32 = memubh(Rx32++#s4:1:circ(Mu2))
L2_loadbsw2_pci Rd32 = membh(Rx32++#s4:1:circ(Mu2))
L2_loadbsw4_pci Rdd32 = membh(Rx32++#s4:1:circ(Mu2))

L2_loadbzw2_pcr Rd32 = memubh(Rx32++I:circ(Mu2))
L2_loadbzw4_pcr Rdd32 = memubh(Rx32++I:circ(Mu2))
L2_loadbsw2_pcr Rd32 = membh(Rx32++I:circ(Mu2))
L2_loadbsw4_pcr Rdd32 = membh(Rx32++I:circ(Mu2))

Test cases in tests/tcg/hexagon/load_unpack.c

Signed-off-by: Taylor Simpson 
---
 target/hexagon/gen_tcg.h  | 118 +
 target/hexagon/genptr.c   |  13 +
 target/hexagon/imported/encode_pp.def |   6 +
 target/hexagon/imported/ldst.idef |  43 +++
 target/hexagon/macros.h   |  27 +-
 tests/tcg/hexagon/Makefile.target |   1 +
 tests/tcg/hexagon/load_unpack.c   | 474 ++
 7 files changed, 681 insertions(+), 1 deletion(-)
 create mode 100644 tests/tcg/hexagon/load_unpack.c

diff --git a/target/hexagon/gen_tcg.h b/target/hexagon/gen_tcg.h
index 038e16e..55ba5f2 100644
--- a/target/hexagon/gen_tcg.h
+++ b/target/hexagon/gen_tcg.h
@@ -167,6 +167,124 @@
 #define fGEN_TCG_L2_loadrd_pi(SHORTCODE)   SHORTCODE
 
 /*
+ * These instructions load 2 bytes and places them in
+ * two halves of the destination register.
+ * The GET_EA macro determines the addressing mode.
+ * The fGB macro determines whether to zero-extend or
+ * sign-extend.
+ */
+#define fGEN_TCG_loadbXw2(GET_EA, fGB) \
+do { \
+TCGv ireg = tcg_temp_new(); \
+TCGv tmp = tcg_temp_new(); \
+TCGv tmpV = tcg_temp_new(); \
+TCGv BYTE = tcg_temp_new(); \
+int i; \
+GET_EA; \
+fLOAD(1, 2, u, EA, tmpV); \
+tcg_gen_movi_tl(RdV, 0); \
+for (i = 0; i < 2; i++) { \
+fSETHALF(i, RdV, fGB(i, tmpV)); \
+} \
+tcg_temp_free(ireg); \
+tcg_temp_free(tmp); \
+tcg_temp_free(tmpV); \
+tcg_temp_free(BYTE); \
+} while (0)
+
+#define fGEN_TCG_L2_loadbzw2_io(SHORTCODE) \
+fGEN_TCG_loadbXw2(fEA_RI(RsV, siV), fGETUBYTE)
+#define fGEN_TCG_L4_loadbzw2_ur(SHORTCODE) \
+fGEN_TCG_loadbXw2(fEA_IRs(UiV, RtV, uiV), fGETUBYTE)
+#define fGEN_TCG_L2_loadbsw2_io(SHORTCODE) \
+fGEN_TCG_loadbXw2(fEA_RI(RsV, siV), fGETBYTE)
+#define fGEN_TCG_L4_loadbsw2_ur(SHORTCODE) \
+fGEN_TCG_loadbXw2(fEA_IRs(UiV, RtV, uiV), fGETBYTE)
+#define fGEN_TCG_L4_loadbzw2_ap(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_ap, fGETUBYTE)
+#define fGEN_TCG_L2_loadbzw2_pr(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pr, fGETUBYTE)
+#define fGEN_TCG_L2_loadbzw2_pbr(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pbr, fGETUBYTE)
+#define fGEN_TCG_L2_loadbzw2_pi(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pi, fGETUBYTE)
+#define fGEN_TCG_L4_loadbsw2_ap(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_ap, fGETBYTE)
+#define fGEN_TCG_L2_loadbsw2_pr(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pr, fGETBYTE)
+#define fGEN_TCG_L2_loadbsw2_pbr(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pbr, fGETBYTE)
+#define fGEN_TCG_L2_loadbsw2_pi(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pi, fGETBYTE)
+#define fGEN_TCG_L2_loadbzw2_pci(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pci, fGETUBYTE)
+#define fGEN_TCG_L2_loadbsw2_pci(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pci, fGETBYTE)
+#define fGEN_TCG_L2_loadbzw2_pcr(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pcr(1), fGETUBYTE)
+#define fGEN_TCG_L2_loadbsw2_pcr(SHORTCODE) \
+fGEN_TCG_loadbXw2(GET_EA_pcr(1), fGETBYTE)
+
+/*
+ * These instructions lo

[PATCH 06/15] Hexagon (target/hexagon) utility function changes

[Taylor Simpson]

Remove unused carry_from_add64 function
Change type of softfloat_roundingmodes

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/arch.c   | 15 +--
 target/hexagon/arch.h   |  1 -
 target/hexagon/macros.h |  2 --
 3 files changed, 1 insertion(+), 17 deletions(-)

diff --git a/target/hexagon/arch.c b/target/hexagon/arch.c
index 09de124..bb51f19 100644
--- a/target/hexagon/arch.c
+++ b/target/hexagon/arch.c
@@ -76,19 +76,6 @@ uint64_t deinterleave(uint64_t src)
 return myeven | (myodd << 32);
 }
 
-uint32_t carry_from_add64(uint64_t a, uint64_t b, uint32_t c)
-{
-uint64_t tmpa, tmpb, tmpc;
-tmpa = fGETUWORD(0, a);
-tmpb = fGETUWORD(0, b);
-tmpc = tmpa + tmpb + c;
-tmpa = fGETUWORD(1, a);
-tmpb = fGETUWORD(1, b);
-tmpc = tmpa + tmpb + fGETUWORD(1, tmpc);
-tmpc = fGETUWORD(1, tmpc);
-return tmpc;
-}
-
 int32_t conv_round(int32_t a, int n)
 {
 int64_t val;
@@ -108,7 +95,7 @@ int32_t conv_round(int32_t a, int n)
 
 /* Floating Point Stuff */
 
-static const int softfloat_roundingmodes[] = {
+static const FloatRoundMode softfloat_roundingmodes[] = {
 float_round_nearest_even,
 float_round_to_zero,
 float_round_down,
diff --git a/target/hexagon/arch.h b/target/hexagon/arch.h
index 1f7f036..6e0b0d9 100644
--- a/target/hexagon/arch.h
+++ b/target/hexagon/arch.h
@@ -22,7 +22,6 @@
 
 uint64_t interleave(uint32_t odd, uint32_t even);
 uint64_t deinterleave(uint64_t src);
-uint32_t carry_from_add64(uint64_t a, uint64_t b, uint32_t c);
 int32_t conv_round(int32_t a, int n);
 void arch_fpop_start(CPUHexagonState *env);
 void arch_fpop_end(CPUHexagonState *env);
diff --git a/target/hexagon/macros.h b/target/hexagon/macros.h
index cfcb817..8cb211d 100644
--- a/target/hexagon/macros.h
+++ b/target/hexagon/macros.h
@@ -341,8 +341,6 @@ static inline void gen_logical_not(TCGv dest, TCGv src)
 #define fWRITE_LC0(VAL) WRITE_RREG(HEX_REG_LC0, VAL)
 #define fWRITE_LC1(VAL) WRITE_RREG(HEX_REG_LC1, VAL)
 
-#define fCARRY_FROM_ADD(A, B, C) carry_from_add64(A, B, C)
-
 #define fSET_OVERFLOW() SET_USR_FIELD(USR_OVF, 1)
 #define fSET_LPCFG(VAL) SET_USR_FIELD(USR_LPCFG, (VAL))
 #define fGET_LPCFG (GET_USR_FIELD(USR_LPCFG))
-- 
2.7.4

[PATCH 14/15] Hexagon (target/hexagon) load into shifted register instructions

[Taylor Simpson]

The following instructions are added
L2_loadalignb_io  Ryy32 = memb_fifo(Rs32+#s11:1)
L2_loadalignh_io  Ryy32 = memh_fifo(Rs32+#s11:1)
L4_loadalignb_ur  Ryy32 = memb_fifo(Rt32<<#u2+#U6)
L4_loadalignh_ur  Ryy32 = memh_fifo(Rt32<<#u2+#U6)
L4_loadalignb_ap  Ryy32 = memb_fifo(Re32=#U6)
L4_loadalignh_ap  Ryy32 = memh_fifo(Re32=#U6)
L2_loadalignb_pr  Ryy32 = memb_fifo(Rx32++Mu2)
L2_loadalignh_pr  Ryy32 = memh_fifo(Rx32++Mu2)
L2_loadalignb_pbr Ryy32 = memb_fifo(Rx32++Mu2:brev)
L2_loadalignh_pbr Ryy32 = memh_fifo(Rx32++Mu2:brev)
L2_loadalignb_pi  Ryy32 = memb_fifo(Rx32++#s4:1)
L2_loadalignh_pi  Ryy32 = memh_fifo(Rx32++#s4:1)
L2_loadalignb_pci Ryy32 = memb_fifo(Rx32++#s4:1:circ(Mu2))
L2_loadalignh_pci Ryy32 = memh_fifo(Rx32++#s4:1:circ(Mu2))
L2_loadalignb_pcr Ryy32 = memb_fifo(Rx32++I:circ(Mu2))
L2_loadalignh_pcr Ryy32 = memh_fifo(Rx32++I:circ(Mu2))

Test cases in tests/tcg/hexagon/load_align.c

Signed-off-by: Taylor Simpson 
---
 target/hexagon/gen_tcg.h  |  78 +++
 target/hexagon/imported/encode_pp.def |   3 +
 target/hexagon/imported/ldst.idef |  19 ++
 tests/tcg/hexagon/Makefile.target |   1 +
 tests/tcg/hexagon/load_align.c| 415 ++
 5 files changed, 516 insertions(+)
 create mode 100644 tests/tcg/hexagon/load_align.c

diff --git a/target/hexagon/gen_tcg.h b/target/hexagon/gen_tcg.h
index 55ba5f2..79ec060 100644
--- a/target/hexagon/gen_tcg.h
+++ b/target/hexagon/gen_tcg.h
@@ -285,6 +285,84 @@
 fGEN_TCG_loadbXw4(GET_EA_pi, fGETBYTE)
 
 /*
+ * These instructions load a half word, shift the destination right by 16 bits
+ * and place the loaded value in the high half word of the destination pair.
+ * The GET_EA macro determines the addressing mode.
+ */
+#define fGEN_TCG_loadalignh(GET_EA) \
+do { \
+TCGv ireg = tcg_temp_new(); \
+TCGv tmp = tcg_temp_new(); \
+TCGv tmpV = tcg_temp_new(); \
+TCGv_i64 tmp_i64 = tcg_temp_new_i64(); \
+tcg_gen_concat_i32_i64(RyyV, hex_gpr[RyyN], hex_gpr[RyyN + 1]); \
+GET_EA;  \
+fLOAD(1, 2, u, EA, tmpV);  \
+tcg_gen_extu_i32_i64(tmp_i64, tmpV); \
+tcg_gen_shli_i64(tmp_i64, tmp_i64, 48); \
+tcg_gen_shri_i64(RyyV, RyyV, 16); \
+tcg_gen_or_i64(RyyV, RyyV, tmp_i64); \
+tcg_temp_free(ireg); \
+tcg_temp_free(tmp); \
+tcg_temp_free(tmpV); \
+tcg_temp_free_i64(tmp_i64); \
+} while (0)
+
+#define fGEN_TCG_L4_loadalignh_ur(SHORTCODE) \
+fGEN_TCG_loadalignh(fEA_IRs(UiV, RtV, uiV))
+#define fGEN_TCG_L2_loadalignh_io(SHORTCODE) \
+fGEN_TCG_loadalignh(fEA_RI(RsV, siV))
+#define fGEN_TCG_L2_loadalignh_pci(SHORTCODE) \
+fGEN_TCG_loadalignh(GET_EA_pci)
+#define fGEN_TCG_L2_loadalignh_pcr(SHORTCODE) \
+fGEN_TCG_loadalignh(GET_EA_pcr(1))
+#define fGEN_TCG_L4_loadalignh_ap(SHORTCODE) \
+fGEN_TCG_loadalignh(GET_EA_ap)
+#define fGEN_TCG_L2_loadalignh_pr(SHORTCODE) \
+fGEN_TCG_loadalignh(GET_EA_pr)
+#define fGEN_TCG_L2_loadalignh_pbr(SHORTCODE) \
+fGEN_TCG_loadalignh(GET_EA_pbr)
+#define fGEN_TCG_L2_loadalignh_pi(SHORTCODE) \
+fGEN_TCG_loadalignh(GET_EA_pi)
+
+/* Same as above, but loads a byte instead of half word */
+#define fGEN_TCG_loadalignb(GET_EA) \
+do { \
+TCGv ireg = tcg_temp_new(); \
+TCGv tmp = tcg_temp_new(); \
+TCGv tmpV = tcg_temp_new(); \
+TCGv_i64 tmp_i64 = tcg_temp_new_i64(); \
+tcg_gen_concat_i32_i64(RyyV, hex_gpr[RyyN], hex_gpr[RyyN + 1]); \
+GET_EA;  \
+fLOAD(1, 1, u, EA, tmpV);  \
+tcg_gen_extu_i32_i64(tmp_i64, tmpV); \
+tcg_gen_shli_i64(tmp_i64, tmp_i64, 56); \
+tcg_gen_shri_i64(RyyV, RyyV, 8); \
+tcg_gen_or_i64(RyyV, RyyV, tmp_i64); \
+tcg_temp_free(ireg); \
+tcg_temp_free(tmp); \
+tcg_temp_free(tmpV); \
+tcg_temp_free_i64(tmp_i64); \
+} while (0)
+
+#define fGEN_TCG_L2_loadalignb_io(SHORTCODE) \
+fGEN_TCG_loadalignb(fEA_RI(RsV, siV))
+#define fGEN_TCG_L4_loadalignb_ur(SHORTCODE) \
+fGEN_TCG_loadalignb(fEA_IRs(UiV, RtV, uiV))
+#define fGEN_TCG_L2_loadalignb_pci(SHORTCODE) \
+fGEN_TCG_loadalignb(GET_EA_pci)
+#define fGEN_TCG_L2_loadalignb_pcr(SHORTCODE) \
+fGEN_TCG_loadalignb(GET_EA_pcr(0))
+#define fGEN_TCG_L4_loadalignb_ap(SHORTCODE) \
+fGEN_TCG_loadalignb(GET_EA_ap)
+#define fGEN_TCG_L2_loadalignb_pr(SHORTCODE) \
+fGEN_TCG_loadalignb(GET_EA_pr)
+#define fGEN_TCG_L2_loadalignb_pbr(SHORTCODE) \
+fGEN_TCG_loadalignb(GET_EA_pbr)
+#define fGEN_TCG_L2_loadalignb_pi(SHORTCODE) \
+fGEN_TCG_loadalignb(GET_EA_pi)
+
+/*
  * Predicated loads
  * Here is a primer to understand the tag names
  *
diff --git a/target/hexagon/imported/encode_pp.def 
b/target/hexagon/imported/encode_pp.def
index e3582eb..dc4eba4 100644
--- a/targ

[PATCH 15/15] Hexagon (target/hexagon) CABAC decode bin

[Taylor Simpson]

The following instruction is added
S2_cabacdecbinRdd32=decbin(Rss32,Rtt32)

Test cases added to tests/tcg/hexagon/misc.c

Signed-off-by: Taylor Simpson 
---
 target/hexagon/arch.c | 91 +++
 target/hexagon/arch.h |  4 ++
 target/hexagon/imported/encode_pp.def |  1 +
 target/hexagon/imported/macros.def| 15 ++
 target/hexagon/imported/shift.idef| 47 ++
 target/hexagon/macros.h   |  4 ++
 tests/tcg/hexagon/misc.c  | 28 +++
 7 files changed, 190 insertions(+)

diff --git a/target/hexagon/arch.c b/target/hexagon/arch.c
index d756dfa..35fdcba 100644
--- a/target/hexagon/arch.c
+++ b/target/hexagon/arch.c
@@ -27,6 +27,97 @@
 #define SF_MANTBITS23
 #define float32_nanmake_float32(0x)
 
+/*
+ * These three tables are used by the cabacdecbin instruction
+ */
+const uint8_t rLPS_table_64x4[64][4] = {
+{128, 176, 208, 240},
+{128, 167, 197, 227},
+{128, 158, 187, 216},
+{123, 150, 178, 205},
+{116, 142, 169, 195},
+{111, 135, 160, 185},
+{105, 128, 152, 175},
+{100, 122, 144, 166},
+{95, 116, 137, 158},
+{90, 110, 130, 150},
+{85, 104, 123, 142},
+{81, 99, 117, 135},
+{77, 94, 111, 128},
+{73, 89, 105, 122},
+{69, 85, 100, 116},
+{66, 80, 95, 110},
+{62, 76, 90, 104},
+{59, 72, 86, 99},
+{56, 69, 81, 94},
+{53, 65, 77, 89},
+{51, 62, 73, 85},
+{48, 59, 69, 80},
+{46, 56, 66, 76},
+{43, 53, 63, 72},
+{41, 50, 59, 69},
+{39, 48, 56, 65},
+{37, 45, 54, 62},
+{35, 43, 51, 59},
+{33, 41, 48, 56},
+{32, 39, 46, 53},
+{30, 37, 43, 50},
+{29, 35, 41, 48},
+{27, 33, 39, 45},
+{26, 31, 37, 43},
+{24, 30, 35, 41},
+{23, 28, 33, 39},
+{22, 27, 32, 37},
+{21, 26, 30, 35},
+{20, 24, 29, 33},
+{19, 23, 27, 31},
+{18, 22, 26, 30},
+{17, 21, 25, 28},
+{16, 20, 23, 27},
+{15, 19, 22, 25},
+{14, 18, 21, 24},
+{14, 17, 20, 23},
+{13, 16, 19, 22},
+{12, 15, 18, 21},
+{12, 14, 17, 20},
+{11, 14, 16, 19},
+{11, 13, 15, 18},
+{10, 12, 15, 17},
+{10, 12, 14, 16},
+{9, 11, 13, 15},
+{9, 11, 12, 14},
+{8, 10, 12, 14},
+{8, 9, 11, 13},
+{7, 9, 11, 12},
+{7, 9, 10, 12},
+{7, 8, 10, 11},
+{6, 8, 9, 11},
+{6, 7, 9, 10},
+{6, 7, 8, 9},
+{2, 2, 2, 2}
+};
+
+const uint8_t AC_next_state_MPS_64[64] = {
+1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
+11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
+21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
+31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
+41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
+51, 52, 53, 54, 55, 56, 57, 58, 59, 60,
+61, 62, 62, 63
+};
+
+
+const uint8_t AC_next_state_LPS_64[64] = {
+0, 0, 1, 2, 2, 4, 4, 5, 6, 7,
+8, 9, 9, 11, 11, 12, 13, 13, 15, 15,
+16, 16, 18, 18, 19, 19, 21, 21, 22, 22,
+23, 24, 24, 25, 26, 26, 27, 27, 28, 29,
+29, 30, 30, 30, 31, 32, 32, 33, 33, 33,
+34, 34, 35, 35, 35, 36, 36, 36, 37, 37,
+37, 38, 38, 63
+};
+
 #define BITS_MASK_8 0xULL
 #define PAIR_MASK_8 0xULL
 #define NYBL_MASK_8 0x0f0f0f0f0f0f0f0fULL
diff --git a/target/hexagon/arch.h b/target/hexagon/arch.h
index 544288e..f6fdd88 100644
--- a/target/hexagon/arch.h
+++ b/target/hexagon/arch.h
@@ -20,6 +20,10 @@
 
 #include "qemu/int128.h"
 
+extern const uint8_t rLPS_table_64x4[64][4];
+extern const uint8_t AC_next_state_MPS_64[64];
+extern const uint8_t AC_next_state_LPS_64[64];
+
 uint64_t interleave(uint32_t odd, uint32_t even);
 uint64_t deinterleave(uint64_t src);
 int32_t conv_round(int32_t a, int n);
diff --git a/target/hexagon/imported/encode_pp.def 
b/target/hexagon/imported/encode_pp.def
index dc4eba4..35ae3d2 100644
--- a/target/hexagon/imported/encode_pp.def
+++ b/target/hexagon/imported/encode_pp.def
@@ -1767,6 +1767,7 @@ SH_RRR_ENC(S4_vxsubaddh,
"0001","01-","-","110","d")
 SH_RRR_ENC(S4_vxaddsubhr,   "0001","11-","-","00-","d")
 SH_RRR_ENC(S4_vxsubaddhr,   "0001","11-","-","01-","d")
 SH_RRR_ENC(S4_extractp_rp,  "0001","11-","-","10-","d")
+SH_RRR_ENC(S2_cabacdecbin,  "0001","11-","-","11-","d") /* implicit P0 
write */
 
 
 DEF_FIELDROW_DESC32(ICLASS_S3op" 0010  PP-- ","[#2] 
Rdd=(Rss,Rtt,Pu)")
diff --git a/target/hexagon/imported/macros.def 
b/target/hexagon/imported/macros.def
index 56c99b1..32ed3bf 100755
--- a/target/hexagon/imported/macros.def
+++ b/target/hexagon/imported/macros.def
@@ -92,6 +92,21 @@ DEF_MACRO(
 /* attribs */
 )
 
+
+DEF_MACRO(
+fINSERT_RANGE,
+{
+int offset=LOWBIT;
+int width=HIBIT-LOWBIT+1;
+/* clear bits where new bits go */
+INREG &= ~(((fCONSTLL(1)<>29)&3];
+rLPS  = rLPS << 23;   /* left aligned */
+
+/* calculate rMPS */
+rMPS= (range&0xff80) - rLPS;
+
+/* most probable region 

[PATCH 04/15] Hexagon (target/hexagon) decide if pred has been written at TCG gen time

[Taylor Simpson]

Multiple writes to the same preg are and'ed together.  Rather than
generating a runtime check, we can determine at TCG generation time
if the predicate has previously been written in the packet.

Test added to tests/tcg/hexagon/misc.c

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/gen_tcg_funcs.py |  2 +-
 target/hexagon/genptr.c | 22 +++---
 target/hexagon/translate.c  |  9 +++--
 target/hexagon/translate.h  |  2 ++
 tests/tcg/hexagon/misc.c| 19 +++
 5 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/target/hexagon/gen_tcg_funcs.py b/target/hexagon/gen_tcg_funcs.py
index db9f663..7ceb25b 100755
--- a/target/hexagon/gen_tcg_funcs.py
+++ b/target/hexagon/gen_tcg_funcs.py
@@ -316,7 +316,7 @@ def genptr_dst_write(f, tag, regtype, regid):
 print("Bad register parse: ", regtype, regid)
 elif (regtype == "P"):
 if (regid in {"d", "e", "x"}):
-f.write("gen_log_pred_write(%s%sN, %s%sV);\n" % \
+f.write("gen_log_pred_write(ctx, %s%sN, %s%sV);\n" % \
 (regtype, regid, regtype, regid))
 f.write("ctx_log_pred_write(ctx, %s%sN);\n" % \
 (regtype, regid))
diff --git a/target/hexagon/genptr.c b/target/hexagon/genptr.c
index 349b949..9e9dfd5 100644
--- a/target/hexagon/genptr.c
+++ b/target/hexagon/genptr.c
@@ -122,20 +122,28 @@ static void gen_log_reg_write_pair(int rnum, TCGv_i64 val)
 #endif
 }
 
-static inline void gen_log_pred_write(int pnum, TCGv val)
+static inline void gen_log_pred_write(DisasContext *ctx, int pnum, TCGv val)
 {
 TCGv zero = tcg_const_tl(0);
 TCGv base_val = tcg_temp_new();
 TCGv and_val = tcg_temp_new();
 TCGv pred_written = tcg_temp_new();
 
-/* Multiple writes to the same preg are and'ed together */
 tcg_gen_andi_tl(base_val, val, 0xff);
-tcg_gen_and_tl(and_val, base_val, hex_new_pred_value[pnum]);
-tcg_gen_andi_tl(pred_written, hex_pred_written, 1 << pnum);
-tcg_gen_movcond_tl(TCG_COND_NE, hex_new_pred_value[pnum],
-   pred_written, zero,
-   and_val, base_val);
+
+/*
+ * Section 6.1.3 of the Hexagon V67 Programmer's Reference Manual
+ *
+ * Multiple writes to the same preg are and'ed together
+ * If this is the first predicate write in the packet, do a
+ * straight assignment.  Otherwise, do an and.
+ */
+if (!test_bit(pnum, ctx->pregs_written)) {
+tcg_gen_mov_tl(hex_new_pred_value[pnum], base_val);
+} else {
+tcg_gen_and_tl(hex_new_pred_value[pnum],
+   hex_new_pred_value[pnum], base_val);
+}
 tcg_gen_ori_tl(hex_pred_written, hex_pred_written, 1 << pnum);
 
 tcg_temp_free(zero);
diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c
index 19b9bc7..2d7ec12 100644
--- a/target/hexagon/translate.c
+++ b/target/hexagon/translate.c
@@ -173,6 +173,7 @@ static void gen_start_packet(DisasContext *ctx, Packet *pkt)
 ctx->reg_log_idx = 0;
 bitmap_zero(ctx->regs_written, TOTAL_PER_THREAD_REGS);
 ctx->preg_log_idx = 0;
+bitmap_zero(ctx->pregs_written, NUM_PREGS);
 for (i = 0; i < STORES_MAX; i++) {
 ctx->store_width[i] = 0;
 }
@@ -227,7 +228,7 @@ static void mark_implicit_pred_write(DisasContext *ctx, 
Insn *insn,
 }
 }
 
-static void mark_implicit_writes(DisasContext *ctx, Insn *insn)
+static void mark_implicit_reg_writes(DisasContext *ctx, Insn *insn)
 {
 mark_implicit_reg_write(ctx, insn, A_IMPLICIT_WRITES_FP,  HEX_REG_FP);
 mark_implicit_reg_write(ctx, insn, A_IMPLICIT_WRITES_SP,  HEX_REG_SP);
@@ -236,7 +237,10 @@ static void mark_implicit_writes(DisasContext *ctx, Insn 
*insn)
 mark_implicit_reg_write(ctx, insn, A_IMPLICIT_WRITES_SA0, HEX_REG_SA0);
 mark_implicit_reg_write(ctx, insn, A_IMPLICIT_WRITES_LC1, HEX_REG_LC1);
 mark_implicit_reg_write(ctx, insn, A_IMPLICIT_WRITES_SA1, HEX_REG_SA1);
+}
 
+static void mark_implicit_pred_writes(DisasContext *ctx, Insn *insn)
+{
 mark_implicit_pred_write(ctx, insn, A_IMPLICIT_WRITES_P0, 0);
 mark_implicit_pred_write(ctx, insn, A_IMPLICIT_WRITES_P1, 1);
 mark_implicit_pred_write(ctx, insn, A_IMPLICIT_WRITES_P2, 2);
@@ -247,8 +251,9 @@ static void gen_insn(CPUHexagonState *env, DisasContext 
*ctx,
  Insn *insn, Packet *pkt)
 {
 if (insn->generate) {
-mark_implicit_writes(ctx, insn);
+mark_implicit_reg_writes(ctx, insn);
 insn->generate(env, ctx, insn, pkt);
+mark_implicit_pred_writes(ctx, insn);
 } else {
 gen_exception_end_tb(ctx, HEX_EXCP_INVALID_OPCODE);
 }
diff --git a/target/hexagon/translate.h b/target/hexagon/translate.h
index 12506c8..0ecfbd7 100644
--- a/target/hexagon/translate.h
+++ b/target/hexagon/translate.h
@@ -34,6 +34,7 @@ typedef struct DisasContext {
 DECLARE_BITMAP(regs_written, TOTAL_PER_THREAD_REGS);
 i

[PATCH 12/15] Hexagon (target/hexagon) bit reverse (brev) addressing

[Taylor Simpson]

The following instructions are added
L2_loadrub_pbr  Rd32 = memub(Rx32++Mu2:brev)
L2_loadrb_pbr   Rd32 = memb(Rx32++Mu2:brev)
L2_loadruh_pbr  Rd32 = memuh(Rx32++Mu2:brev)
L2_loadrh_pbr   Rd32 = memh(Rx32++Mu2:brev)
L2_loadri_pbr   Rd32 = memw(Rx32++Mu2:brev)
L2_loadrd_pbr   Rdd32 = memd(Rx32++Mu2:brev)
S2_storerb_pbr  memb(Rx32++Mu2:brev).=.Rt32
S2_storerh_pbr  memh(Rx32++Mu2:brev).=.Rt32
S2_storerf_pbr  memh(Rx32++Mu2:brev).=.Rt.H32
S2_storeri_pbr  memw(Rx32++Mu2:brev).=.Rt32
S2_storerd_pbr  memd(Rx32++Mu2:brev).=.Rt32
S2_storerinew_pbr   memw(Rx32++Mu2:brev).=.Nt8.new
S2_storerbnew_pbr   memw(Rx32++Mu2:brev).=.Nt8.new
S2_storerhnew_pbr   memw(Rx32++Mu2:brev).=.Nt8.new

Test cases in tests/tcg/hexagon/brev.c

Signed-off-by: Taylor Simpson 
---
 target/hexagon/gen_tcg.h  |  28 +
 target/hexagon/imported/encode_pp.def |   5 +
 target/hexagon/imported/ldst.idef |   2 +
 target/hexagon/imported/macros.def|   6 ++
 target/hexagon/macros.h   |  15 +++
 tests/tcg/hexagon/Makefile.target |   1 +
 tests/tcg/hexagon/brev.c  | 185 ++
 7 files changed, 242 insertions(+)
 create mode 100644 tests/tcg/hexagon/brev.c

diff --git a/target/hexagon/gen_tcg.h b/target/hexagon/gen_tcg.h
index 2d64590..038e16e 100644
--- a/target/hexagon/gen_tcg.h
+++ b/target/hexagon/gen_tcg.h
@@ -37,6 +37,7 @@
  * _sp   stack pointer relativer0 = memw(r29+#12)
  * _ap   absolute set  r0 = memw(r1=##variable)
  * _pr   post increment register   r0 = memw(r1++m1)
+ * _pbr  post increment bit reverser0 = memw(r1++m1:brev)
  * _pi   post increment immediate  r0 = memb(r1++#1)
  * _pci  post increment circular immediate r0 = memw(r1++#4:circ(m0))
  * _pcr  post increment circular register  r0 = memw(r1++I:circ(m0))
@@ -53,6 +54,11 @@
 fEA_REG(RxV); \
 fPM_M(RxV, MuV); \
 } while (0)
+#define GET_EA_pbr \
+do { \
+fEA_BREVR(RxV); \
+fPM_M(RxV, MuV); \
+} while (0)
 #define GET_EA_pi \
 do { \
 fEA_REG(RxV); \
@@ -142,16 +148,22 @@
   fGEN_TCG_PCR(3, fLOAD(1, 8, u, EA, RddV))
 
 #define fGEN_TCG_L2_loadrub_pr(SHORTCODE)  SHORTCODE
+#define fGEN_TCG_L2_loadrub_pbr(SHORTCODE) SHORTCODE
 #define fGEN_TCG_L2_loadrub_pi(SHORTCODE)  SHORTCODE
 #define fGEN_TCG_L2_loadrb_pr(SHORTCODE)   SHORTCODE
+#define fGEN_TCG_L2_loadrb_pbr(SHORTCODE)  SHORTCODE
 #define fGEN_TCG_L2_loadrb_pi(SHORTCODE)   SHORTCODE
 #define fGEN_TCG_L2_loadruh_pr(SHORTCODE)  SHORTCODE
+#define fGEN_TCG_L2_loadruh_pbr(SHORTCODE) SHORTCODE
 #define fGEN_TCG_L2_loadruh_pi(SHORTCODE)  SHORTCODE
 #define fGEN_TCG_L2_loadrh_pr(SHORTCODE)   SHORTCODE
+#define fGEN_TCG_L2_loadrh_pbr(SHORTCODE)  SHORTCODE
 #define fGEN_TCG_L2_loadrh_pi(SHORTCODE)   SHORTCODE
 #define fGEN_TCG_L2_loadri_pr(SHORTCODE)   SHORTCODE
+#define fGEN_TCG_L2_loadri_pbr(SHORTCODE)  SHORTCODE
 #define fGEN_TCG_L2_loadri_pi(SHORTCODE)   SHORTCODE
 #define fGEN_TCG_L2_loadrd_pr(SHORTCODE)   SHORTCODE
+#define fGEN_TCG_L2_loadrd_pbr(SHORTCODE)  SHORTCODE
 #define fGEN_TCG_L2_loadrd_pi(SHORTCODE)   SHORTCODE
 
 /*
@@ -282,41 +294,57 @@
 tcg_temp_free(tmp); \
 } while (0)
 
+#define fGEN_TCG_S2_storerb_pbr(SHORTCODE) \
+fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerb_pci(SHORTCODE) \
 fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerb_pcr(SHORTCODE) \
 fGEN_TCG_STORE_pcr(0, fSTORE(1, 1, EA, fGETBYTE(0, RtV)))
 
+#define fGEN_TCG_S2_storerh_pbr(SHORTCODE) \
+fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerh_pci(SHORTCODE) \
 fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerh_pcr(SHORTCODE) \
 fGEN_TCG_STORE_pcr(1, fSTORE(1, 2, EA, fGETHALF(0, RtV)))
 
+#define fGEN_TCG_S2_storerf_pbr(SHORTCODE) \
+fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerf_pci(SHORTCODE) \
 fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerf_pcr(SHORTCODE) \
 fGEN_TCG_STORE_pcr(1, fSTORE(1, 2, EA, fGETHALF(1, RtV)))
 
+#define fGEN_TCG_S2_storeri_pbr(SHORTCODE) \
+fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storeri_pci(SHORTCODE) \
 fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storeri_pcr(SHORTCODE) \
 fGEN_TCG_STORE_pcr(2, fSTORE(1, 4, EA, RtV))
 
+#define fGEN_TCG_S2_storerd_pbr(SHORTCODE) \
+fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerd_pci(SHORTCODE) \
 fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerd_pcr(SHORTCODE) \
 fGEN_TCG_STORE_pcr(3, fSTORE(1, 8, EA, RttV))
 
+#define fGEN_TCG_S2_storerbnew_pbr(SHORTCODE) \
+fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerbnew_pci(SHORTCODE) \
 fGEN_TCG_STORE(SHORTCODE)
 #define fGEN_TCG_S2_storerbnew_pc

[PATCH 00/15] Hexagon (target/hexagon) update

[Taylor Simpson]

This patch series is a significant update for the Hexagon target
The first 9 patches address feedback from Richard Henderson 

The next 6 patches add the remaining instructions for the Hexagon scalar 
core

The patches are logically independent but are organized as a series to avoid 
potential
confilcts if they are merged out of order.

Taylor Simpson (15):
  Hexagon (target/hexagon) TCG generation cleanup
  Hexagon (target/hexagon) remove unnecessary inline directives
  Hexagon (target/hexagon) properly generate TB end for DISAS_NORETURN
  Hexagon (target/hexagon) decide if pred has been written at TCG gen time
  Hexagon (target/hexagon) change variables from int to bool when appropriate
  Hexagon (target/hexagon) utility function changes
  Hexagon (target/hexagon) use softfloat default NaN and tininess
  Hexagon (target/hexagon) replace float32_mul_pow2 with float32_scalbn
  Hexagon (target/hexagon) use softfloat for float-to-int conversions
  Hexagon (target/hexagon) instructions with multiple definitions
  Hexagon (target/hexagon) circular addressing
  Hexagon (target/hexagon) bit reverse (brev) addressing
  Hexagon (target/hexagon) load and unpack bytes instructions
  Hexagon (target/hexagon) load into shifted register instructions
  Hexagon (target/hexagon) CABAC decode bin

 fpu/softfloat-specialize.c.inc|   5 +
 linux-user/hexagon/cpu_loop.c |   2 +-
 target/hexagon/arch.c | 191 +++---
 target/hexagon/arch.h |   7 +-
 target/hexagon/conv_emu.c | 177 -
 target/hexagon/conv_emu.h |  31 ---
 target/hexagon/cpu.c  |  14 +-
 target/hexagon/cpu.h  |   6 +-
 target/hexagon/cpu_bits.h |   2 +-
 target/hexagon/decode.c   |  80 +++---
 target/hexagon/fma_emu.c  |  40 +--
 target/hexagon/gen_tcg.h  | 458 
 target/hexagon/gen_tcg_funcs.py   |   2 +-
 target/hexagon/genptr.c   | 243 -
 target/hexagon/helper.h   |  22 +-
 target/hexagon/imported/alu.idef  |  44 
 target/hexagon/imported/encode_pp.def |  30 +++
 target/hexagon/imported/float.idef|  32 +++
 target/hexagon/imported/ldst.idef |  68 +
 target/hexagon/imported/macros.def|  47 
 target/hexagon/imported/shift.idef|  47 
 target/hexagon/insn.h |  21 +-
 target/hexagon/macros.h   | 174 -
 target/hexagon/meson.build|   1 -
 target/hexagon/op_helper.c| 373 ++
 target/hexagon/translate.c|  79 +++---
 target/hexagon/translate.h|   7 +-
 tests/tcg/hexagon/Makefile.target |   5 +
 tests/tcg/hexagon/brev.c  | 185 +
 tests/tcg/hexagon/circ.c  | 393 
 tests/tcg/hexagon/fpstuff.c   | 242 +
 tests/tcg/hexagon/load_align.c| 415 +
 tests/tcg/hexagon/load_unpack.c   | 474 ++
 tests/tcg/hexagon/misc.c  |  47 
 tests/tcg/hexagon/multi_result.c  | 249 ++
 35 files changed, 3726 insertions(+), 487 deletions(-)
 delete mode 100644 target/hexagon/conv_emu.c
 delete mode 100644 target/hexagon/conv_emu.h
 create mode 100644 tests/tcg/hexagon/brev.c
 create mode 100644 tests/tcg/hexagon/circ.c
 create mode 100644 tests/tcg/hexagon/load_align.c
 create mode 100644 tests/tcg/hexagon/load_unpack.c
 create mode 100644 tests/tcg/hexagon/multi_result.c

-- 
2.7.4

[PATCH 11/15] Hexagon (target/hexagon) circular addressing

[Taylor Simpson]

The following instructions are added
L2_loadrub_pci  Rd32 = memub(Rx32++#s4:0:circ(Mu2))
L2_loadrb_pci   Rd32 = memb(Rx32++#s4:0:circ(Mu2))
L2_loadruh_pci  Rd32 = memuh(Rx32++#s4:1:circ(Mu2))
L2_loadrh_pci   Rd32 = memh(Rx32++#s4:1:circ(Mu2))
L2_loadri_pci   Rd32 = memw(Rx32++#s4:2:circ(Mu2))
L2_loadrd_pci   Rdd32 = memd(Rx32++#s4:3:circ(Mu2))
S2_storerb_pci  memb(Rx32++#s4:0:circ(Mu2)) = Rt32
S2_storerh_pci  memh(Rx32++#s4:1:circ(Mu2)) = Rt32
S2_storerf_pci  memh(Rx32++#s4:1:circ(Mu2)) = Rt.H32
S2_storeri_pci  memw(Rx32++#s4:2:circ(Mu2)) = Rt32
S2_storerd_pci  memd(Rx32++#s4:3:circ(Mu2)) = Rtt32
S2_storerbnew_pci   memb(Rx32++#s4:0:circ(Mu2)) = Nt8.new
S2_storerhnew_pci   memw(Rx32++#s4:1:circ(Mu2)) = Nt8.new
S2_storerinew_pci   memw(Rx32++#s4:2:circ(Mu2)) = Nt8.new
L2_loadrub_pcr  Rd32 = memub(Rx32++I:circ(Mu2))
L2_loadrb_pcr   Rd32 = memb(Rx32++I:circ(Mu2))
L2_loadruh_pcr  Rd32 = memuh(Rx32++I:circ(Mu2))
L2_loadrh_pcr   Rd32 = memh(Rx32++I:circ(Mu2))
L2_loadri_pcr   Rd32 = memw(Rx32++I:circ(Mu2))
L2_loadrd_pcr   Rdd32 = memd(Rx32++I:circ(Mu2))
S2_storerb_pcr  memb(Rx32++I:circ(Mu2)) = Rt32
S2_storerh_pcr  memh(Rx32++I:circ(Mu2)) = Rt32
S2_storerf_pcr  memh(Rx32++I:circ(Mu2)) = Rt32.H32
S2_storeri_pcr  memw(Rx32++I:circ(Mu2)) = Rt32
S2_storerd_pcr  memd(Rx32++I:circ(Mu2)) = Rtt32
S2_storerbnew_pcr   memb(Rx32++I:circ(Mu2)) = Nt8.new
S2_storerhnew_pcr   memh(Rx32++I:circ(Mu2)) = Nt8.new
S2_storerinew_pcr   memw(Rx32++I:circ(Mu2)) = Nt8.new

Test cases in tests/tcg/hexagon/circ.c

Signed-off-by: Taylor Simpson 
---
 target/hexagon/gen_tcg.h  | 127 +++
 target/hexagon/genptr.c   |  85 
 target/hexagon/imported/encode_pp.def |   9 +
 target/hexagon/imported/ldst.idef |   4 +
 target/hexagon/imported/macros.def|  26 +++
 target/hexagon/macros.h   |  87 
 tests/tcg/hexagon/Makefile.target |   1 +
 tests/tcg/hexagon/circ.c  | 393 ++
 8 files changed, 732 insertions(+)
 create mode 100644 tests/tcg/hexagon/circ.c

diff --git a/target/hexagon/gen_tcg.h b/target/hexagon/gen_tcg.h
index 0b2c2ca..2d64590 100644
--- a/target/hexagon/gen_tcg.h
+++ b/target/hexagon/gen_tcg.h
@@ -38,6 +38,8 @@
  * _ap   absolute set  r0 = memw(r1=##variable)
  * _pr   post increment register   r0 = memw(r1++m1)
  * _pi   post increment immediate  r0 = memb(r1++#1)
+ * _pci  post increment circular immediate r0 = memw(r1++#4:circ(m0))
+ * _pcr  post increment circular register  r0 = memw(r1++I:circ(m0))
  */
 
 /* Macros for complex addressing modes */
@@ -56,7 +58,28 @@
 fEA_REG(RxV); \
 fPM_I(RxV, siV); \
 } while (0)
+#define GET_EA_pci \
+do { \
+fEA_REG(RxV); \
+fPM_CIRI(RxV, siV, MuV); \
+} while (0)
+#define GET_EA_pcr(SHIFT) \
+do { \
+fEA_REG(RxV); \
+fPM_CIRR(RxV, fREAD_IREG(MuV, (SHIFT)), MuV); \
+} while (0)
 
+/*
+ * Many instructions will work with just macro redefinitions
+ * with the caveat that they need a tmp variable to carry a
+ * value between them.
+ */
+#define fGEN_TCG_tmp(SHORTCODE) \
+do { \
+TCGv tmp = tcg_temp_new(); \
+SHORTCODE; \
+tcg_temp_free(tmp); \
+} while (0)
 
 /* Instructions with multiple definitions */
 #define fGEN_TCG_LOAD_AP(RES, SIZE, SIGN) \
@@ -80,6 +103,44 @@
 #define fGEN_TCG_L4_loadrd_ap(SHORTCODE) \
 fGEN_TCG_LOAD_AP(RddV, 8, u)
 
+#define fGEN_TCG_L2_loadrub_pci(SHORTCODE) \
+  fGEN_TCG_tmp(SHORTCODE)
+#define fGEN_TCG_L2_loadrb_pci(SHORTCODE) \
+  fGEN_TCG_tmp(SHORTCODE)
+#define fGEN_TCG_L2_loadruh_pci(SHORTCODE) \
+  fGEN_TCG_tmp(SHORTCODE)
+#define fGEN_TCG_L2_loadrh_pci(SHORTCODE) \
+  fGEN_TCG_tmp(SHORTCODE)
+#define fGEN_TCG_L2_loadri_pci(SHORTCODE) \
+  fGEN_TCG_tmp(SHORTCODE)
+#define fGEN_TCG_L2_loadrd_pci(SHORTCODE) \
+  fGEN_TCG_tmp(SHORTCODE)
+
+#define fGEN_TCG_PCR(SHIFT, LOAD) \
+do { \
+TCGv ireg = tcg_temp_new(); \
+TCGv tmp = tcg_temp_new(); \
+fEA_REG(RxV); \
+fREAD_IREG(MuV, SHIFT); \
+gen_helper_fcircadd(RxV, RxV, ireg, MuV, fREAD_CSREG(MuN)); \
+LOAD; \
+tcg_temp_free(tmp); \
+tcg_temp_free(ireg); \
+} while (0)
+
+#define fGEN_TCG_L2_loadrub_pcr(SHORTCODE) \
+  fGEN_TCG_PCR(0, fLOAD(1, 1, u, EA, RdV))
+#define fGEN_TCG_L2_loadrb_pcr(SHORTCODE) \
+  fGEN_TCG_PCR(0, fLOAD(1, 1, s, EA, RdV))
+#define fGEN_TCG_L2_loadruh_pcr(SHORTCODE) \
+  fGEN_TCG_PCR(1, fLOAD(1, 2, u, EA, RdV))
+#define fGEN_TCG_L2_loadrh_pcr(SHORTCODE) \
+  fGEN_TCG_PCR(1, fLOAD(1, 2, s, EA,

[PATCH 03/15] Hexagon (target/hexagon) properly generate TB end for DISAS_NORETURN

[Taylor Simpson]

When exiting a TB, generate all the code before returning from
hexagon_tr_translate_packet so that nothing needs to be done in
hexagon_tr_tb_stop.

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/translate.c | 62 +-
 target/hexagon/translate.h |  3 ---
 2 files changed, 34 insertions(+), 31 deletions(-)

diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c
index 5d92ab0..19b9bc7 100644
--- a/target/hexagon/translate.c
+++ b/target/hexagon/translate.c
@@ -54,16 +54,41 @@ static const char * const hexagon_prednames[] = {
   "p0", "p1", "p2", "p3"
 };
 
-void gen_exception(int excp)
+static void gen_exception(int excp)
 {
 TCGv_i32 helper_tmp = tcg_const_i32(excp);
 gen_helper_raise_exception(cpu_env, helper_tmp);
 tcg_temp_free_i32(helper_tmp);
 }
 
-void gen_exception_debug(void)
+static void gen_exec_counters(DisasContext *ctx)
+{
+tcg_gen_addi_tl(hex_gpr[HEX_REG_QEMU_PKT_CNT],
+hex_gpr[HEX_REG_QEMU_PKT_CNT], ctx->num_packets);
+tcg_gen_addi_tl(hex_gpr[HEX_REG_QEMU_INSN_CNT],
+hex_gpr[HEX_REG_QEMU_INSN_CNT], ctx->num_insns);
+}
+
+static void gen_end_tb(DisasContext *ctx)
 {
-gen_exception(EXCP_DEBUG);
+gen_exec_counters(ctx);
+tcg_gen_mov_tl(hex_gpr[HEX_REG_PC], hex_next_PC);
+if (ctx->base.singlestep_enabled) {
+gen_exception(EXCP_DEBUG);
+} else {
+tcg_gen_exit_tb(NULL, 0);
+}
+ctx->base.is_jmp = DISAS_NORETURN;
+}
+
+static void gen_exception_end_tb(DisasContext *ctx, int excp)
+{
+gen_exec_counters(ctx);
+tcg_gen_mov_tl(hex_gpr[HEX_REG_PC], hex_next_PC);
+gen_exception(excp);
+tcg_gen_exit_tb(NULL, 0);
+ctx->base.is_jmp = DISAS_NORETURN;
+
 }
 
 #if HEX_DEBUG
@@ -225,8 +250,7 @@ static void gen_insn(CPUHexagonState *env, DisasContext 
*ctx,
 mark_implicit_writes(ctx, insn);
 insn->generate(env, ctx, insn, pkt);
 } else {
-gen_exception(HEX_EXCP_INVALID_OPCODE);
-ctx->base.is_jmp = DISAS_NORETURN;
+gen_exception_end_tb(ctx, HEX_EXCP_INVALID_OPCODE);
 }
 }
 
@@ -447,14 +471,6 @@ static void update_exec_counters(DisasContext *ctx, Packet 
*pkt)
 ctx->num_insns += num_real_insns;
 }
 
-static void gen_exec_counters(DisasContext *ctx)
-{
-tcg_gen_addi_tl(hex_gpr[HEX_REG_QEMU_PKT_CNT],
-hex_gpr[HEX_REG_QEMU_PKT_CNT], ctx->num_packets);
-tcg_gen_addi_tl(hex_gpr[HEX_REG_QEMU_INSN_CNT],
-hex_gpr[HEX_REG_QEMU_INSN_CNT], ctx->num_insns);
-}
-
 static void gen_commit_packet(DisasContext *ctx, Packet *pkt)
 {
 gen_reg_writes(ctx);
@@ -478,7 +494,7 @@ static void gen_commit_packet(DisasContext *ctx, Packet 
*pkt)
 #endif
 
 if (pkt->pkt_has_cof) {
-ctx->base.is_jmp = DISAS_NORETURN;
+gen_end_tb(ctx);
 }
 }
 
@@ -491,8 +507,7 @@ static void decode_and_translate_packet(CPUHexagonState 
*env, DisasContext *ctx)
 
 nwords = read_packet_words(env, ctx, words);
 if (!nwords) {
-gen_exception(HEX_EXCP_INVALID_PACKET);
-ctx->base.is_jmp = DISAS_NORETURN;
+gen_exception_end_tb(ctx, HEX_EXCP_INVALID_PACKET);
 return;
 }
 
@@ -505,8 +520,7 @@ static void decode_and_translate_packet(CPUHexagonState 
*env, DisasContext *ctx)
 gen_commit_packet(ctx, &pkt);
 ctx->base.pc_next += pkt.encod_pkt_size_in_bytes;
 } else {
-gen_exception(HEX_EXCP_INVALID_PACKET);
-ctx->base.is_jmp = DISAS_NORETURN;
+gen_exception_end_tb(ctx, HEX_EXCP_INVALID_PACKET);
 }
 }
 
@@ -537,8 +551,7 @@ static bool hexagon_tr_breakpoint_check(DisasContextBase 
*dcbase, CPUState *cpu,
 DisasContext *ctx = container_of(dcbase, DisasContext, base);
 
 tcg_gen_movi_tl(hex_gpr[HEX_REG_PC], ctx->base.pc_next);
-ctx->base.is_jmp = DISAS_NORETURN;
-gen_exception_debug();
+gen_exception_end_tb(ctx, EXCP_DEBUG);
 /*
  * The address covered by the breakpoint must be included in
  * [tb->pc, tb->pc + tb->size) in order to for it to be
@@ -601,19 +614,12 @@ static void hexagon_tr_tb_stop(DisasContextBase *dcbase, 
CPUState *cpu)
 gen_exec_counters(ctx);
 tcg_gen_movi_tl(hex_gpr[HEX_REG_PC], ctx->base.pc_next);
 if (ctx->base.singlestep_enabled) {
-gen_exception_debug();
+gen_exception(EXCP_DEBUG);
 } else {
 tcg_gen_exit_tb(NULL, 0);
 }
 break;
 case DISAS_NORETURN:
-gen_exec_counters(ctx);
-tcg_gen_mov_tl(hex_gpr[HEX_REG_PC], hex_next_PC);
-if (ctx->base.singlestep_enabled) {
-gen_exception_debug();
-} else {
-tcg_gen_exit_tb(NULL, 0);
-}
 break;
 default:
 g_assert_not_reached();
diff --git a/target/hexagon/translate.h b/target/hexagon/translate.h
index 938f7fb..12506c8 100644
--- a/target/hexagon/translate.h
+++ b/target/hexagon/t

[PATCH 08/15] Hexagon (target/hexagon) replace float32_mul_pow2 with float32_scalbn

[Taylor Simpson]

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/arch.c | 28 +++-
 1 file changed, 11 insertions(+), 17 deletions(-)

diff --git a/target/hexagon/arch.c b/target/hexagon/arch.c
index bb51f19..40b6e3d 100644
--- a/target/hexagon/arch.c
+++ b/target/hexagon/arch.c
@@ -143,12 +143,6 @@ void arch_fpop_end(CPUHexagonState *env)
 }
 }
 
-static float32 float32_mul_pow2(float32 a, uint32_t p, float_status *fp_status)
-{
-float32 b = make_float32((SF_BIAS + p) << SF_MANTBITS);
-return float32_mul(a, b, fp_status);
-}
-
 int arch_sf_recip_common(float32 *Rs, float32 *Rt, float32 *Rd, int *adjust,
  float_status *fp_status)
 {
@@ -217,22 +211,22 @@ int arch_sf_recip_common(float32 *Rs, float32 *Rt, 
float32 *Rd, int *adjust,
 if ((n_exp - d_exp + SF_BIAS) <= SF_MANTBITS) {
 /* Near quotient underflow / inexact Q */
 PeV = 0x80;
-RtV = float32_mul_pow2(RtV, -64, fp_status);
-RsV = float32_mul_pow2(RsV, 64, fp_status);
+RtV = float32_scalbn(RtV, -64, fp_status);
+RsV = float32_scalbn(RsV, 64, fp_status);
 } else if ((n_exp - d_exp + SF_BIAS) > (SF_MAXEXP - 24)) {
 /* Near quotient overflow */
 PeV = 0x40;
-RtV = float32_mul_pow2(RtV, 32, fp_status);
-RsV = float32_mul_pow2(RsV, -32, fp_status);
+RtV = float32_scalbn(RtV, 32, fp_status);
+RsV = float32_scalbn(RsV, -32, fp_status);
 } else if (n_exp <= SF_MANTBITS + 2) {
-RtV = float32_mul_pow2(RtV, 64, fp_status);
-RsV = float32_mul_pow2(RsV, 64, fp_status);
+RtV = float32_scalbn(RtV, 64, fp_status);
+RsV = float32_scalbn(RsV, 64, fp_status);
 } else if (d_exp <= 1) {
-RtV = float32_mul_pow2(RtV, 32, fp_status);
-RsV = float32_mul_pow2(RsV, 32, fp_status);
+RtV = float32_scalbn(RtV, 32, fp_status);
+RsV = float32_scalbn(RsV, 32, fp_status);
 } else if (d_exp > 252) {
-RtV = float32_mul_pow2(RtV, -32, fp_status);
-RsV = float32_mul_pow2(RsV, -32, fp_status);
+RtV = float32_scalbn(RtV, -32, fp_status);
+RsV = float32_scalbn(RsV, -32, fp_status);
 }
 RdV = 0;
 ret = 1;
@@ -274,7 +268,7 @@ int arch_sf_invsqrt_common(float32 *Rs, float32 *Rd, int 
*adjust,
 /* Basic checks passed */
 r_exp = float32_getexp(RsV);
 if (r_exp <= 24) {
-RsV = float32_mul_pow2(RsV, 64, fp_status);
+RsV = float32_scalbn(RsV, 64, fp_status);
 PeV = 0xe0;
 }
 RdV = 0;
-- 
2.7.4

[PATCH 05/15] Hexagon (target/hexagon) change variables from int to bool when appropriate

[Taylor Simpson]

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/cpu_bits.h  |  2 +-
 target/hexagon/decode.c| 80 +++---
 target/hexagon/insn.h  | 21 ++--
 target/hexagon/op_helper.c |  8 ++---
 target/hexagon/translate.c |  6 ++--
 target/hexagon/translate.h |  2 +-
 6 files changed, 60 insertions(+), 59 deletions(-)

diff --git a/target/hexagon/cpu_bits.h b/target/hexagon/cpu_bits.h
index 96af834..96fef71 100644
--- a/target/hexagon/cpu_bits.h
+++ b/target/hexagon/cpu_bits.h
@@ -47,7 +47,7 @@ static inline uint32_t iclass_bits(uint32_t encoding)
 return iclass;
 }
 
-static inline int is_packet_end(uint32_t endocing)
+static inline bool is_packet_end(uint32_t endocing)
 {
 uint32_t bits = parse_bits(endocing);
 return ((bits == 0x3) || (bits == 0x0));
diff --git a/target/hexagon/decode.c b/target/hexagon/decode.c
index 65d97ce..dffe1d1 100644
--- a/target/hexagon/decode.c
+++ b/target/hexagon/decode.c
@@ -340,8 +340,8 @@ static void decode_split_cmpjump(Packet *pkt)
 if (GET_ATTRIB(pkt->insn[i].opcode, A_NEWCMPJUMP)) {
 last = pkt->num_insns;
 pkt->insn[last] = pkt->insn[i];/* copy the instruction */
-pkt->insn[last].part1 = 1;/* last instruction does the CMP */
-pkt->insn[i].part1 = 0;/* existing instruction does the JUMP */
+pkt->insn[last].part1 = true;  /* last insn does the CMP */
+pkt->insn[i].part1 = false;/* existing insn does the JUMP 
*/
 pkt->num_insns++;
 }
 }
@@ -354,7 +354,7 @@ static void decode_split_cmpjump(Packet *pkt)
 }
 }
 
-static int decode_opcode_can_jump(int opcode)
+static bool decode_opcode_can_jump(int opcode)
 {
 if ((GET_ATTRIB(opcode, A_JUMP)) ||
 (GET_ATTRIB(opcode, A_CALL)) ||
@@ -362,15 +362,15 @@ static int decode_opcode_can_jump(int opcode)
 (opcode == J2_pause)) {
 /* Exception to A_JUMP attribute */
 if (opcode == J4_hintjumpr) {
-return 0;
+return false;
 }
-return 1;
+return true;
 }
 
-return 0;
+return false;
 }
 
-static int decode_opcode_ends_loop(int opcode)
+static bool decode_opcode_ends_loop(int opcode)
 {
 return GET_ATTRIB(opcode, A_HWLOOP0_END) ||
GET_ATTRIB(opcode, A_HWLOOP1_END);
@@ -383,9 +383,9 @@ static void decode_set_insn_attr_fields(Packet *pkt)
 int numinsns = pkt->num_insns;
 uint16_t opcode;
 
-pkt->pkt_has_cof = 0;
-pkt->pkt_has_endloop = 0;
-pkt->pkt_has_dczeroa = 0;
+pkt->pkt_has_cof = false;
+pkt->pkt_has_endloop = false;
+pkt->pkt_has_dczeroa = false;
 
 for (i = 0; i < numinsns; i++) {
 opcode = pkt->insn[i].opcode;
@@ -394,14 +394,14 @@ static void decode_set_insn_attr_fields(Packet *pkt)
 }
 
 if (GET_ATTRIB(opcode, A_DCZEROA)) {
-pkt->pkt_has_dczeroa = 1;
+pkt->pkt_has_dczeroa = true;
 }
 
 if (GET_ATTRIB(opcode, A_STORE)) {
 if (pkt->insn[i].slot == 0) {
-pkt->pkt_has_store_s0 = 1;
+pkt->pkt_has_store_s0 = true;
 } else {
-pkt->pkt_has_store_s1 = 1;
+pkt->pkt_has_store_s1 = true;
 }
 }
 
@@ -422,9 +422,9 @@ static void decode_set_insn_attr_fields(Packet *pkt)
  */
 static void decode_shuffle_for_execution(Packet *packet)
 {
-int changed = 0;
+bool changed = false;
 int i;
-int flag;/* flag means we've seen a non-memory instruction */
+bool flag;/* flag means we've seen a non-memory instruction */
 int n_mems;
 int last_insn = packet->num_insns - 1;
 
@@ -437,7 +437,7 @@ static void decode_shuffle_for_execution(Packet *packet)
 }
 
 do {
-changed = 0;
+changed = false;
 /*
  * Stores go last, must not reorder.
  * Cannot shuffle stores past loads, either.
@@ -445,13 +445,13 @@ static void decode_shuffle_for_execution(Packet *packet)
  * then a store, shuffle the store to the front.  Don't shuffle
  * stores wrt each other or a load.
  */
-for (flag = n_mems = 0, i = last_insn; i >= 0; i--) {
+for (flag = false, n_mems = 0, i = last_insn; i >= 0; i--) {
 int opcode = packet->insn[i].opcode;
 
 if (flag && GET_ATTRIB(opcode, A_STORE)) {
 decode_send_insn_to(packet, i, last_insn - n_mems);
 n_mems++;
-changed = 1;
+changed = true;
 } else if (GET_ATTRIB(opcode, A_STORE)) {
 n_mems++;
 } else if (GET_ATTRIB(opcode, A_LOAD)) {
@@ -466,7 +466,7 @@ static void decode_shuffle_for_execution(Packet *packet)
  * a .new value
  */
 } else {
-flag = 1;
+flag = true;
 }
   

[PATCH v4 09/10] Add the function of colo_bitmap_clear_diry.

[leirao]

From: "Rao, Lei" 

When we use continuous dirty memory copy for flushing ram cache on
secondary VM, we can also clean up the bitmap of contiguous dirty
page memory. This also can reduce the VM stop time during checkpoint.

Signed-off-by: Lei Rao 
---
 migration/ram.c | 29 +
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index a258466..ae1e659 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -855,6 +855,30 @@ unsigned long colo_bitmap_find_dirty(RAMState *rs, 
RAMBlock *rb,
 return first;
 }
 
+/**
+ * colo_bitmap_clear_dirty:when we flush ram cache to ram, we will use
+ * continuous memory copy, so we can also clean up the bitmap of contiguous
+ * dirty memory.
+ */
+static inline bool colo_bitmap_clear_dirty(RAMState *rs,
+   RAMBlock *rb,
+   unsigned long start,
+   unsigned long num)
+{
+bool ret;
+unsigned long i = 0;
+
+qemu_mutex_lock(&rs->bitmap_mutex);
+for (i = 0; i < num; i++) {
+ret = test_and_clear_bit(start + i, rb->bmap);
+if (ret) {
+rs->migration_dirty_pages--;
+}
+}
+qemu_mutex_unlock(&rs->bitmap_mutex);
+return ret;
+}
+
 static inline bool migration_bitmap_clear_dirty(RAMState *rs,
 RAMBlock *rb,
 unsigned long page)
@@ -3700,7 +3724,6 @@ void colo_flush_ram_cache(void)
 void *src_host;
 unsigned long offset = 0;
 unsigned long num = 0;
-unsigned long i = 0;
 
 memory_global_dirty_log_sync();
 WITH_RCU_READ_LOCK_GUARD() {
@@ -3722,9 +3745,7 @@ void colo_flush_ram_cache(void)
 num = 0;
 block = QLIST_NEXT_RCU(block, next);
 } else {
-for (i = 0; i < num; i++) {
-migration_bitmap_clear_dirty(ram_state, block, offset + i);
-}
+colo_bitmap_clear_dirty(ram_state, block, offset, num);
 dst_host = block->host
  + (((ram_addr_t)offset) << TARGET_PAGE_BITS);
 src_host = block->colo_cache
-- 
1.8.3.1

[PATCH v4 05/10] Add a function named packet_new_nocopy for COLO.

[leirao]

From: "Rao, Lei" 

Use the packet_new_nocopy instead of packet_new in the
filter-rewriter module. There will be one less memory
copy in the processing of each network packet.

Signed-off-by: Lei Rao 
---
 net/colo.c| 23 +++
 net/colo.h|  1 +
 net/filter-rewriter.c |  3 +--
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/net/colo.c b/net/colo.c
index ef00609..58106a8 100644
--- a/net/colo.c
+++ b/net/colo.c
@@ -174,6 +174,29 @@ Packet *packet_new(const void *data, int size, int 
vnet_hdr_len)
 return pkt;
 }
 
+/*
+ * packet_new_nocopy will not copy data, so the caller can't release
+ * the data. And it will be released in packet_destroy.
+ */
+Packet *packet_new_nocopy(void *data, int size, int vnet_hdr_len)
+{
+Packet *pkt = g_slice_new(Packet);
+
+pkt->data = data;
+pkt->size = size;
+pkt->creation_ms = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+pkt->vnet_hdr_len = vnet_hdr_len;
+pkt->tcp_seq = 0;
+pkt->tcp_ack = 0;
+pkt->seq_end = 0;
+pkt->header_size = 0;
+pkt->payload_size = 0;
+pkt->offset = 0;
+pkt->flags = 0;
+
+return pkt;
+}
+
 void packet_destroy(void *opaque, void *user_data)
 {
 Packet *pkt = opaque;
diff --git a/net/colo.h b/net/colo.h
index 573ab91..d91cd24 100644
--- a/net/colo.h
+++ b/net/colo.h
@@ -101,6 +101,7 @@ bool connection_has_tracked(GHashTable 
*connection_track_table,
 ConnectionKey *key);
 void connection_hashtable_reset(GHashTable *connection_track_table);
 Packet *packet_new(const void *data, int size, int vnet_hdr_len);
+Packet *packet_new_nocopy(void *data, int size, int vnet_hdr_len);
 void packet_destroy(void *opaque, void *user_data);
 void packet_destroy_partial(void *opaque, void *user_data);
 
diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c
index 10fe393..cb3a96c 100644
--- a/net/filter-rewriter.c
+++ b/net/filter-rewriter.c
@@ -270,8 +270,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf,
 vnet_hdr_len = nf->netdev->vnet_hdr_len;
 }
 
-pkt = packet_new(buf, size, vnet_hdr_len);
-g_free(buf);
+pkt = packet_new_nocopy(buf, size, vnet_hdr_len);
 
 /*
  * if we get tcp packet
-- 
1.8.3.1

[PATCH 09/15] Hexagon (target/hexagon) use softfloat for float-to-int conversions

[Taylor Simpson]

Use the proper return for helpers that convert to unsigned
Remove target/hexagon/conv_emu.[ch]

Address feedback from Richard Henderson 

Signed-off-by: Taylor Simpson 
---
 target/hexagon/conv_emu.c   | 177 
 target/hexagon/conv_emu.h   |  31 
 target/hexagon/fma_emu.c|   1 -
 target/hexagon/helper.h |  16 ++--
 target/hexagon/meson.build  |   1 -
 target/hexagon/op_helper.c  | 169 --
 tests/tcg/hexagon/fpstuff.c | 145 
 7 files changed, 281 insertions(+), 259 deletions(-)
 delete mode 100644 target/hexagon/conv_emu.c
 delete mode 100644 target/hexagon/conv_emu.h

diff --git a/target/hexagon/conv_emu.c b/target/hexagon/conv_emu.c
deleted file mode 100644
index 3985b10..000
--- a/target/hexagon/conv_emu.c
+++ /dev/null
@@ -1,177 +0,0 @@
-/*
- *  Copyright(c) 2019-2021 Qualcomm Innovation Center, Inc. All Rights 
Reserved.
- *
- *  This program is free software; you can redistribute it and/or modify
- *  it under the terms of the GNU General Public License as published by
- *  the Free Software Foundation; either version 2 of the License, or
- *  (at your option) any later version.
- *
- *  This program is distributed in the hope that it will be useful,
- *  but WITHOUT ANY WARRANTY; without even the implied warranty of
- *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- *  GNU General Public License for more details.
- *
- *  You should have received a copy of the GNU General Public License
- *  along with this program; if not, see .
- */
-
-#include "qemu/osdep.h"
-#include "qemu/host-utils.h"
-#include "fpu/softfloat.h"
-#include "macros.h"
-#include "conv_emu.h"
-
-#define LL_MAX_POS 0x7fffULL
-#define MAX_POS 0x7fffU
-
-static uint64_t conv_f64_to_8u_n(float64 in, int will_negate,
- float_status *fp_status)
-{
-uint8_t sign = float64_is_neg(in);
-if (float64_is_infinity(in)) {
-float_raise(float_flag_invalid, fp_status);
-if (float64_is_neg(in)) {
-return 0ULL;
-} else {
-return ~0ULL;
-}
-}
-if (float64_is_any_nan(in)) {
-float_raise(float_flag_invalid, fp_status);
-return ~0ULL;
-}
-if (float64_is_zero(in)) {
-return 0;
-}
-if (sign) {
-float_raise(float_flag_invalid, fp_status);
-return 0;
-}
-if (float64_lt(in, float64_half, fp_status)) {
-/* Near zero, captures large fracshifts, denorms, etc */
-float_raise(float_flag_inexact, fp_status);
-switch (get_float_rounding_mode(fp_status)) {
-case float_round_down:
-if (will_negate) {
-return 1;
-} else {
-return 0;
-}
-case float_round_up:
-if (!will_negate) {
-return 1;
-} else {
-return 0;
-}
-default:
-return 0;/* nearest or towards zero */
-}
-}
-return float64_to_uint64(in, fp_status);
-}
-
-static void clr_float_exception_flags(uint8_t flag, float_status *fp_status)
-{
-uint8_t flags = fp_status->float_exception_flags;
-flags &= ~flag;
-set_float_exception_flags(flags, fp_status);
-}
-
-static uint32_t conv_df_to_4u_n(float64 fp64, int will_negate,
-float_status *fp_status)
-{
-uint64_t tmp;
-tmp = conv_f64_to_8u_n(fp64, will_negate, fp_status);
-if (tmp > 0xULL) {
-clr_float_exception_flags(float_flag_inexact, fp_status);
-float_raise(float_flag_invalid, fp_status);
-return ~0U;
-}
-return (uint32_t)tmp;
-}
-
-uint64_t conv_df_to_8u(float64 in, float_status *fp_status)
-{
-return conv_f64_to_8u_n(in, 0, fp_status);
-}
-
-uint32_t conv_df_to_4u(float64 in, float_status *fp_status)
-{
-return conv_df_to_4u_n(in, 0, fp_status);
-}
-
-int64_t conv_df_to_8s(float64 in, float_status *fp_status)
-{
-uint8_t sign = float64_is_neg(in);
-uint64_t tmp;
-if (float64_is_any_nan(in)) {
-float_raise(float_flag_invalid, fp_status);
-return -1;
-}
-if (sign) {
-float64 minus_fp64 = float64_abs(in);
-tmp = conv_f64_to_8u_n(minus_fp64, 1, fp_status);
-} else {
-tmp = conv_f64_to_8u_n(in, 0, fp_status);
-}
-if (tmp > (LL_MAX_POS + sign)) {
-clr_float_exception_flags(float_flag_inexact, fp_status);
-float_raise(float_flag_invalid, fp_status);
-tmp = (LL_MAX_POS + sign);
-}
-if (sign) {
-return -tmp;
-} else {
-return tmp;
-}
-}
-
-int32_t conv_df_to_4s(float64 in, float_status *fp_status)
-{
-uint8_t sign = float64_is_neg(in);
-uint64_t tmp;
-if (float64_is_any_nan(in)) {
-float_raise(float_flag_invalid, fp_status);
-return -1;

Re: [PATCH qemu v16] spapr: Implement Open Firmware client interface

[David Gibson]

On Tue, Mar 23, 2021 at 01:58:30PM +1100, Alexey Kardashevskiy wrote:
> The PAPR platform which describes an OS environment that's presented by
> a combination of a hypervisor and firmware. The features it specifies
> require collaboration between the firmware and the hypervisor.
> 
> Since the beginning, the runtime component of the firmware (RTAS) has
> been implemented as a 20 byte shim which simply forwards it to
> a hypercall implemented in qemu. The boot time firmware component is
> SLOF - but a build that's specific to qemu, and has always needed to be
> updated in sync with it. Even though we've managed to limit the amount
> of runtime communication we need between qemu and SLOF, there's some,
> and it has become increasingly awkward to handle as we've implemented
> new features.
> 
> This implements a boot time OF client interface (CI) which is
> enabled by a new "x-vof" pseries machine option (stands for "Virtual Open
> Firmware). When enabled, QEMU implements the custom H_OF_CLIENT hcall
> which implements Open Firmware Client Interface (OF CI). This allows
> using a smaller stateless firmware which does not have to manage
> the device tree.
> 
> The new "vof.bin" firmware image is included with source code under
> pc-bios/. It also includes RTAS blob.
> 
> This implements a handful of CI methods just to get -kernel/-initrd
> working. In particular, this implements the device tree fetching and
> simple memory allocator - "claim" (an OF CI memory allocator) and updates
> "/memory@0/available" to report the client about available memory.
> 
> This implements changing some device tree properties which we know how
> to deal with, the rest is ignored. To allow changes, this skips
> fdt_pack() when x-vof=on as not packing the blob leaves some room for
> appending.
> 
> In absence of SLOF, this assigns phandles to device tree nodes to make
> device tree traversing work.
> 
> When x-vof=on, this adds "/chosen" every time QEMU (re)builds a tree.
> 
> This adds basic instances support which are managed by a hash map
> ihandle -> [phandle].
> 
> Before the guest started, the used memory is:
> 0..e60 - the initial firmware
> 8000..1 - stack
> 40.. - kernel
> 3ea.. - initramdisk
> 
> This OF CI does not implement "interpret".
> 
> Unlike SLOF, this does not format uninitialized nvram. Instead, this
> includes a disk image with pre-formatted nvram.
> 
> With this basic support, this can only boot into kernel directly.
> However this is just enough for the petitboot kernel and initradmdisk to
> boot from any possible source. Note this requires reasonably recent guest
> kernel with:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=df5be5be8735
> 
> The immediate benefit is much faster booting time which especially
> crucial with fully emulated early CPU bring up environments. Also this
> may come handy when/if GRUB-in-the-userspace sees light of the day.
> 
> This separates VOF and sPAPR in a hope that VOF bits may be reused by
> other POWERPC boards which do not support pSeries.
> 
> This is coded in assumption that later on we might be adding support for
> booting from QEMU backends (blockdev is the first candidate) without
> devices/drivers in between as OF1275 does not require that and
> it is quite easy to so.
> 
> Signed-off-by: Alexey Kardashevskiy 

I have some comments below, but they're basically all trivial at this
point.  We've missed qemu-6.0 obviously, but I'm hoping I can merge
the next spin to my ppc-for-6.1 tree.

> ---
> 
> The example command line is:
> 
> /home/aik/pbuild/qemu-killslof-localhost-ppc64/qemu-system-ppc64 \
> -nodefaults \
> -chardev stdio,id=STDIO0,signal=off,mux=on \
> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
> -mon id=MON0,chardev=STDIO0,mode=readline \
> -nographic \
> -vga none \
> -enable-kvm \
> -m 2G \
> -machine 
> pseries,x-vof=on,cap-cfpc=broken,cap-sbbc=broken,cap-ibs=broken,cap-ccf-assist=off
>  \
> -kernel pbuild/kernel-le-guest/vmlinux \
> -initrd pb/rootfs.cpio.xz \
> -drive 
> id=DRIVE0,if=none,file=./p/qemu-killslof/pc-bios/vof-nvram.bin,format=raw \

Removing the need for a prebuild NVRAM image is something I'd like to
see as a followup.

> -global spapr-nvram.drive=DRIVE0 \
> -snapshot \
> -smp 8,threads=8 \
> -L /home/aik/t/qemu-ppc64-bios/ \
> -trace events=qemu_trace_events \
> -d guest_errors \
> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.tmux26 \
> -mon chardev=SOCKET0,mode=control
> 
> ---
> Changes:
> v16:
> * rebased on dwg/ppc-for-6.1
> * s/SpaprVofInterface/VofMachineInterface/
> 
> v15:
> * bugfix: claimed memory for the VOF itself
> * ditched OF_STACK_ADDR and allocate one instead, now it starts from 0x8000
> because it is aligned to its size (no particular reason though)
> * coding style
> * moved nvram.bin up one level
> * ditched bool in the firmware
> * made debugging code conditional using trace_event_get_state() + 
> qemu_loglevel_mask()
> * renamed the CAS interface t

[PATCH v4 04/10] Remove migrate_set_block_enabled in checkpoint

[leirao]

From: "Rao, Lei" 

We can detect disk migration in migrate_prepare, if disk migration
is enabled in COLO mode, we can directly report an error.and there
is no need to disable block migration at every checkpoint.

Signed-off-by: Lei Rao 
Signed-off-by: Zhang Chen 
Reviewed-by: Li Zhijian 
---
 migration/colo.c  | 6 --
 migration/migration.c | 4 
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/migration/colo.c b/migration/colo.c
index de27662..1aaf316 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -435,12 +435,6 @@ static int colo_do_checkpoint_transaction(MigrationState 
*s,
 if (failover_get_state() != FAILOVER_STATUS_NONE) {
 goto out;
 }
-
-/* Disable block migration */
-migrate_set_block_enabled(false, &local_err);
-if (local_err) {
-goto out;
-}
 qemu_mutex_lock_iothread();
 
 #ifdef CONFIG_REPLICATION
diff --git a/migration/migration.c b/migration/migration.c
index ca8b97b..4578f22 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2219,6 +2219,10 @@ static bool migrate_prepare(MigrationState *s, bool blk, 
bool blk_inc,
 }
 
 if (blk || blk_inc) {
+if (migrate_colo_enabled()) {
+error_setg(errp, "No disk migration is required in COLO mode");
+return false;
+}
 if (migrate_use_block() || migrate_use_block_incremental()) {
 error_setg(errp, "Command options are incompatible with "
"current migration capabilities");
-- 
1.8.3.1

[PATCH v4 06/10] Add the function of colo_compare_cleanup

[leirao]

From: "Rao, Lei" 

This patch fixes the following:
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x7f6ae4559859 in __GI_abort () at abort.c:79
#2  0x559aaa386720 in error_exit (err=16, msg=0x559aaa5973d0 
<__func__.16227> "qemu_mutex_destroy") at util/qemu-thread-posix.c:36
#3  0x559aaa3868c5 in qemu_mutex_destroy (mutex=0x559aabffe828) at 
util/qemu-thread-posix.c:69
#4  0x559aaa2f93a8 in char_finalize (obj=0x559aabffe800) at 
chardev/char.c:285
#5  0x559aaa23318a in object_deinit (obj=0x559aabffe800, 
type=0x559aabfd7d20) at qom/object.c:606
#6  0x559aaa2331b8 in object_deinit (obj=0x559aabffe800, 
type=0x559aabfd9060) at qom/object.c:610
#7  0x559aaa233200 in object_finalize (data=0x559aabffe800) at 
qom/object.c:620
#8  0x559aaa234202 in object_unref (obj=0x559aabffe800) at 
qom/object.c:1074
#9  0x559aaa2356b6 in object_finalize_child_property 
(obj=0x559aac0dac10, name=0x559aac778760 "compare0-0", opaque=0x559aabffe800) 
at qom/object.c:1584
#10 0x559aaa232f70 in object_property_del_all (obj=0x559aac0dac10) at 
qom/object.c:557
#11 0x559aaa2331ed in object_finalize (data=0x559aac0dac10) at 
qom/object.c:619
#12 0x559aaa234202 in object_unref (obj=0x559aac0dac10) at 
qom/object.c:1074
#13 0x559aaa2356b6 in object_finalize_child_property 
(obj=0x559aac0c75c0, name=0x559aac0dadc0 "chardevs", opaque=0x559aac0dac10) at 
qom/object.c:1584
#14 0x559aaa233071 in object_property_del_child (obj=0x559aac0c75c0, 
child=0x559aac0dac10, errp=0x0) at qom/object.c:580
#15 0x559aaa233155 in object_unparent (obj=0x559aac0dac10) at 
qom/object.c:599
#16 0x559aaa2fb721 in qemu_chr_cleanup () at chardev/char.c:1159
#17 0x559aa9f9b110 in main (argc=54, argv=0x7ffeb62fa998, 
envp=0x7ffeb62fab50) at vl.c:4539

When chardev is cleaned up, chr_write_lock needs to be destroyed. But
the colo-compare module is not cleaned up normally before it when the
guest poweroff. It is holding chr_write_lock at this time. This will
cause qemu crash.So we add the function of colo_compare_cleanup() before
qemu_chr_cleanup() to fix the bug.

Signed-off-by: Lei Rao 
---
 net/colo-compare.c | 10 ++
 net/colo-compare.h |  1 +
 net/net.c  |  4 
 3 files changed, 15 insertions(+)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index c142c08..5b538f4 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -1402,6 +1402,16 @@ static void colo_compare_init(Object *obj)
  compare_set_vnet_hdr);
 }
 
+void colo_compare_cleanup(void)
+{
+CompareState *tmp = NULL;
+CompareState *n = NULL;
+
+QTAILQ_FOREACH_SAFE(tmp, &net_compares, next, n) {
+object_unparent(OBJECT(tmp));
+}
+}
+
 static void colo_compare_finalize(Object *obj)
 {
 CompareState *s = COLO_COMPARE(obj);
diff --git a/net/colo-compare.h b/net/colo-compare.h
index 22ddd51..b055270 100644
--- a/net/colo-compare.h
+++ b/net/colo-compare.h
@@ -20,5 +20,6 @@
 void colo_notify_compares_event(void *opaque, int event, Error **errp);
 void colo_compare_register_notifier(Notifier *notify);
 void colo_compare_unregister_notifier(Notifier *notify);
+void colo_compare_cleanup(void);
 
 #endif /* QEMU_COLO_COMPARE_H */
diff --git a/net/net.c b/net/net.c
index 725a4e1..8fcb2e7 100644
--- a/net/net.c
+++ b/net/net.c
@@ -53,6 +53,7 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 #include "sysemu/sysemu.h"
+#include "net/colo-compare.h"
 #include "net/filter.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi/hmp-output-visitor.h"
@@ -1463,6 +1464,9 @@ void net_cleanup(void)
 {
 NetClientState *nc;
 
+/*cleanup colo compare module for COLO*/
+colo_compare_cleanup();
+
 /* We may del multiple entries during qemu_del_net_client(),
  * so QTAILQ_FOREACH_SAFE() is also not safe here.
  */
-- 
1.8.3.1

[PATCH v4 08/10] Reduce the PVM stop time during Checkpoint

[leirao]

From: "Rao, Lei" 

When flushing memory from ram cache to ram during every checkpoint
on secondary VM, we can copy continuous chunks of memory instead of
4096 bytes per time to reduce the time of VM stop during checkpoint.

Signed-off-by: Lei Rao 
---
 migration/ram.c | 45 ++---
 1 file changed, 42 insertions(+), 3 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index c69a8e0..a258466 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -822,6 +822,39 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs, 
RAMBlock *rb,
 return next;
 }
 
+/*
+ * colo_bitmap_find_diry:find contiguous dirty pages from start
+ *
+ * Returns the page offset within memory region of the start of the contiguout
+ * dirty page
+ *
+ * @rs: current RAM state
+ * @rb: RAMBlock where to search for dirty pages
+ * @start: page where we start the search
+ * @num: the number of contiguous dirty pages
+ */
+static inline
+unsigned long colo_bitmap_find_dirty(RAMState *rs, RAMBlock *rb,
+ unsigned long start, unsigned long *num)
+{
+unsigned long size = rb->used_length >> TARGET_PAGE_BITS;
+unsigned long *bitmap = rb->bmap;
+unsigned long first, next;
+
+if (ramblock_is_ignored(rb)) {
+return size;
+}
+
+first = find_next_bit(bitmap, size, start);
+if (first >= size) {
+return first;
+}
+next = find_next_zero_bit(bitmap, size, first + 1);
+assert(next >= first);
+*num = next - first;
+return first;
+}
+
 static inline bool migration_bitmap_clear_dirty(RAMState *rs,
 RAMBlock *rb,
 unsigned long page)
@@ -3666,6 +3699,8 @@ void colo_flush_ram_cache(void)
 void *dst_host;
 void *src_host;
 unsigned long offset = 0;
+unsigned long num = 0;
+unsigned long i = 0;
 
 memory_global_dirty_log_sync();
 WITH_RCU_READ_LOCK_GUARD() {
@@ -3679,19 +3714,23 @@ void colo_flush_ram_cache(void)
 block = QLIST_FIRST_RCU(&ram_list.blocks);
 
 while (block) {
-offset = migration_bitmap_find_dirty(ram_state, block, offset);
+offset = colo_bitmap_find_dirty(ram_state, block, offset, &num);
 
 if (((ram_addr_t)offset) << TARGET_PAGE_BITS
 >= block->used_length) {
 offset = 0;
+num = 0;
 block = QLIST_NEXT_RCU(block, next);
 } else {
-migration_bitmap_clear_dirty(ram_state, block, offset);
+for (i = 0; i < num; i++) {
+migration_bitmap_clear_dirty(ram_state, block, offset + i);
+}
 dst_host = block->host
  + (((ram_addr_t)offset) << TARGET_PAGE_BITS);
 src_host = block->colo_cache
  + (((ram_addr_t)offset) << TARGET_PAGE_BITS);
-memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
+memcpy(dst_host, src_host, TARGET_PAGE_SIZE * num);
+offset += num;
 }
 }
 }
-- 
1.8.3.1

[PATCH v4 02/10] Fix the qemu crash when guest shutdown during checkpoint

[leirao]

From: "Rao, Lei" 

This patch fixes the following:
qemu-system-x86_64: invalid runstate transition: 'colo' ->'shutdown'
Aborted (core dumped)

Signed-off-by: Lei Rao 
Reviewed-by: Li Zhijian 
---
 softmmu/runstate.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/softmmu/runstate.c b/softmmu/runstate.c
index ce8977c..1564057 100644
--- a/softmmu/runstate.c
+++ b/softmmu/runstate.c
@@ -126,6 +126,7 @@ static const RunStateTransition runstate_transitions_def[] 
= {
 { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH },
 
 { RUN_STATE_COLO, RUN_STATE_RUNNING },
+{ RUN_STATE_COLO, RUN_STATE_SHUTDOWN},
 
 { RUN_STATE_RUNNING, RUN_STATE_DEBUG },
 { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR },
-- 
1.8.3.1

[PATCH v4 03/10] Optimize the function of filter_send

[leirao]

From: "Rao, Lei" 

The iov_size has been calculated in filter_send(). we can directly
return the size.In this way, this is no need to repeat calculations
in filter_redirector_receive_iov();

Signed-off-by: Lei Rao 
Reviewed-by: Li Zhijian 
---
 net/filter-mirror.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/net/filter-mirror.c b/net/filter-mirror.c
index f8e6500..f20240c 100644
--- a/net/filter-mirror.c
+++ b/net/filter-mirror.c
@@ -88,7 +88,7 @@ static int filter_send(MirrorState *s,
 goto err;
 }
 
-return 0;
+return size;
 
 err:
 return ret < 0 ? ret : -EIO;
@@ -159,7 +159,7 @@ static ssize_t filter_mirror_receive_iov(NetFilterState *nf,
 int ret;
 
 ret = filter_send(s, iov, iovcnt);
-if (ret) {
+if (ret < 0) {
 error_report("filter mirror send failed(%s)", strerror(-ret));
 }
 
@@ -182,10 +182,10 @@ static ssize_t 
filter_redirector_receive_iov(NetFilterState *nf,
 
 if (qemu_chr_fe_backend_connected(&s->chr_out)) {
 ret = filter_send(s, iov, iovcnt);
-if (ret) {
+if (ret < 0) {
 error_report("filter redirector send failed(%s)", strerror(-ret));
 }
-return iov_size(iov, iovcnt);
+return ret;
 } else {
 return 0;
 }
-- 
1.8.3.1

[PATCH v4 10/10] Fixed calculation error of pkt->header_size in fill_pkt_tcp_info()

[leirao]

From: "Rao, Lei" 

The data pointer has skipped vnet_hdr_len in the function of
parse_packet_early().So, we can not subtract vnet_hdr_len again
when calculating pkt->header_size in fill_pkt_tcp_info(). Otherwise,
it will cause network packet comparsion errors and greatly increase
the frequency of checkpoints.

Signed-off-by: Lei Rao 
Signed-off-by: Zhang Chen 
Reviewed-by: Li Zhijian 
---
 net/colo-compare.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 5b538f4..b100e7b 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -211,7 +211,7 @@ static void fill_pkt_tcp_info(void *data, uint32_t *max_ack)
 pkt->tcp_ack = ntohl(tcphd->th_ack);
 *max_ack = *max_ack > pkt->tcp_ack ? *max_ack : pkt->tcp_ack;
 pkt->header_size = pkt->transport_header - (uint8_t *)pkt->data
-   + (tcphd->th_off << 2) - pkt->vnet_hdr_len;
+   + (tcphd->th_off << 2);
 pkt->payload_size = pkt->size - pkt->header_size;
 pkt->seq_end = pkt->tcp_seq + pkt->payload_size;
 pkt->flags = tcphd->th_flags;
-- 
1.8.3.1

[PATCH v4 07/10] Reset the auto-converge counter at every checkpoint.

[leirao]

From: "Rao, Lei" 

if we don't reset the auto-converge counter,
it will continue to run with COLO running,
and eventually the system will hang due to the
CPU throttle reaching DEFAULT_MIGRATE_MAX_CPU_THROTTLE.

Signed-off-by: Lei Rao 
Reviewed-by: Dr. David Alan Gilbert 
---
 migration/colo.c | 4 
 migration/ram.c  | 9 +
 migration/ram.h  | 1 +
 3 files changed, 14 insertions(+)

diff --git a/migration/colo.c b/migration/colo.c
index 1aaf316..723ffb8 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -459,6 +459,10 @@ static int colo_do_checkpoint_transaction(MigrationState 
*s,
 if (ret < 0) {
 goto out;
 }
+
+if (migrate_auto_converge()) {
+mig_throttle_counter_reset();
+}
 /*
  * Only save VM's live state, which not including device state.
  * TODO: We may need a timeout mechanism to prevent COLO process
diff --git a/migration/ram.c b/migration/ram.c
index 40e7895..c69a8e0 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -652,6 +652,15 @@ static void mig_throttle_guest_down(uint64_t 
bytes_dirty_period,
 }
 }
 
+void mig_throttle_counter_reset(void)
+{
+RAMState *rs = ram_state;
+
+rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+rs->num_dirty_pages_period = 0;
+rs->bytes_xfer_prev = ram_counters.transferred;
+}
+
 /**
  * xbzrle_cache_zero_page: insert a zero page in the XBZRLE cache
  *
diff --git a/migration/ram.h b/migration/ram.h
index 6378bb3..3f78175 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -50,6 +50,7 @@ bool ramblock_is_ignored(RAMBlock *block);
 int xbzrle_cache_resize(uint64_t new_size, Error **errp);
 uint64_t ram_bytes_remaining(void);
 uint64_t ram_bytes_total(void);
+void mig_throttle_counter_reset(void);
 
 uint64_t ram_pagesize_summary(void);
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
-- 
1.8.3.1

[PATCH v4 01/10] Remove some duplicate trace code.

[leirao]

From: "Rao, Lei" 

There is the same trace code in the colo_compare_packet_payload.

Signed-off-by: Lei Rao 
Reviewed-by: Li Zhijian 
---
 net/colo-compare.c | 13 -
 1 file changed, 13 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 9d1ad99..c142c08 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -590,19 +590,6 @@ static int colo_packet_compare_other(Packet *spkt, Packet 
*ppkt)
 uint16_t offset = ppkt->vnet_hdr_len;
 
 trace_colo_compare_main("compare other");
-if (trace_event_get_state_backends(TRACE_COLO_COMPARE_IP_INFO)) {
-char pri_ip_src[20], pri_ip_dst[20], sec_ip_src[20], sec_ip_dst[20];
-
-strcpy(pri_ip_src, inet_ntoa(ppkt->ip->ip_src));
-strcpy(pri_ip_dst, inet_ntoa(ppkt->ip->ip_dst));
-strcpy(sec_ip_src, inet_ntoa(spkt->ip->ip_src));
-strcpy(sec_ip_dst, inet_ntoa(spkt->ip->ip_dst));
-
-trace_colo_compare_ip_info(ppkt->size, pri_ip_src,
-   pri_ip_dst, spkt->size,
-   sec_ip_src, sec_ip_dst);
-}
-
 if (ppkt->size != spkt->size) {
 trace_colo_compare_main("Other: payload size of packets are 
different");
 return -1;
-- 
1.8.3.1

[PATCH v4 00/10] Fixed some bugs and optimized some codes for COLO

[leirao]

From: Rao, Lei 

Changes since v3:
--Remove cpu_throttle_stop from mig_throttle_counter_reset.

Changes since v2:
--Add a function named packet_new_nocopy.
--Continue to optimize the function of colo_flush_ram_cache.

Changes since v1:
--Reset the state of the auto-converge counters at every checkpoint 
instead of directly disabling.
--Treat the filter_send function returning zero as a normal case.

The series of patches include:
Fixed some bugs of qemu crash.
Optimized some code to reduce the time of checkpoint.
Remove some unnecessary code to improve COLO.

Rao, Lei (10):
  Remove some duplicate trace code.
  Fix the qemu crash when guest shutdown during checkpoint
  Optimize the function of filter_send
  Remove migrate_set_block_enabled in checkpoint
  Add a function named packet_new_nocopy for COLO.
  Add the function of colo_compare_cleanup
  Reset the auto-converge counter at every checkpoint.
  Reduce the PVM stop time during Checkpoint
  Add the function of colo_bitmap_clear_diry.
  Fixed calculation error of pkt->header_size in fill_pkt_tcp_info()

 migration/colo.c  | 10 +++
 migration/migration.c |  4 +++
 migration/ram.c   | 75 ---
 migration/ram.h   |  1 +
 net/colo-compare.c| 25 -
 net/colo-compare.h|  1 +
 net/colo.c| 23 
 net/colo.h|  1 +
 net/filter-mirror.c   |  8 +++---
 net/filter-rewriter.c |  3 +--
 net/net.c |  4 +++
 softmmu/runstate.c|  1 +
 12 files changed, 127 insertions(+), 29 deletions(-)

-- 
1.8.3.1

Questions about qtest on interrupt controller

[LIU Zhiwei]

Hi folks,

When I am writing  qtest  cases for an interrupt controller,  I 
encounter some problems.


*1) Can't intercept both the irq in and irq out for the same device.*

It's a necessary feature for an interrupt controller qtest. However, as 
you can see  from qtest.c,  when a device has intercepted irq out or irq 
in, it can't intercept another.


" if (irq_intercept_dev) {

    qtest_send_prefix(chr);

    if (irq_intercept_dev != dev) {

    qtest_send(chr, "FAIL IRQ intercept already enabled\n");

    } else {

    qtest_send(chr, "OK\n");

    }

    return; }

"

In my opinion,  this code in qtest is protecting from intercepting irqs 
repeatedly. But irq in and out are different directions, they should be 
intercepted simultaneously. Is it right?


*2) Can't get  the right IRQ num.*

If  I  intercept an IRQ out,  I can get the num of IRQ if the IRQ 
raises, .  But the really IRQ I want to get is the IRQ that the 
interrupt controller selected out. However I can't get this IRQ from 
qtest_irq_handler.


Are there already some methods to surmount these problems?  Any advice 
is well appreciated. Thanks very much.


Zhiwei

Re: [PULL V2 16/20] qapi: net: Add query-netdev command

[Jason Wang]

在 2021/3/17 上午11:34, Jason Wang 写道:


在 2021/3/17 上午5:37, Peter Maydell 写道:

On Mon, 15 Mar 2021 at 09:15, Jason Wang  wrote:

From: Alexey Kirillov 

The query-netdev command is used to get the configuration of the 
current

network device backends (netdevs).
This is the QMP analog of the HMP command "info network" but only for
netdevs (i.e. excluding NIC and hubports).

The query-netdev command returns an array of objects of the NetdevInfo
type, which are an extension of Netdev type. It means that response can
be used for netdev-add after small modification. This can be useful for
recreate the same netdev configuration.

Information about the network device is filled in when it is created or
modified and is available through the NetClientState->stored_config.

Signed-off-by: Alexey Kirillov 
Acked-by: Markus Armbruster 
Signed-off-by: Jason Wang 
---

Hi; Coverity complains about a memory leak in this code
(CID 1450842):

@@ -581,15 +693,25 @@ static int net_slirp_init(NetClientState 
*peer, const char *model,

  s->poll_notifier.notify = net_slirp_poll_notify;
  main_loop_poll_add_notifier(&s->poll_notifier);

+    stored_hostfwd = &stored->hostfwd;
+    stored_guestfwd = &stored->guestfwd;
+
  for (config = slirp_configs; config; config = config->next) {
+    String *element = g_new0(String, 1);

Here we allocate memory...


+
+    element->str = g_strdup(config->str);
  if (config->flags & SLIRP_CFG_HOSTFWD) {
  if (slirp_hostfwd(s, config->str, errp) < 0) {
  goto error;

...but if we take this error-exit path we have neither freed nor
kept a pointer to that memory.



Yes.





  }
+    stored->has_hostfwd = true;
+    QAPI_LIST_APPEND(stored_hostfwd, element);
  } else {
  if (slirp_guestfwd(s, config->str, errp) < 0) {
  goto error;

Similarly here.


  }
+    stored->has_guestfwd = true;
+    QAPI_LIST_APPEND(stored_guestfwd, element);
  }
  }
  #ifndef _WIN32

More generally, what state is the net backend init function
supposed to leave 'stored' in if it fails? Is it the backend's
responsibility to free everything that it might have allocated
and left a pointer to? eg if we did
    stored->hostname = g_strdup(vhostname);
do we need to go back and free(stored->hostname) ? Or is the caller
guaranteeing to clean up 'stored' somehow ? Or is the backend
supposed to not touch 'stored' until it's sure it's going to
succeed ? (presumably not, as the current code does not do this...)



Clean and free in the function that do the allocation seems better 
(self-conatined).





This commit has no comments describing or documenting the
API requirements the new functionality imposes on a net backend:
could we have a followup patch which adds some documentation,
please, so that authors of future backends know what they have to
implement ?



Alexey, plase send patches to fix the above issues and document the API.

Thanks



Alexey, any update on this. If it takes time I tend to revert this and 
let's re-try for 6.1?


Thanks







thanks
-- PMM

RE: [PATCH v3 07/10] Reset the auto-converge counter at every checkpoint.

[Rao, Lei]

After testing, I think you are right.
Will remove the cpu_throttle_stop() in V4.

Thanks,
Lei.

-Original Message-
From: Dr. David Alan Gilbert  
Sent: Thursday, March 25, 2021 12:40 AM
To: Rao, Lei 
Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; 
jasow...@redhat.com; quint...@redhat.com; pbonz...@redhat.com; 
lukasstra...@web.de; qemu-devel@nongnu.org
Subject: Re: [PATCH v3 07/10] Reset the auto-converge counter at every 
checkpoint.

* leirao (lei@intel.com) wrote:
> From: "Rao, Lei" 
> 
> if we don't reset the auto-converge counter, it will continue to run 
> with COLO running, and eventually the system will hang due to the CPU 
> throttle reaching DEFAULT_MIGRATE_MAX_CPU_THROTTLE.
> 
> Signed-off-by: Lei Rao 
> ---
>  migration/colo.c |  4 
>  migration/ram.c  | 10 ++
>  migration/ram.h  |  1 +
>  3 files changed, 15 insertions(+)
> 
> diff --git a/migration/colo.c b/migration/colo.c index 
> 1aaf316..723ffb8 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -459,6 +459,10 @@ static int colo_do_checkpoint_transaction(MigrationState 
> *s,
>  if (ret < 0) {
>  goto out;
>  }
> +
> +if (migrate_auto_converge()) {
> +mig_throttle_counter_reset();
> +}
>  /*
>   * Only save VM's live state, which not including device state.
>   * TODO: We may need a timeout mechanism to prevent COLO process 
> diff --git a/migration/ram.c b/migration/ram.c index 72143da..e795a8d 
> 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -652,6 +652,16 @@ static void mig_throttle_guest_down(uint64_t 
> bytes_dirty_period,
>  }
>  }
>  
> +void mig_throttle_counter_reset(void) {
> +RAMState *rs = ram_state;
> +
> +rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> +rs->num_dirty_pages_period = 0;
> +rs->bytes_xfer_prev = ram_counters.transferred;
> +cpu_throttle_stop();

I think this is right, so:

Reviewed-by: Dr. David Alan Gilbert 

however, do you really need the cpu_throttle_stop?  Shouldn't the previous 
iteration have called that in migration_iteration_finish() ?

Dave

> +}
> +
>  /**
>   * xbzrle_cache_zero_page: insert a zero page in the XBZRLE cache
>   *
> diff --git a/migration/ram.h b/migration/ram.h index 6378bb3..3f78175 
> 100644
> --- a/migration/ram.h
> +++ b/migration/ram.h
> @@ -50,6 +50,7 @@ bool ramblock_is_ignored(RAMBlock *block);  int 
> xbzrle_cache_resize(uint64_t new_size, Error **errp);  uint64_t 
> ram_bytes_remaining(void);  uint64_t ram_bytes_total(void);
> +void mig_throttle_counter_reset(void);
>  
>  uint64_t ram_pagesize_summary(void);
>  int ram_save_queue_pages(const char *rbname, ram_addr_t start, 
> ram_addr_t len);
> --
> 1.8.3.1
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [PATCH 3/3] tests: Aspeed HACE Scatter-Gather tests

[Joel Stanley]

On Wed, 24 Mar 2021 at 22:39, Klaus Heinrich Kiwi
 wrote:
>
> Expand current Aspeed HACE testsuite to also include Scatter-Gather of
> sha256 and sha512 operations.
>
> Signed-off-by: Klaus Heinrich Kiwi 
> ---
>  tests/qtest/aspeed_hace-test.c | 164 ++---
>  1 file changed, 153 insertions(+), 11 deletions(-)
>
> diff --git a/tests/qtest/aspeed_hace-test.c b/tests/qtest/aspeed_hace-test.c
> index 2b624b6b09..ae3f963449 100644
> --- a/tests/qtest/aspeed_hace-test.c
> +++ b/tests/qtest/aspeed_hace-test.c
> @@ -35,6 +35,11 @@
>  #define HACE_HASH_DATA_LEN   0x2c
>  #define HACE_HASH_CMD0x30
>
> +struct aspeed_sg_list {
> +uint32_t len;
> +uint32_t phy_addr;
> +} __attribute__ ((__packed__));
> +
>  /*
>   * Test vector is the ascii "abc"
>   *
> @@ -63,6 +68,33 @@ static const uint8_t test_result_md5[] = {
>  0x90, 0x01, 0x50, 0x98, 0x3c, 0xd2, 0x4f, 0xb0, 0xd6, 0x96, 0x3f, 0x7d,
>  0x28, 0xe1, 0x7f, 0x72};
>
> +/*
> + * The Scatter-Gather Test vector is the ascii "abc" "def" "ghi", broken
> + * into blocks of 3 characters as shown
> + *
> + * Expected results were generated using command line utitiles:
> + *
> + *  echo -n -e 'abcdefghi' | dd of=/tmp/test
> + *  for hash in sha512sum sha256sum; do $hash /tmp/test; done
> + *
> + */
> +static const uint8_t test_vector_sg1[] = {0x61, 0x62, 0x63};
> +static const uint8_t test_vector_sg2[] = {0x64, 0x65, 0x66};
> +static const uint8_t test_vector_sg3[] = {0x67, 0x68, 0x69};
> +
> +static const uint8_t test_result_sg_sha512[] = {
> +0xf2, 0x2d, 0x51, 0xd2, 0x52, 0x92, 0xca, 0x1d, 0x0f, 0x68, 0xf6, 0x9a,
> +0xed, 0xc7, 0x89, 0x70, 0x19, 0x30, 0x8c, 0xc9, 0xdb, 0x46, 0xef, 0xb7,
> +0x5a, 0x03, 0xdd, 0x49, 0x4f, 0xc7, 0xf1, 0x26, 0xc0, 0x10, 0xe8, 0xad,
> +0xe6, 0xa0, 0x0a, 0x0c, 0x1a, 0x5f, 0x1b, 0x75, 0xd8, 0x1e, 0x0e, 0xd5,
> +0xa9, 0x3c, 0xe9, 0x8d, 0xc9, 0xb8, 0x33, 0xdb, 0x78, 0x39, 0x24, 0x7b,
> +0x1d, 0x9c, 0x24, 0xfe};
> +
> +static const uint8_t test_result_sg_sha256[] = {
> +0x19, 0xcc, 0x02, 0xf2, 0x6d, 0xf4, 0x3c, 0xc5, 0x71, 0xbc, 0x9e, 0xd7,
> +0xb0, 0xc4, 0xd2, 0x92, 0x24, 0xa3, 0xec, 0x22, 0x95, 0x29, 0x22, 0x17,
> +0x25, 0xef, 0x76, 0xd0, 0x21, 0xc8, 0x32, 0x6f};
> +
>
>  static void write_regs(QTestState *s, uint32_t base, uint32_t src,
> uint32_t length, uint32_t out, uint32_t method)
> @@ -167,28 +199,124 @@ static void test_sha512(const char *machine, const 
> uint32_t base,
>  test_result_sha512, sizeof(digest));
>  }
>
> +static void test_sha256_sg(const char *machine, const uint32_t base,
> +const uint32_t src_addr)
> +{
> +QTestState *s = qtest_init(machine);
> +
> +const uint32_t src_addr_1 = src_addr + 0x100;
> +const uint32_t src_addr_2 = src_addr + 0x200;
> +const uint32_t src_addr_3 = src_addr + 0x300;
> +const uint32_t digest_addr = src_addr + 0x400;
> +uint8_t digest[32] = {0};
> +struct aspeed_sg_list array[] = {
> +{ sizeof(test_vector_sg1),  src_addr_1},
> +{ sizeof(test_vector_sg2),  src_addr_2},
> +{ sizeof(test_vector_sg3) | 1u << 31,   src_addr_3},

These sizeofs are always going to be 3.

I assume 1 << 31 is to indicate the final entry? Perhaps add a define for it.

> +};
> +
> +/* Check engine is idle, no busy or irq bits set */
> +g_assert_cmphex(qtest_readl(s, base + HACE_STS), ==, 0);
> +
> +/* Write test vector into memory */
> +qtest_memwrite(s, src_addr_1, test_vector_sg1, sizeof(test_vector_sg1));
> +qtest_memwrite(s, src_addr_2, test_vector_sg2, sizeof(test_vector_sg2));
> +qtest_memwrite(s, src_addr_3, test_vector_sg3, sizeof(test_vector_sg3));

It would simplify your test case if you wrote the test vector to the
one memory location.

> +qtest_memwrite(s, src_addr, array, sizeof(array));
> +
> +write_regs(s, base, src_addr,
> +   (sizeof(test_vector_sg1)
> ++ sizeof(test_vector_sg2)
> ++ sizeof(test_vector_sg3)),
> +   digest_addr, HACE_ALGO_SHA256 | HACE_SG_EN);
> +
> +/* Check hash IRQ status is asserted */
> +g_assert_cmphex(qtest_readl(s, base + HACE_STS), ==, 0x0200);
> +
> +/* Clear IRQ status and check status is deasserted */
> +qtest_writel(s, base + HACE_STS, 0x0200);
> +g_assert_cmphex(qtest_readl(s, base + HACE_STS), ==, 0);
> +
> +/* Read computed digest from memory */
> +qtest_memread(s, digest_addr, digest, sizeof(digest));
> +
> +/* Check result of computation */
> +g_assert_cmpmem(digest, sizeof(digest),
> +test_result_sg_sha256, sizeof(digest));
> +}
> +


>  struct masks {
> -uint32_t src;
> +uint32_t src_direct;

You could leave this one the same.

> +uint32_t src_sg;

You add this, but haven't written a new test to use it.

>  uint32_t dest;
>  uint

Re: [PATCH v10 0/7] Pegasos2 emulation

[David Gibson]

On Tue, Mar 23, 2021 at 01:57:25PM +0100, BALATON Zoltan wrote:
> On Tue, 23 Mar 2021, David Gibson wrote:
> > On Wed, Mar 17, 2021 at 02:17:51AM +0100, BALATON Zoltan wrote:
> > > Hello,
> > > 
> > > This is adding a new PPC board called pegasos2. More info on it can be
> > > found at:
> > > 
> > > https://osdn.net/projects/qmiga/wiki/SubprojectPegasos2
> > > 
> > > Currently it needs a firmware ROM image that I cannot include due to
> > > original copyright holder (bPlan) did not release it under a free
> > > licence but I have plans to write a replacement in the future. With
> > > the original board firmware it can boot MorphOS now as:
> > > 
> > > qemu-system-ppc -M pegasos2 -cdrom morphos.iso -device ati-vga,romfile="" 
> > > -serial stdio
> > > 
> > > then enter "boot cd boot.img" at the firmware "ok" prompt as described
> > > in the MorphOS.readme. To boot Linux use same command line with e.g.
> > > -cdrom debian-8.11.0-powerpc-netinst.iso then enter
> > > "boot cd install/pegasos"
> > > 
> > > The last patch adds the actual board code after previous patches
> > > adding VT8231 and MV64361 system controller chip emulation.
> > 
> > I've applied 1..5 to a new ppc-for-6.1 branch.  Sorry it didn't make
> > it for 6.0, I just didn't have time to look this over until too late.
> 
> Thanks but maybe you should wait if the dropped original first patch can be
> reviewed now and brought back. It would be easier for me to resubmit whole
> series rebased than port the dropped patch on top of a partly applied
> series. Also first patches without the last two is not much useful as the
> vt8231 model these add would not be used by anything else.

Ok, I've dropped your patches from ppc-for-6.1 again, and I'll wait
for the next posting.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [PATCH v10 7/7] hw/ppc: Add emulation of Genesi/bPlan Pegasos II

[David Gibson]

On Wed, Mar 24, 2021 at 12:21:02PM +0100, BALATON Zoltan wrote:
> On Wed, 24 Mar 2021, David Gibson wrote:
> > On Tue, Mar 23, 2021 at 02:01:27PM +0100, BALATON Zoltan wrote:
> > > On Tue, 23 Mar 2021, David Gibson wrote:
> > > > On Wed, Mar 17, 2021 at 02:17:51AM +0100, BALATON Zoltan wrote:
> > > > > Add new machine called pegasos2 emulating the Genesi/bPlan Pegasos II,
> > > > > a PowerPC board based on the Marvell MV64361 system controller and the
> > > > > VIA VT8231 integrated south bridge/superio chips. It can run Linux,
> > > > > AmigaOS and a wide range of MorphOS versions. Currently a firmware ROM
> > > > > image is needed to boot and only MorphOS has a video driver to produce
> > > > > graphics output. Linux could work too but distros that supported this
> > > > > machine don't include usual video drivers so those only run with
> > > > > serial console for now.
> > > > > 
> > > > > Signed-off-by: BALATON Zoltan 
> > > > > Reviewed-by: Philippe Mathieu-Daudé 
> > > > > ---
> > > > >  MAINTAINERS |  10 ++
> > > > >  default-configs/devices/ppc-softmmu.mak |   2 +
> > > > >  hw/ppc/Kconfig  |   9 ++
> > > > >  hw/ppc/meson.build  |   2 +
> > > > >  hw/ppc/pegasos2.c   | 144 
> > > > > 
> > > > >  5 files changed, 167 insertions(+)
> > > > >  create mode 100644 hw/ppc/pegasos2.c
> > > > > 
> > > > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > > > index b6ab3d25a7..1c3c55ef09 100644
> > > > > --- a/MAINTAINERS
> > > > > +++ b/MAINTAINERS
> > > > > @@ -1353,6 +1353,16 @@ F: pc-bios/canyonlands.dt[sb]
> > > > >  F: pc-bios/u-boot-sam460ex-20100605.bin
> > > > >  F: roms/u-boot-sam460ex
> > > > > 
> > > > > +pegasos2
> > > > > +M: BALATON Zoltan 
> > > > > +R: David Gibson 
> > > > > +L: qemu-...@nongnu.org
> > > > > +S: Maintained
> > > > > +F: hw/ppc/pegasos2.c
> > > > > +F: hw/pci-host/mv64361.c
> > > > > +F: hw/pci-host/mv643xx.h
> > > > > +F: include/hw/pci-host/mv64361.h
> > > > 
> > > > Oh, sorry about the comment in the previous patch.
> > > > 
> > > > >  RISC-V Machines
> > > > >  ---
> > > > >  OpenTitan
> > > > > diff --git a/default-configs/devices/ppc-softmmu.mak 
> > > > > b/default-configs/devices/ppc-softmmu.mak
> > > > > index 61b78b844d..4535993d8d 100644
> > > > > --- a/default-configs/devices/ppc-softmmu.mak
> > > > > +++ b/default-configs/devices/ppc-softmmu.mak
> > > > > @@ -14,5 +14,7 @@ CONFIG_SAM460EX=y
> > > > >  CONFIG_MAC_OLDWORLD=y
> > > > >  CONFIG_MAC_NEWWORLD=y
> > > > > 
> > > > > +CONFIG_PEGASOS2=y
> > > > 
> > > > I don't think we can have this default to enabled while it requires a
> > > > non-free ROM to start.
> > > 
> > > Not having it enabled though does not help those who might want to use it 
> > > as
> > > they are not people who can compile their own QEMU but rely on binaries so
> > > adding it without also enabling it is like it wasn't there at all in
> > > practice.
> > 
> > Not convinced, sorry.  If it's not usable out of the box, having to
> > build from source is kind of expected.  Or you could convince someone
> 
> I accept your point however there's a difference of only needing a ROM image
> to be able to use it from your distro's binary and having to rebuild it from
> source. So to me needing a ROM does not make it expected having to rebuild
> it. Needing to configure it some way would make that expected.

Well, you could always lobby the distro to include it.  Honestly I
think the distros would probably choose not to include something for
such a niche case, even if it was enabled by default upstream.   At
least.. if they were paying attention, which they're likely not.  But
sneaking something in via a side route is not something I'm going to
encourage.

> > (or do it yourself) to provide prebuild binaries for this purpose that
> > have the right things enabled.
> 
> There are people who provide binaries with patches for such purposes but
> that limits the availability of it compared to having it in all distro
> binaries without further effort. As I said I also plan to solve this
> eventually but I'd probably need VOF for that. Will that be merged at last?

I hope so, yes.  I've finally had some time to review recent rounds.
It'll be 6.1 material as well, obviously.  Note that there are some
issues you'll have to sort out to use this for Pegasos which doesn't
have an actual hypervisor mode.

> Other alternatives would be modifying SLOF, OpenBIOS or OpenFirmware or
> rewrite SmartFirmware to free it from its non-distributable parts but I
> think VOF would be a simpler way also avoiding adding another full OF
> implementation to QEMU that already has more than there should be.
> 
> Regards,
> BALATON Zoltan
> 
> > > I can attempt to make some guests boot without a ROM but since
> > > guests expect an OpenFirmware client interface, I'll need something to
> > > provide that. I'm waiting for VOF to be merged for this.
> > > 
> > >

Re: [PATCH v10 6/7] hw/pci-host: Add emulation of Marvell MV64361 PPC system controller

[David Gibson]

On Wed, Mar 24, 2021 at 12:11:50PM +0100, BALATON Zoltan wrote:
> On Wed, 24 Mar 2021, David Gibson wrote:
> > On Tue, Mar 23, 2021 at 02:31:07PM +0100, BALATON Zoltan wrote:
> > > On Tue, 23 Mar 2021, David Gibson wrote:
> > > > On Wed, Mar 17, 2021 at 02:17:51AM +0100, BALATON Zoltan wrote:
> > [snip]
> > > > > +static void setup_mem_windows(MV64361State *s, uint32_t val)
> > > > > +{
> > > > > +MV64361PCIState *p;
> > > > > +MemoryRegion *mr;
> > > > > +uint32_t mask;
> > > > > +int i;
> > > > > +
> > > > > +val &= 0x1f;
> > > > > +for (mask = 1, i = 0; i < 21; i++, mask <<= 1) {
> > > > 
> > > > Having a loop, where nearly the entire body is a switch over the loop
> > > > variable seems a rather odd choice to me, compared to just unrolling
> > > > the whole thing.  Or alternatively, maybe more can be be factored out
> > > > of the switch into common body code.
> > > 
> > > The loop is really over the bits in val that say which memory regions to
> > 
> > I see that, but it doesn't change the point.
> > 
> > > enable so depending on this we need to touch different mem regions. For
> > > selecting those memory regions and what to do with them a switch seems to 
> > > be
> > > obvious choice. I probably can't factor out anything as these lines in
> > > switch cases are similar but all differ in some little details (like which
> > > PCI bus, name of the region, etc.). Unrolling could be possible but would
> > > just add lines between cases that's now in the loop head so I really don't
> > 
> > I see only 2 common lines, which basically balances about the case and
> > break lines in every switchcase.
> 
> Not sure what you mean by that. To me that means that these cannot be
> factored out as they are in the middle so can't put them neither before nor
> after the switch (without adding more local variables that would just make
> the result longer than it is now).

I'm saying that I don't think unrolling would actually make things
longer, because the small amount of duplication would balanced by
removing the switch boilerplate.

> Does that mean you prefer this to be unrolled then (written without a for
> just repeating the if ((val & mask) != (s->base_addr_enable & mask)) at
> every case)? That would also be longer by about 20 lines as we also log
> regions that are not in the switch that would need their enable bits checked
> separately if it's unrolled. Basically the trace is the common part of the
> loop and handling of the individual regions are branching out from the
> switch as they are different enough that factoring out the common parts is
> not shorter than this way due to then needing local variables to hold the
> different parts (name, address, size, base) the assigning of which are as
> many or more lines than the map_pci_region call that could be factored out
> that way.
> 
> I don't see why it is a problem to have a switch in a loop. If you insist I
> can try to turn the switch into if else but I don't see how would that be
> any better. Please explain a bit more what would you prefer here as I'm not
> sure what to do with this. To me this is the shortest and simplest way to
> write this.

Hmm... you know, it still seems kinda weird to me, but I am getting
too hung up on details here.  It's good enough, go with it.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [PATCH 0/4] DEVICE_NOT_DELETED/DEVICE_UNPLUG_ERROR QAPI events

[David Gibson]

On Wed, Mar 24, 2021 at 04:09:59PM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 3/23/21 10:40 PM, David Gibson wrote:
> > On Tue, Mar 23, 2021 at 02:10:22PM -0300, Daniel Henrique Barboza wrote:
> > > 
> > > 
> > > On 3/22/21 10:12 PM, David Gibson wrote:
> > > > On Fri, Mar 12, 2021 at 05:07:36PM -0300, Daniel Henrique Barboza wrote:
> > > > > Hi,
> > > > > 
> > > > > This series adds 2 new QAPI events, DEVICE_NOT_DELETED and
> > > > > DEVICE_UNPLUG_ERROR. They were (and are still being) discussed in [1].
> > > > > 
> > > > > Patches 1 and 3 are independent of the ppc patches and can be applied
> > > > > separately. Patches 2 and 4 are based on David's ppc-for-6.0 branch 
> > > > > and
> > > > > are dependent on the QAPI patches.
> > > > 
> > > > Implementation looks fine, but I think there's a bit more to discuss
> > > > before we can apply.
> > > > 
> > > > I think it would make sense to re-order this and put UNPLUG_ERROR
> > > > first.  Its semantics are clearer, and I think there's a stronger case
> > > > for it.
> > > 
> > > Alright
> > > 
> > > > 
> > > > I'm a bit less sold on DEVICE_NOT_DELETED, after consideration.  Does
> > > > it really tell the user/management anything useful beyond what
> > > > receiving neither a DEVICE_DELETED nor a DEVICE_UNPLUG_ERROR does?
> > > 
> > > 
> > > It informs that the hotunplug operation exceed the timeout that QEMU
> > > internals considers adequate, but QEMU can't assert that it was caused
> > > by an error or an unexpected delay. The end result is that the device
> > > is not going to be deleted from QMP, so DEVICE_NOT_DELETED.
> > 
> > Is it, though?  I mean, it is with this implementation for papr:
> > because we clear the unplug_requested flag, even if the guest later
> > tries to complete the unplug, it will fail.
> > 
> > But if I understand what Markus was saying correctly, that might not
> > be possible for all hotplug systems.  I believe Markus was suggesting
> > that DEVICE_NOT_DELETED could just mean that we haven't deleted the
> > device yet, but it could still happen later.
> > 
> > And in that case, I'm not yet sold on the value of a message that
> > essentially just means "Ayup, still dunno what's happening, sorry".
> > 
> > > Perhaps we should just be straightforward and create a 
> > > DEVICE_UNPLUG_TIMEOUT
> > > event.
> > 
> > Hm... what if we added a "reason" field to UNPLUG_ERROR.  That could
> > be "guest rejected hotplug", or something more specific, in the rare
> > case that the guest has a way of signalling something more specific,
> > or "timeout" - but the later *only* to be sent in cases where on the
> > timeout we're able to block any later completion of the unplug (as we
> > can on papr).
> 
> I believe that's already covered by the existing API:
> 
> 
> +# @DEVICE_UNPLUG_ERROR:
> +#
> +# Emitted when a device hot unplug error occurs.
> +#
> +# @device: device name
> +#
> +# @msg: Informative message

Oh, sorry, I missed that

> The 'informative message' would be the reason the event occurred. In patch
> 4/4, for the memory hotunplug refused by the guest, it is being set as:
> 
>  qapi_error = g_strdup_printf("Memory hotunplug rejected by the guest "
>   "for device %s", dev->id);
>  qapi_event_send_device_unplug_error(dev->id, qapi_error);
> 
> 
> 
> We could use the same DEVICE_UNPLUG_ERROR event in the CPU hotunplug timeout
> case (currently on patch 2/4) by just changing 'msg', e.g.:
> 
> 
>  qapi_error = g_strdup_printf("CPU hotunplug timeout for device %s", 
> dev->id);
>  qapi_event_send_device_unplug_error(dev->id, qapi_error);

I think that makes sense for the cases on papr.  Less sure about the
cases Markus has mentioned.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [PATCH v3 2/3] spapr: nvdimm: Implement H_SCM_FLUSH hcall

[David Gibson]

On Wed, Mar 24, 2021 at 09:34:06AM +0530, Aneesh Kumar K.V wrote:
> On 3/24/21 8:37 AM, David Gibson wrote:
> > On Tue, Mar 23, 2021 at 09:47:38AM -0400, Shivaprasad G Bhat wrote:
> > > The patch adds support for the SCM flush hcall for the nvdimm devices.
> > > To be available for exploitation by guest through the next patch.
> > > 
> > > The hcall expects the semantics such that the flush to return
> > > with H_BUSY when the operation is expected to take longer time along
> > > with a continue_token. The hcall to be called again providing the
> > > continue_token to get the status. So, all fresh requsts are put into
> > > a 'pending' list and flush worker is submitted to the thread pool.
> > > The thread pool completion callbacks move the requests to 'completed'
> > > list, which are cleaned up after reporting to guest in subsequent
> > > hcalls to get the status.
> > > 
> > > The semantics makes it necessary to preserve the continue_tokens
> > > and their return status even across migrations. So, the pre_save
> > > handler for the device waits for the flush worker to complete and
> > > collects all the hcall states from 'completed' list. The necessary
> > > nvdimm flush specific vmstate structures are added to the spapr
> > > machine vmstate.
> > > 
> > > Signed-off-by: Shivaprasad G Bhat 
> > 
> > An overal question: surely the same issue must arise on x86 with
> > file-backed NVDIMMs.  How do they handle this case?
> 
> On x86 we have different ways nvdimm can be discovered. ACPI NFIT, e820 map
> and virtio_pmem. Among these virio_pmem always operated with synchronous dax
> disabled and both ACPI and e820 doesn't have the ability to differentiate
> support for synchronous dax.

Ok.  And for the virtio-pmem case, how are the extra flushes actually
done on x86?

> With that I would expect users to use virtio_pmem when using using file
> backed NVDIMMS

So... should we prevent advertising an NVDIMM through ACPI or e820 if
it doesn't have sync-dax enabled?

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [PATCH v5 00/10] KVM: Dirty ring support (QEMU part)

[Keqian Zhu]

Peter,

On 2021/3/24 23:09, Peter Xu wrote:
> On Wed, Mar 24, 2021 at 10:56:22AM +0800, Keqian Zhu wrote:
>> Hi Peter,
>>
>> On 2021/3/23 22:34, Peter Xu wrote:
>>> Keqian,
>>>
>>> On Tue, Mar 23, 2021 at 02:40:43PM +0800, Keqian Zhu wrote:
>> The second question is that you observed longer migration time 
>> (55s->73s) when guest
>> has 24G ram and dirty rate is 800M/s. I am not clear about the reason. 
>> As with dirty
>> ring enabled, Qemu can get dirty info faster which means it handles 
>> dirty page more
>> quick, and guest can be throttled which means dirty page is generated 
>> slower. What's
>> the rationale for the longer migration time?
>
> Because dirty ring is more sensitive to dirty rate, while dirty bitmap is 
> more
 Emm... Sorry that I'm very clear about this... I think that higher dirty 
 rate doesn't cause
 slower dirty_log_sync compared to that of legacy bitmap mode. Besides, 
 higher dirty rate
 means we may have more full-exit, which can properly limit the dirty rate. 
 So it seems that
 dirty ring "prefers" higher dirty rate.
>>>
>>> When I measured the 800MB/s it's in the guest, after throttling.
>>>
>>> Imagine another example: a VM has 1G memory keep dirtying with 10GB/s.  
>>> Dirty
>>> logging will need to collect even less for each iteration because memory 
>>> size
>>> shrinked, collect even less frequent due to the high dirty rate, however 
>>> dirty
>>> ring will use 100% cpu power to collect dirty pages because the ring keeps 
>>> full.
>> Looks good.
>>
>> We have many places to collect dirty pages: the background reaper, vCPU exit 
>> handler,
>> and the migration thread. I think migration time is closely related to the 
>> migration thread.
>>
>> The migration thread calls kvm_dirty_ring_flush().
>> 1. kvm_cpu_synchronize_kick_all() will wait vcpu handles full-exit.
>> 2. kvm_dirty_ring_reap() collects and resets dirty pages.
>> The above two operation will spend more time with higher dirty rate.
>>
>> But I suddenly realize that the key problem maybe not at this. Though we 
>> have separate
>> "reset" operation for dirty ring, actually it is performed right after we 
>> collect dirty
>> ring to kvmslot. So in dirty ring mode, it likes legacy bitmap mode without 
>> manual_dirty_clear.
>>
>> If we can "reset" dirty ring just before we really handle the dirty pages, 
>> we can have
>> shorter migration time. But the design of dirty ring doesn't allow this, 
>> because we must
>> perform reset to make free space...
> 
> This is a very good point.
> 
> Dirty ring should have been better in quite some ways already, but from that
> pov as you said it goes a bit backwards on reprotection of pages (not to
> mention currently we can't even reset the ring per-vcpu; that seems to be not
> fully matching the full locality that the rings have provided as well; but
> Paolo and I discussed with that issue, it's about TLB flush expensiveness, so
> we still need to think more of it..).
> 
> Ideally the ring could have been both per-vcpu but also bi-directional (then
> we'll have 2*N rings, N=vcpu number), so as to split the state transition into
> "dirty ring" and "reprotect ring", then that reprotect ring will be the clear
> dirty log.  That'll look more like virtio as used ring.  However we'll still
> need to think about the TLB flush issue too as Paolo used to mention, as
> that'll exist too with any per-vcpu flush model (each reprotect of page will
> need a tlb flush of all vcpus).
> 
> Or.. maybe we can make the flush ring a standalone one, so we need N dirty 
> ring
> and one global flush ring.
Yep, have separate "reprotect" ring(s) is a good idea.

> 
> Anyway.. Before that, I'd still think the next step should be how to integrate
> qemu to fully leverage current ring model, so as to be able to throttle in
> per-vcpu fashion.
> 
> The major issue (IMHO) with huge VM migration is:
> 
>   1. Convergence
>   2. Responsiveness
> 
> Here we'll have a chance to solve (1) by highly throttle the working vcpu
> threads, meanwhile still keep (2) by not throttle user interactive threads.
> I'm not sure whether this will really work as expected, but just show what I'm
> thinking about it.  These may not matter a lot yet with further improving ring
> reset mechanism, which definitely sounds even better, but seems orthogonal.
> 
> That's also why I think we should still merge this series first as a fundation
> for the rest.
I see.

> 
>>
>>>

> sensitive to memory footprint.  In above 24G mem + 800MB/s dirty rate
> condition, dirty bitmap seems to be more efficient, say, collecting dirty
> bitmap of 24G mem (24G/4K/8=0.75MB) for each migration cycle is fast 
> enough.
>
> Not to mention that current implementation of dirty ring in QEMU is not
> complete - we still have two more layers of dirty bitmap, so it's 
> actually a
> mixture of dirty bitmap and dirty ring.  This series is

Re: [PATCH 1/3] aspeed: Coding Style cleanups on do_hash_operation

[Klaus Heinrich Kiwi]

Hi Cedric,

On 3/24/2021 7:57 PM, Cédric Le Goater wrote:

On 3/24/21 11:38 PM, Klaus Heinrich Kiwi wrote:

Basically using camelCase for some variables...

I don't think CamelCase applies to variables, only types.


Thanks, I think I mis-interpreted your comment on this.

Will re-factor the patches with that in mind, but let's see
if there are other comments first.

 -Klaus

--
Klaus Heinrich Kiwi 

[ANNOUNCE] QEMU 6.0.0-rc0 is now available

[Michael Roth]

Hello,

On behalf of the QEMU Team, I'd like to announce the availability of the
first release candidate for the QEMU 6.0 release.  This release is meant
for testing purposes and should not be used in a production environment.

  http://download.qemu-project.org/qemu-6.0.0-rc0.tar.xz
  http://download.qemu-project.org/qemu-6.0.0-rc0.tar.xz.sig

You can help improve the quality of the QEMU 6.0 release by testing this
release and reporting bugs on Launchpad:

  https://bugs.launchpad.net/qemu/

The release plan, as well a documented known issues for release
candidates, are available at:

  http://wiki.qemu.org/Planning/6.0

Please add entries to the ChangeLog for the 6.0 release below:

  http://wiki.qemu.org/ChangeLog/6.0

Thank you to everyone involved!

Re: [PATCH v2 03/10] Python: add utility function for retrieving port redirection

[Cleber Rosa]

On Wed, Mar 24, 2021 at 05:35:07PM -0300, Willian Rampazzo wrote:
> On Tue, Mar 23, 2021 at 7:16 PM Cleber Rosa  wrote:
> >
> > Slightly different versions for the same utility code are currently
> > present on different locations.  This unifies them all, giving
> > preference to the version from virtiofs_submounts.py, because of the
> > last tweaks added to it.
> >
> > While at it, this adds a "qemu.utils" module to host the utility
> > function and a test.
> >
> > Signed-off-by: Cleber Rosa 
> > Reviewed-by: Wainer dos Santos Moschetta 
> > ---
> >  python/qemu/utils.py | 35 
> >  tests/acceptance/info_usernet.py | 29 
> >  tests/acceptance/linux_ssh_mips_malta.py | 16 +--
> >  tests/acceptance/virtiofs_submounts.py   | 21 --
> >  tests/vm/basevm.py   |  7 ++---
> >  5 files changed, 78 insertions(+), 30 deletions(-)
> >  create mode 100644 python/qemu/utils.py
> >  create mode 100644 tests/acceptance/info_usernet.py
> >
> > diff --git a/python/qemu/utils.py b/python/qemu/utils.py
> > new file mode 100644
> > index 00..89a246ab30
> > --- /dev/null
> > +++ b/python/qemu/utils.py
> > @@ -0,0 +1,35 @@
> > +"""
> > +QEMU utility library
> > +
> > +This offers miscellaneous utility functions, which may not be easily
> > +distinguishable or numerous to be in their own module.
> > +"""
> > +
> > +# Copyright (C) 2021 Red Hat Inc.
> > +#
> > +# Authors:
> > +#  Cleber Rosa 
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.  See
> > +# the COPYING file in the top-level directory.
> > +#
> > +
> > +import re
> > +from typing import Optional
> > +
> > +
> > +def get_info_usernet_hostfwd_port(info_usernet_output: str) -> 
> > Optional[int]:
> > +"""
> > +Returns the port given to the hostfwd parameter via info usernet
> > +
> > +:param info_usernet_output: output generated by hmp command "info 
> > usernet"
> > +:param info_usernet_output: str
> > +:return: the port number allocated by the hostfwd option
> > +:rtype: int
> > +"""
> > +for line in info_usernet_output.split('\r\n'):
> > +regex = r'TCP.HOST_FORWARD.*127\.0\.0\.1\s+(\d+)\s+10\.'
> > +match = re.search(regex, line)
> > +if match is not None:
> > +return int(match[1])
> > +return None
> > diff --git a/tests/acceptance/info_usernet.py 
> > b/tests/acceptance/info_usernet.py
> > new file mode 100644
> > index 00..9c1fd903a0
> > --- /dev/null
> > +++ b/tests/acceptance/info_usernet.py
> > @@ -0,0 +1,29 @@
> > +# Test for the hmp command "info usernet"
> > +#
> > +# Copyright (c) 2021 Red Hat, Inc.
> > +#
> > +# Author:
> > +#  Cleber Rosa 
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2 or
> > +# later.  See the COPYING file in the top-level directory.
> > +
> > +from avocado_qemu import Test
> > +
> > +from qemu.utils import get_info_usernet_hostfwd_port
> > +
> > +
> > +class InfoUsernet(Test):
> > +
> > +def test_hostfwd(self):
> > +self.vm.add_args('-netdev', 
> > 'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
> > +self.vm.launch()
> > +res = self.vm.command('human-monitor-command',
> > +  command_line='info usernet')
> 
> This pattern is repeated every time a test needs to call
> get_info_usernet_hostfwd_port. Do you see any downside on moving this
> line inside the function and passing self.vm as an argument? This
> would abstract the need to run info usernet before calling the
> function.
> 

My original plan was to follow this up with a utility in QEMUMachine
itself (because that is the vm).  It can still be done, but:

 * I don't expect *tests* to be calling this often, rather a base
   class for tests;

 * I'm avoiding changing QEMUMachine too much, given it's a more
   generic class and used by other tests (iotests, vm, etc).

Hopefully when we have a better laid out structure for adding tests to
QEMUMachine itself, it'll be confortable to change it more
drastically.

> > +port = get_info_usernet_hostfwd_port(res)
> > +self.assertIsNotNone(port,
> > + ('"info usernet" output content does not seem 
> > to '
> > +  'contain the redirected port'))
> > +self.assertGreater(port, 0,
> > +   ('Found a redirected port that is not greater 
> > than'
> > +' zero'))
> > diff --git a/tests/acceptance/linux_ssh_mips_malta.py 
> > b/tests/acceptance/linux_ssh_mips_malta.py
> > index 6dbd02d49d..052008f02d 100644
> > --- a/tests/acceptance/linux_ssh_mips_malta.py
> > +++ b/tests/acceptance/linux_ssh_mips_malta.py
> > @@ -18,6 +18,8 @@
> >  from avocado.utils import archive
> >  from avocado.utils import ssh
> >
> > +from qemu.utils import get_info_usernet_hostfwd_port
> > +
> >
> >  class LinuxSSH(Test):
> >
> > @@ -70,18 +72,14 

Re: [PATCH v2 05/10] Acceptance Tests: add port redirection for ssh by default

[Cleber Rosa]

On Wed, Mar 24, 2021 at 11:36:53AM +0100, Auger Eric wrote:
> Hi Cleber,
> On 3/23/21 11:15 PM, Cleber Rosa wrote:
> > For users of the LinuxTest class, let's set up the VM with the port
> > redirection for SSH, instead of requiring each test to set the same
> > arguments.
> > 
> > Signed-off-by: Cleber Rosa 
> > ---
> >  tests/acceptance/avocado_qemu/__init__.py | 4 +++-
> >  tests/acceptance/virtiofs_submounts.py| 4 
> >  2 files changed, 3 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> > b/tests/acceptance/avocado_qemu/__init__.py
> > index 67f75f66e5..e75b002c70 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -309,10 +309,12 @@ class LinuxTest(Test, LinuxSSHMixIn):
> >  timeout = 900
> >  chksum = None
> >  
> > -def setUp(self, ssh_pubkey=None):
> > +def setUp(self, ssh_pubkey=None, network_device_type='virtio-net'):
> I would be interested in testing with HW bridging too, when a bridge is
> available. Do you think we could have the netdev configurable too?
> This would be helpful to test vhost for instance.
>

Right, I knew from the start that the user mode network would only
go so far.  TBH, I think it went too further than I expected.  But,
requiring, or supporting, other network modes can add quite a bit
of complexity, depending on how much you want the framework to do.

Anyway, this is a valid point/request.  For the lack of a better place,
and given that this may be a larger effort, I'm tracking it here:

  https://gitlab.com/cleber.gnu/qemu/-/issues/3

> With respect the network device type, I am currently working on SMMU
> test and I need to call the parent setUp-) with the following args now:
> 
> super(IOMMU, self).setUp(pubkey,
> 'virtio-net-pci,iommu_platform=on,disable-modern=off,disable-legacy=on')
> 
> It works but I am not sure you had such kind of scenario in mind?
>

I see where you're coming from, and I share the slight feeling of abusing
the feature... but I think it's OK at this point.  I mean, I believe it's
better to focus on say, the HW bridging support as this at least works.

> Thanks
> 
> Eric
>

Cheers,
- Cleber.


signature.asc
Description: PGP signature

Re: [PATCH v2 05/10] Acceptance Tests: add port redirection for ssh by default

[Cleber Rosa]

On Wed, Mar 24, 2021 at 10:10:50AM +0100, Auger Eric wrote:
> Hi Cleber,
> 
> On 3/23/21 11:15 PM, Cleber Rosa wrote:
> > For users of the LinuxTest class, let's set up the VM with the port
> > redirection for SSH, instead of requiring each test to set the same
> also sets the network device to virtio-net. This may be worth mentioning
> here in the commit msg.

Absolutely, I've added that note.

> > arguments.
> > 
> > Signed-off-by: Cleber Rosa 
> Reviewed-by: Eric Auger 
> 
> Thanks
> 
> Eric
>

Thank you!
- Cleber


signature.asc
Description: PGP signature

Re: gitlab-ci: Only build /staging branch?

[Cleber Rosa]

On Wed, Mar 24, 2021 at 07:01:12PM +0100, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> Peter's current workflow is push to /staging and if his
> testing succeeds, he pushes the same commit as /master.
> 
> IMO there is no point in building /master branch, as it
> has already been built earlier as /staging.
>

Hi,

I agree, I currently see no point in running the entire pipeline
again.

But we need to watch out for the job conditional rules. For instance,
both of these pipelines are about the exact same commit:

   https://gitlab.com/qemu-project/qemu/-/pipelines/275925246
   https://gitlab.com/qemu-project/qemu/-/pipelines/275925289

But one has 113, while the other has 119 jobs.  Also, we may want
to check if tags also trigger new (unnecessary) pipelines.

> Regards,
> 
> Phil.
> 

Best,
- Cleber


signature.asc
Description: PGP signature

Re: [PATCH v2 05/10] Acceptance Tests: add port redirection for ssh by default

[Cleber Rosa]

On Wed, Mar 24, 2021 at 12:30:18PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Wed, Mar 24, 2021 at 2:23 AM Cleber Rosa  wrote:
> 
> > For users of the LinuxTest class, let's set up the VM with the port
> > redirection for SSH, instead of requiring each test to set the same
> > arguments.
> >
> > Signed-off-by: Cleber Rosa 
> > ---
> >  tests/acceptance/avocado_qemu/__init__.py | 4 +++-
> >  tests/acceptance/virtiofs_submounts.py| 4 
> >  2 files changed, 3 insertions(+), 5 deletions(-)
> >
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py
> > b/tests/acceptance/avocado_qemu/__init__.py
> > index 67f75f66e5..e75b002c70 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -309,10 +309,12 @@ class LinuxTest(Test, LinuxSSHMixIn):
> >  timeout = 900
> >  chksum = None
> >
> > -def setUp(self, ssh_pubkey=None):
> > +def setUp(self, ssh_pubkey=None, network_device_type='virtio-net'):
> >  super(LinuxTest, self).setUp()
> >  self.vm.add_args('-smp', '2')
> >  self.vm.add_args('-m', '1024')
> > +self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0
> > -:22',
> > + '-device', '%s,netdev=vnet' %
> > network_device_type)
> >  self.set_up_boot()
> >  if ssh_pubkey is None:
> >  ssh_pubkey, self.ssh_key = self.set_up_existing_ssh_keys()
> > diff --git a/tests/acceptance/virtiofs_submounts.py
> > b/tests/acceptance/virtiofs_submounts.py
> > index bed8ce44df..e10a935ac4 100644
> > --- a/tests/acceptance/virtiofs_submounts.py
> > +++ b/tests/acceptance/virtiofs_submounts.py
> > @@ -207,10 +207,6 @@ def setUp(self):
> >  self.vm.add_args('-kernel', vmlinuz,
> >   '-append', 'console=ttyS0 root=/dev/sda1')
> >
> > -# Allow us to connect to SSH
> > -self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0
> > -:22',
> > - '-device', 'virtio-net,netdev=vnet')
> > -
> >  self.require_accelerator("kvm")
> >  self.vm.add_args('-accel', 'kvm')
> >
> > --
> > 2.25.4
> >
> >
> Looks fine, I suppose it could eventually be in LinuxSSHMixIn too for other
> users.
>

That's a good point, should be possible.  I'll look into that.

> Reviewed-by: Marc-André Lureau 
> 
> 
> -- 
> Marc-André Lureau

Thanks,
- Cleber.


signature.asc
Description: PGP signature

Re: [PATCH v2 04/10] Acceptance Tests: move useful ssh methods to base class

[Cleber Rosa]

On Wed, Mar 24, 2021 at 10:07:31AM +0100, Auger Eric wrote:
> Hi Cleber,
> 
> On 3/23/21 11:15 PM, Cleber Rosa wrote:
> > Both the virtiofs submounts and the linux ssh mips malta tests
> > contains useful methods related to ssh that deserve to be made
> > available to other tests.  Let's move them to the base LinuxTest
> nit: strictly speaking they are moved to another class which is
> inherited by LinuxTest, right?
> > class.
> > 
> > The method that helps with setting up an ssh connection will now
> > support both key and password based authentication, defaulting to key
> > based.
> > 
> > Signed-off-by: Cleber Rosa 
> > Reviewed-by: Wainer dos Santos Moschetta 
> > Reviewed-by: Willian Rampazzo 
> > ---
> >  tests/acceptance/avocado_qemu/__init__.py | 48 ++-
> >  tests/acceptance/linux_ssh_mips_malta.py  | 38 ++
> >  tests/acceptance/virtiofs_submounts.py| 37 -
> >  3 files changed, 50 insertions(+), 73 deletions(-)
> > 
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> > b/tests/acceptance/avocado_qemu/__init__.py
> > index 83b1741ec8..67f75f66e5 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -20,6 +20,7 @@
> >  from avocado.utils import cloudinit
> >  from avocado.utils import datadrainer
> >  from avocado.utils import network
> > +from avocado.utils import ssh
> >  from avocado.utils import vmimage
> >  from avocado.utils.path import find_command
> >  
> > @@ -43,6 +44,8 @@
> >  from qemu.accel import kvm_available
> >  from qemu.accel import tcg_available
> >  from qemu.machine import QEMUMachine
> > +from qemu.utils import get_info_usernet_hostfwd_port
> > +
> >  
> >  def is_readable_executable_file(path):
> >  return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
> > @@ -253,7 +256,50 @@ def fetch_asset(self, name,
> >  cancel_on_missing=cancel_on_missing)
> >  
> >  
> > -class LinuxTest(Test):
> > +class LinuxSSHMixIn:
> > +"""Contains utility methods for interacting with a guest via SSH."""
> > +
> > +def ssh_connect(self, username, credential, credential_is_key=True):
> > +self.ssh_logger = logging.getLogger('ssh')
> > +res = self.vm.command('human-monitor-command',
> > +  command_line='info usernet')
> > +port = get_info_usernet_hostfwd_port(res)
> > +self.assertIsNotNone(port)
> > +self.assertGreater(port, 0)
> > +self.log.debug('sshd listening on port: %d', port)
> > +if credential_is_key:
> > +self.ssh_session = ssh.Session('127.0.0.1', port=port,
> > +   user=username, key=credential)
> > +else:
> > +self.ssh_session = ssh.Session('127.0.0.1', port=port,
> > +   user=username, 
> > password=credential)
> > +for i in range(10):
> > +try:
> > +self.ssh_session.connect()
> > +return
> > +except:
> > +time.sleep(4)
> > +pass
> > +self.fail('ssh connection timeout')
> > +
> > +def ssh_command(self, command):
> > +self.ssh_logger.info(command)
> > +result = self.ssh_session.cmd(command)
> > +stdout_lines = [line.rstrip() for line
> > +in result.stdout_text.splitlines()]
> > +for line in stdout_lines:
> > +self.ssh_logger.info(line)
> > +stderr_lines = [line.rstrip() for line
> > +in result.stderr_text.splitlines()]
> > +for line in stderr_lines:
> > +self.ssh_logger.warning(line)
> > +
> > +self.assertEqual(result.exit_status, 0,
> > + f'Guest command failed: {command}')
> > +return stdout_lines, stderr_lines
> > +
> > +
> > +class LinuxTest(Test, LinuxSSHMixIn):
> >  """Facilitates having a cloud-image Linux based available.
> >  
> >  For tests that indend to interact with guests, this is a better choice
> > diff --git a/tests/acceptance/linux_ssh_mips_malta.py 
> > b/tests/acceptance/linux_ssh_mips_malta.py
> > index 052008f02d..3f590a081f 100644
> > --- a/tests/acceptance/linux_ssh_mips_malta.py
> > +++ b/tests/acceptance/linux_ssh_mips_malta.py
> > @@ -12,7 +12,7 @@
> >  import time
> >  
> >  from avocado import skipUnless
> > -from avocado_qemu import Test
> > +from avocado_qemu import Test, LinuxSSHMixIn
> >  from avocado_qemu import wait_for_console_pattern
> >  from avocado.utils import process
> >  from avocado.utils import archive
> > @@ -21,7 +21,7 @@
> >  from qemu.utils import get_info_usernet_hostfwd_port
> Can't you remove this now?
> >  
> >  

Yes, good catch!

> > -class LinuxSSH(Test):
> > +class LinuxSSH(Test, LinuxSSHMixIn):
> out of curiosity why can't it be migrated to a LinuxTest?

LinuxTest (currently) assumes that it'l

Re: [PATCH v2 07/10] Acceptance Tests: set up SSH connection by default after boot for LinuxTest

[Cleber Rosa]

On Wed, Mar 24, 2021 at 10:22:47AM +0100, Auger Eric wrote:
> Hi Cleber,
> 
> On 3/23/21 11:15 PM, Cleber Rosa wrote:
> > The LinuxTest specifically targets users that need to interact with Linux
> > guests.  So, it makes sense to give a connection by default, and avoid
> > requiring it as boiler-plate code.
> > 
> > Signed-off-by: Cleber Rosa 
> > ---
> >  tests/acceptance/avocado_qemu/__init__.py | 5 -
> >  tests/acceptance/virtiofs_submounts.py| 1 -
> >  2 files changed, 4 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> > b/tests/acceptance/avocado_qemu/__init__.py
> > index 535f63a48d..4960142bcc 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -390,7 +390,7 @@ def set_up_cloudinit(self, ssh_pubkey=None):
> >  cloudinit_iso = self.prepare_cloudinit(ssh_pubkey)
> >  self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
> >  
> > -def launch_and_wait(self):
> > +def launch_and_wait(self, set_up_ssh_connection=True):
> >  self.vm.set_console()
> >  self.vm.launch()
> >  console_drainer = 
> > datadrainer.LineLogger(self.vm.console_socket.fileno(),
> > @@ -398,3 +398,6 @@ def launch_and_wait(self):
> >  console_drainer.start()
> >  self.log.info('VM launched, waiting for boot confirmation from 
> > guest')
> >  cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), 
> > self.name)
> > +if set_up_ssh_connection:
> > +self.log.info('Setting up the SSH connection')
> > +self.ssh_connect(self.username, self.ssh_key)
> > diff --git a/tests/acceptance/virtiofs_submounts.py 
> > b/tests/acceptance/virtiofs_submounts.py
> > index e10a935ac4..e019d3b896 100644
> > --- a/tests/acceptance/virtiofs_submounts.py
> > +++ b/tests/acceptance/virtiofs_submounts.py
> > @@ -136,7 +136,6 @@ def set_up_virtiofs(self):
> >  
> >  def launch_vm(self):
> >  self.launch_and_wait()
> > -self.ssh_connect('root', self.ssh_key)
> >  
> >  def set_up_nested_mounts(self):
> >  scratch_dir = os.path.join(self.shared_dir, 'scratch')
> > 
> what about launch_and_wait calls in boot_linux.py. Don't you want to
> force ssh connection off there?
>

Good point.  I guess one could argue that it doesn't hurt those tests,
and even that it "tests more".  But, I'd argue that less is more here
indeed.

I'll change those launch_and_wait() to include set_up_ssh_connection=False
for those tests.

> Thanks
> 
> Eric

Thanks a lot!
- Cleber.


signature.asc
Description: PGP signature

Re: [PATCH 1/3] aspeed: Coding Style cleanups on do_hash_operation

[Cédric Le Goater]

On 3/24/21 11:38 PM, Klaus Heinrich Kiwi wrote:
> Basically using camelCase for some variables...

I don't think CamelCase applies to variables, only types.

  
https://qemu.readthedocs.io/en/latest/devel/style.html#variable-naming-conventions

C.

 
> 
> Signed-off-by: Klaus Heinrich Kiwi 
> ---
>  hw/misc/aspeed_hace.c | 22 +++---
>  1 file changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/hw/misc/aspeed_hace.c b/hw/misc/aspeed_hace.c
> index 6e5b447a48..93313d2b80 100644
> --- a/hw/misc/aspeed_hace.c
> +++ b/hw/misc/aspeed_hace.c
> @@ -86,24 +86,24 @@ static int hash_algo_lookup(uint32_t mask)
>  
>  static int do_hash_operation(AspeedHACEState *s, int algo)
>  {
> -hwaddr src, len, dest;
> -uint8_t *digest_buf = NULL;
> -size_t digest_len = 0;
> -char *src_buf;
> +uint32_t src, len, dest;
> +uint8_t *digestBuf = NULL;
> +size_t digestLen = 0;
> +char *srcBuf;
>  int rc;
>  
>  src = s->regs[R_HASH_SRC];
>  len = s->regs[R_HASH_SRC_LEN];
>  dest = s->regs[R_HASH_DEST];
>  
> -src_buf = address_space_map(&s->dram_as, src, &len, false,
> -MEMTXATTRS_UNSPECIFIED);
> -if (!src_buf) {
> +srcBuf = address_space_map(&s->dram_as, src, (hwaddr *) &len,
> +   false, MEMTXATTRS_UNSPECIFIED);
> +if (!srcBuf) {
>  qemu_log_mask(LOG_GUEST_ERROR, "%s: failed to map dram\n", __func__);
>  return -EACCES;
>  }
>  
> -rc = qcrypto_hash_bytes(algo, src_buf, len, &digest_buf, &digest_len,
> +rc = qcrypto_hash_bytes(algo, srcBuf, len, &digestBuf, &digestLen,
>  &error_fatal);
>  if (rc < 0) {
>  qemu_log_mask(LOG_GUEST_ERROR, "%s: qcrypto failed\n", __func__);
> @@ -111,14 +111,14 @@ static int do_hash_operation(AspeedHACEState *s, int 
> algo)
>  }
>  
>  rc = address_space_write(&s->dram_as, dest, MEMTXATTRS_UNSPECIFIED,
> - digest_buf, digest_len);
> + digestBuf, digestLen);
>  if (rc) {
>  qemu_log_mask(LOG_GUEST_ERROR,
>"%s: address space write failed\n", __func__);
>  }
> -g_free(digest_buf);
> +g_free(digestBuf);
>  
> -address_space_unmap(&s->dram_as, src_buf, len, false, len);
> +address_space_unmap(&s->dram_as, srcBuf, len, false, len);
>  
>  /*
>   * Set status bits to indicate completion. Testing shows hardware sets
> 

Re: [PATCH 0/6] hw/southbridge: QOM'ify vt82c686 as VT82C686B_SOUTHBRIDGE

[BALATON Zoltan]

On Wed, 24 Mar 2021, Philippe Mathieu-Daudé wrote:

The motivation behind this series is to remove the
isa_get_irq(NULL) call to simplify the ISA generic model.


Could you please wait with this until after my pegasos2 series is merged? 
Otherwise I'll have to rewrite that again for which I don't have much time 
and don't want to miss 6.1 too now that it's almost reviewed and accepted. 
This series is a cleanup that could be more fully done after the other 
machine using vt82c686.c is also there.


Regards,
BALATON Zoltan

[PATCH 3/3] tests: Aspeed HACE Scatter-Gather tests

[Klaus Heinrich Kiwi]

Expand current Aspeed HACE testsuite to also include Scatter-Gather of
sha256 and sha512 operations.

Signed-off-by: Klaus Heinrich Kiwi 
---
 tests/qtest/aspeed_hace-test.c | 164 ++---
 1 file changed, 153 insertions(+), 11 deletions(-)

diff --git a/tests/qtest/aspeed_hace-test.c b/tests/qtest/aspeed_hace-test.c
index 2b624b6b09..ae3f963449 100644
--- a/tests/qtest/aspeed_hace-test.c
+++ b/tests/qtest/aspeed_hace-test.c
@@ -35,6 +35,11 @@
 #define HACE_HASH_DATA_LEN   0x2c
 #define HACE_HASH_CMD0x30
 
+struct aspeed_sg_list {
+uint32_t len;
+uint32_t phy_addr;
+} __attribute__ ((__packed__));
+
 /*
  * Test vector is the ascii "abc"
  *
@@ -63,6 +68,33 @@ static const uint8_t test_result_md5[] = {
 0x90, 0x01, 0x50, 0x98, 0x3c, 0xd2, 0x4f, 0xb0, 0xd6, 0x96, 0x3f, 0x7d,
 0x28, 0xe1, 0x7f, 0x72};
 
+/*
+ * The Scatter-Gather Test vector is the ascii "abc" "def" "ghi", broken
+ * into blocks of 3 characters as shown
+ *
+ * Expected results were generated using command line utitiles:
+ *
+ *  echo -n -e 'abcdefghi' | dd of=/tmp/test
+ *  for hash in sha512sum sha256sum; do $hash /tmp/test; done
+ *
+ */
+static const uint8_t test_vector_sg1[] = {0x61, 0x62, 0x63};
+static const uint8_t test_vector_sg2[] = {0x64, 0x65, 0x66};
+static const uint8_t test_vector_sg3[] = {0x67, 0x68, 0x69};
+
+static const uint8_t test_result_sg_sha512[] = {
+0xf2, 0x2d, 0x51, 0xd2, 0x52, 0x92, 0xca, 0x1d, 0x0f, 0x68, 0xf6, 0x9a,
+0xed, 0xc7, 0x89, 0x70, 0x19, 0x30, 0x8c, 0xc9, 0xdb, 0x46, 0xef, 0xb7,
+0x5a, 0x03, 0xdd, 0x49, 0x4f, 0xc7, 0xf1, 0x26, 0xc0, 0x10, 0xe8, 0xad,
+0xe6, 0xa0, 0x0a, 0x0c, 0x1a, 0x5f, 0x1b, 0x75, 0xd8, 0x1e, 0x0e, 0xd5,
+0xa9, 0x3c, 0xe9, 0x8d, 0xc9, 0xb8, 0x33, 0xdb, 0x78, 0x39, 0x24, 0x7b,
+0x1d, 0x9c, 0x24, 0xfe};
+
+static const uint8_t test_result_sg_sha256[] = {
+0x19, 0xcc, 0x02, 0xf2, 0x6d, 0xf4, 0x3c, 0xc5, 0x71, 0xbc, 0x9e, 0xd7,
+0xb0, 0xc4, 0xd2, 0x92, 0x24, 0xa3, 0xec, 0x22, 0x95, 0x29, 0x22, 0x17,
+0x25, 0xef, 0x76, 0xd0, 0x21, 0xc8, 0x32, 0x6f};
+
 
 static void write_regs(QTestState *s, uint32_t base, uint32_t src,
uint32_t length, uint32_t out, uint32_t method)
@@ -167,28 +199,124 @@ static void test_sha512(const char *machine, const 
uint32_t base,
 test_result_sha512, sizeof(digest));
 }
 
+static void test_sha256_sg(const char *machine, const uint32_t base,
+const uint32_t src_addr)
+{
+QTestState *s = qtest_init(machine);
+
+const uint32_t src_addr_1 = src_addr + 0x100;
+const uint32_t src_addr_2 = src_addr + 0x200;
+const uint32_t src_addr_3 = src_addr + 0x300;
+const uint32_t digest_addr = src_addr + 0x400;
+uint8_t digest[32] = {0};
+struct aspeed_sg_list array[] = {
+{ sizeof(test_vector_sg1),  src_addr_1},
+{ sizeof(test_vector_sg2),  src_addr_2},
+{ sizeof(test_vector_sg3) | 1u << 31,   src_addr_3},
+};
+
+/* Check engine is idle, no busy or irq bits set */
+g_assert_cmphex(qtest_readl(s, base + HACE_STS), ==, 0);
+
+/* Write test vector into memory */
+qtest_memwrite(s, src_addr_1, test_vector_sg1, sizeof(test_vector_sg1));
+qtest_memwrite(s, src_addr_2, test_vector_sg2, sizeof(test_vector_sg2));
+qtest_memwrite(s, src_addr_3, test_vector_sg3, sizeof(test_vector_sg3));
+qtest_memwrite(s, src_addr, array, sizeof(array));
+
+write_regs(s, base, src_addr,
+   (sizeof(test_vector_sg1)
++ sizeof(test_vector_sg2)
++ sizeof(test_vector_sg3)),
+   digest_addr, HACE_ALGO_SHA256 | HACE_SG_EN);
+
+/* Check hash IRQ status is asserted */
+g_assert_cmphex(qtest_readl(s, base + HACE_STS), ==, 0x0200);
+
+/* Clear IRQ status and check status is deasserted */
+qtest_writel(s, base + HACE_STS, 0x0200);
+g_assert_cmphex(qtest_readl(s, base + HACE_STS), ==, 0);
+
+/* Read computed digest from memory */
+qtest_memread(s, digest_addr, digest, sizeof(digest));
+
+/* Check result of computation */
+g_assert_cmpmem(digest, sizeof(digest),
+test_result_sg_sha256, sizeof(digest));
+}
+
+static void test_sha512_sg(const char *machine, const uint32_t base,
+const uint32_t src_addr)
+{
+QTestState *s = qtest_init(machine);
+
+const uint32_t src_addr_1 = src_addr + 0x100;
+const uint32_t src_addr_2 = src_addr + 0x200;
+const uint32_t src_addr_3 = src_addr + 0x300;
+const uint32_t digest_addr = src_addr + 0x400;
+uint8_t digest[64] = {0};
+struct aspeed_sg_list array[] = {
+{ sizeof(test_vector_sg1),  src_addr_1},
+{ sizeof(test_vector_sg2),  src_addr_2},
+{ sizeof(test_vector_sg3) | 1u << 31,   src_addr_3},
+};
+
+/* Check engine is idle, no

[PATCH 2/3] aspeed: Add Scater-Gather support for HACE Hash

[Klaus Heinrich Kiwi]

Complement the Aspeed HACE support with Scatter-Gather hash support for
sha256 and sha512. Scatter-Gather is only supported on AST2600-series.

Signed-off-by: Klaus Heinrich Kiwi 
---
 hw/misc/aspeed_hace.c | 127 --
 include/hw/misc/aspeed_hace.h |   6 ++
 2 files changed, 127 insertions(+), 6 deletions(-)

diff --git a/hw/misc/aspeed_hace.c b/hw/misc/aspeed_hace.c
index 93313d2b80..8a37b1d961 100644
--- a/hw/misc/aspeed_hace.c
+++ b/hw/misc/aspeed_hace.c
@@ -57,6 +57,10 @@
 /* Other cmd bits */
 #define  HASH_IRQ_ENBIT(9)
 #define  HASH_SG_EN BIT(18)
+/* Scatter-gather data list */
+#define  SG_LIST_LAST   BIT(31)
+#define  SG_LIST_LEN_MASK   0x7fff
+#define  SG_LIST_ADDR_MASK  0x7ff8  /* 8-byte aligned */
 
 static const struct {
 uint32_t mask;
@@ -129,6 +133,117 @@ static int do_hash_operation(AspeedHACEState *s, int algo)
 return 0;
 }
 
+static int do_hash_sg_operation(AspeedHACEState *s, int algo)
+{
+uint32_t src, dest, reqSize;
+hwaddr len;
+const size_t reqLen = sizeof(struct aspeed_sg_list);
+struct iovec iov[ASPEED_HACE_MAX_SG];
+unsigned int i = 0;
+unsigned int isLast = 0;
+uint8_t *digestBuf = NULL;
+size_t digestLen = 0, size = 0;
+struct aspeed_sg_list *sgList;
+int rc;
+
+reqSize = s->regs[R_HASH_SRC_LEN];
+dest = s->regs[R_HASH_DEST];
+
+while (!isLast && i < ASPEED_HACE_MAX_SG) {
+src = s->regs[R_HASH_SRC] + (i * reqLen);
+len = reqLen;
+sgList = (struct aspeed_sg_list *) address_space_map(&s->dram_as,
+ src,
+ (hwaddr *) &len,
+   false,
+ MEMTXATTRS_UNSPECIFIED);
+if (!sgList) {
+qemu_log_mask(LOG_GUEST_ERROR,
+ "%s: failed to map dram for SG Array entry '%u' for address 
'0x%0x'\n",
+ __func__, i, src);
+rc = -EACCES;
+goto cleanup;
+}
+if (len != reqLen)
+qemu_log_mask(LOG_GUEST_ERROR,
+ "%s:  Warning: dram map for SG array entry '%u' requested size 
'%lu' != mapped size '%lu'\n",
+ __func__, i, reqLen, len);
+
+isLast = sgList->len & SG_LIST_LAST;
+
+iov[i].iov_len = (hwaddr) (sgList->len & SG_LIST_LEN_MASK);
+iov[i].iov_base = address_space_map(&s->dram_as,
+sgList->phy_addr & SG_LIST_ADDR_MASK,
+&iov[i].iov_len, false,
+MEMTXATTRS_UNSPECIFIED);
+if (!iov[i].iov_base) {
+qemu_log_mask(LOG_GUEST_ERROR,
+ "%s: failed to map dram for SG array entry '%u' for region 
'0x%x', len '%u'\n",
+ __func__, i, sgList->phy_addr & SG_LIST_ADDR_MASK,
+ sgList->len & SG_LIST_LEN_MASK);
+rc = -EACCES;
+goto cleanup;
+}
+if (iov[i].iov_len != (sgList->len & SG_LIST_LEN_MASK))
+qemu_log_mask(LOG_GUEST_ERROR,
+ "%s:  Warning: dram map for SG region entry %u requested size %u 
!= mapped size %lu\n",
+ __func__, i, (sgList->len & SG_LIST_LEN_MASK), iov[i].iov_len);
+
+
+address_space_unmap(&s->dram_as, (void *) sgList, len, false,
+len);
+size += iov[i].iov_len;
+i++;
+}
+
+if (!isLast) {
+qemu_log_mask(LOG_GUEST_ERROR,
+ "%s: Error: Exhausted maximum of '%u' SG array entries\n",
+ __func__, ASPEED_HACE_MAX_SG);
+rc = -ENOTSUP;
+goto cleanup;
+}
+
+if (size != reqSize)
+qemu_log_mask(LOG_GUEST_ERROR,
+ "%s: Warning: requested SG total size %u != actual size %lu\n",
+ __func__, reqSize, size);
+
+rc = qcrypto_hash_bytesv(algo, iov, i, &digestBuf, &digestLen,
+&error_fatal);
+if (rc < 0) {
+qemu_log_mask(LOG_GUEST_ERROR, "%s: qcrypto failed\n",
+  __func__);
+goto cleanup;
+}
+
+rc = address_space_write(&s->dram_as, dest, MEMTXATTRS_UNSPECIFIED,
+ digestBuf, digestLen);
+if (rc)
+qemu_log_mask(LOG_GUEST_ERROR,
+  "%s: address space write failed\n", __func__);
+g_free(digestBuf);
+
+cleanup:
+
+for (; i > 0; i--) {
+address_space_unmap(&s->dram_as, iov[i - 1].iov_base,
+iov[i - 1].iov_len, false,
+iov[i - 1].iov_len);
+}
+
+/*
+ * Set status bits to indicate completion. Testing shows hardware sets
+ * these irrespective of HASH_IRQ_EN.
+ */
+if (!rc) {
+s->regs[R_STATUS] |= HASH_IRQ;
+}
+
+return rc;
+}
+
+
 
 sta

[PATCH 1/3] aspeed: Coding Style cleanups on do_hash_operation

[Klaus Heinrich Kiwi]

Basically using camelCase for some variables...

Signed-off-by: Klaus Heinrich Kiwi 
---
 hw/misc/aspeed_hace.c | 22 +++---
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/hw/misc/aspeed_hace.c b/hw/misc/aspeed_hace.c
index 6e5b447a48..93313d2b80 100644
--- a/hw/misc/aspeed_hace.c
+++ b/hw/misc/aspeed_hace.c
@@ -86,24 +86,24 @@ static int hash_algo_lookup(uint32_t mask)
 
 static int do_hash_operation(AspeedHACEState *s, int algo)
 {
-hwaddr src, len, dest;
-uint8_t *digest_buf = NULL;
-size_t digest_len = 0;
-char *src_buf;
+uint32_t src, len, dest;
+uint8_t *digestBuf = NULL;
+size_t digestLen = 0;
+char *srcBuf;
 int rc;
 
 src = s->regs[R_HASH_SRC];
 len = s->regs[R_HASH_SRC_LEN];
 dest = s->regs[R_HASH_DEST];
 
-src_buf = address_space_map(&s->dram_as, src, &len, false,
-MEMTXATTRS_UNSPECIFIED);
-if (!src_buf) {
+srcBuf = address_space_map(&s->dram_as, src, (hwaddr *) &len,
+   false, MEMTXATTRS_UNSPECIFIED);
+if (!srcBuf) {
 qemu_log_mask(LOG_GUEST_ERROR, "%s: failed to map dram\n", __func__);
 return -EACCES;
 }
 
-rc = qcrypto_hash_bytes(algo, src_buf, len, &digest_buf, &digest_len,
+rc = qcrypto_hash_bytes(algo, srcBuf, len, &digestBuf, &digestLen,
 &error_fatal);
 if (rc < 0) {
 qemu_log_mask(LOG_GUEST_ERROR, "%s: qcrypto failed\n", __func__);
@@ -111,14 +111,14 @@ static int do_hash_operation(AspeedHACEState *s, int algo)
 }
 
 rc = address_space_write(&s->dram_as, dest, MEMTXATTRS_UNSPECIFIED,
- digest_buf, digest_len);
+ digestBuf, digestLen);
 if (rc) {
 qemu_log_mask(LOG_GUEST_ERROR,
   "%s: address space write failed\n", __func__);
 }
-g_free(digest_buf);
+g_free(digestBuf);
 
-address_space_unmap(&s->dram_as, src_buf, len, false, len);
+address_space_unmap(&s->dram_as, srcBuf, len, false, len);
 
 /*
  * Set status bits to indicate completion. Testing shows hardware sets
-- 
2.25.1

[PATCH 0/3] aspeed: HACE hash Scatter-Gather support

[Klaus Heinrich Kiwi]

This series adds support for scatter-gather sha256 and sha512 operations
on Aspeed's HACE (Hash And Crypto Engine) to the Aspeed model. These
operations are supported on AST2600 series of machines.

 -Klaus

Re: [RFC v11 34/55] target/arm: remove broad "else" statements when checking accels

[Richard Henderson]

On 3/23/21 9:46 AM, Claudio Fontana wrote:

@@ -296,7 +297,8 @@ void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
   */
  bitmap_andnot(tmp, kvm_supported, cpu->sve_vq_init, max_vq);
  bitmap_or(cpu->sve_vq_map, cpu->sve_vq_map, tmp, max_vq);
-} else {
+}
+if (tcg_enabled()) {


Better as else if, I think.

r~

Re: [RFC v11 32/55] target/arm: move kvm-const.h, kvm.c, kvm64.c, kvm_arm.h to kvm/

[Richard Henderson]

On 3/23/21 9:46 AM, Claudio Fontana wrote:

  target/arm/kvm-stub.c | 24 


Can you really get rid of this?
If so, I think this should be a separate step.



--- a/target/arm/kvm/meson.build
+++ b/target/arm/kvm/meson.build
@@ -1,3 +1,8 @@
  arm_ss.add(when: 'CONFIG_TCG', if_false: files(
'helper-stubs.c',
  ))
+
+arm_ss.add(when: 'CONFIG_KVM', if_true: files(
+  'kvm.c',
+  'kvm64.c',
+))
\ No newline at end of file


Watch the whitespace warnings.


r~

Re: [RFC v11 31/55] target/arm: remove kvm include file for PSCI and arm-powerctl

[Richard Henderson]

On 3/23/21 9:46 AM, Claudio Fontana wrote:

The QEMU PSCI implementation is not used for KVM,
we do not need the kvm constants header.

Signed-off-by: Claudio Fontana
---
  target/arm/arm-powerctl.h | 2 --
  target/arm/psci.c | 1 -
  2 files changed, 3 deletions(-)


Reviewed-by: Richard Henderson 


r~

Re: [RFC v11 30/55] target/arm: wrap call to aarch64_sve_change_el in tcg_enabled()

[Richard Henderson]

On 3/23/21 9:46 AM, Claudio Fontana wrote:

After this patch it is possible to build only kvm:

./configure --disable-tcg --enable-kvm

Signed-off-by: Claudio Fontana
---
  target/arm/cpu-sysemu.c | 12 +++-
  1 file changed, 7 insertions(+), 5 deletions(-)


Reviewed-by: Richard Henderson 


r~

Re: [RFC v11 29/55] target/arm: cpu: fix style

[Richard Henderson]

On 3/23/21 9:46 AM, Claudio Fontana wrote:

Signed-off-by: Claudio Fontana
Reviewed-by: Philippe Mathieu-Daudé
---
  target/arm/cpu-sysemu.c | 17 +++--
  1 file changed, 11 insertions(+), 6 deletions(-)


Obviously, this should be done before the code moves.


r~

Re: [RFC v11 28/55] target/arm: refactor exception and cpu code

[Richard Henderson]

On 3/23/21 9:46 AM, Claudio Fontana wrote:

move exception code out of tcg/
as we need part of it for KVM too.

put the exception code into separate cpu modules as appropriate,
including:

cpu-sysemu.c
tcg/tcg-cpu.c
tcg/sysemu/tcg-cpu.c

to avoid naming confusion with the existing cpu_tcg.c,
containg cpu models definitions for 32bit TCG-only cpus,
rename this file as tcg/tcg-cpu-models.c


Obviously all of this should not be done in one step.

Isn't tcg/tcg-* redundant?


r~