[PATCH] iotests/testrunner.py: refactor test_field_width

2021-12-10 Thread Vladimir Sementsov-Ogievskiy
A lot of Optional[] types doesn't make code beautiful.
test_field_width defaults to 8, but that is never used in the code.

More over, if we want some default behavior for single call of
test_run(), it should just print the whole test name, not limiting or
expanding its width, so 8 is bad default.

So, just drop the default as unused for now.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

This is a follow-up for "[PATCH 0/3] iotests: multiprocessing!!" and
based on Hanna's block-next branch.

Based-on: <2021120313.2780098-1-vsement...@virtuozzo.com>

 tests/qemu-iotests/testrunner.py | 21 ++---
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index 0feaa396d0..15788f919e 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -174,19 +174,17 @@ def __enter__(self) -> 'TestRunner':
 def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
 self._stack.close()
 
-def test_print_one_line(self, test: str, starttime: str,
+def test_print_one_line(self, test: str,
+test_field_width: int,
+starttime: str,
 endtime: Optional[str] = None, status: str = '...',
 lasttime: Optional[float] = None,
 thistime: Optional[float] = None,
 description: str = '',
-test_field_width: Optional[int] = None,
 end: str = '\n') -> None:
 """ Print short test info before/after test run """
 test = os.path.basename(test)
 
-if test_field_width is None:
-test_field_width = 8
-
 if self.makecheck and status != '...':
 if status and status != 'pass':
 status = f' [{status}]'
@@ -328,7 +326,7 @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
   casenotrun=casenotrun)
 
 def run_test(self, test: str,
- test_field_width: Optional[int] = None,
+ test_field_width: int,
  mp: bool = False) -> TestResult:
 """
 Run one test and print short status
@@ -347,20 +345,21 @@ def run_test(self, test: str,
 
 if not self.makecheck:
 self.test_print_one_line(test=test,
+ test_field_width=test_field_width,
  status = 'started' if mp else '...',
  starttime=start,
  lasttime=last_el,
- end = '\n' if mp else '\r',
- test_field_width=test_field_width)
+ end = '\n' if mp else '\r')
 
 res = self.do_run_test(test, mp)
 
 end = datetime.datetime.now().strftime('%H:%M:%S')
-self.test_print_one_line(test=test, status=res.status,
+self.test_print_one_line(test=test,
+ test_field_width=test_field_width,
+ status=res.status,
  starttime=start, endtime=end,
  lasttime=last_el, thistime=res.elapsed,
- description=res.description,
- test_field_width=test_field_width)
+ description=res.description)
 
 if res.casenotrun:
 print(res.casenotrun)
-- 
2.31.1




Re: [PATCH 2/3] iotests/testrunner.py: move updating last_elapsed to run_tests

2021-12-10 Thread Vladimir Sementsov-Ogievskiy

10.12.2021 17:47, Vladimir Sementsov-Ogievskiy wrote:

10.12.2021 17:25, Kevin Wolf wrote:

Am 06.12.2021 um 18:59 hat John Snow geschrieben:

On Fri, Dec 3, 2021 at 7:22 AM Vladimir Sementsov-Ogievskiy <
vsement...@virtuozzo.com> wrote:


We are going to use do_run_test() in multiprocessing environment, where
we'll not be able to change original runner object.

Happily, the only thing we change is that last_elapsed and it's simple
to do it in run_tests() instead. All other accesses to self in
do_runt_test() and in run_test() are read-only.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  tests/qemu-iotests/testrunner.py | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/testrunner.py
b/tests/qemu-iotests/testrunner.py
index fa842252d3..a9f2feb58c 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -287,7 +287,6 @@ def do_run_test(self, test: str) -> TestResult:
    diff=diff, casenotrun=casenotrun)
  else:
  f_bad.unlink()
-    self.last_elapsed.update(test, elapsed)
  return TestResult(status='pass', elapsed=elapsed,
    casenotrun=casenotrun)

@@ -353,6 +352,9 @@ def run_tests(self, tests: List[str]) -> bool:
  print('\n'.join(res.diff))
  elif res.status == 'not run':
  notrun.append(name)
+    elif res.status == 'pass':
+    assert res.elapsed is not None
+    self.last_elapsed.update(t, res.elapsed)

  sys.stdout.flush()
  if res.interrupted:
--
2.31.1



(I continue to be annoyed by the "None" problem in mypy, but I suppose it
really can't be helped. Nothing for you to change with this patch or
series. I just wish we didn't need so many assertions ...)


I'm inclined to say it's a None problem in our code, not in mypy.
Essentially it comes from the fact that we're abusing a string
(res.status) and None values to distinguish different types of results
that have a different set of valid attributes.

Of course, Python already provides a language feature to distinguish
different types of results that have a different set of attributes and
that wouldn't run into this problem: subclasses.



Agree, you are right. I'll look if it is simple to refactor.



No it's not simple)

Actually, it means making TestResult more smart, and moving (most of) logic of test result 
result parsing (checking different files, etc) to TestResult.. But we'll still want to update 
last_elapsed.. Adding a method "TestResult.update_runnner(runner)", which will be 
no-op in base TestResult and will be actually realized only in TestResult subclass that have 
.elapsed to call runner.update_last_elapsed()? And we'll have a hierarhy like TestResultBase 
-> TestResultWithElapsed -> (TestResultFail, TestResultPass).. At least, that's what I 
imagine, and I don't like it :)

--
Best regards,
Vladimir



Re: [PATCH] spec: Add NBD_OPT_EXTENDED_HEADERS

2021-12-10 Thread Vladimir Sementsov-Ogievskiy

04.12.2021 02:14, Eric Blake wrote:

Add a new negotiation feature where the client and server agree to use
larger packet headers on every packet sent during transmission phase.
This has two purposes: first, it makes it possible to perform
operations like trim, write zeroes, and block status on more than 2^32
bytes in a single command; this in turn requires that some structured
replies from the server also be extended to match.  The wording chosen
here is careful to permit a server to use either flavor in its reply
(that is, a request less than 32-bits can trigger an extended reply,
and conversely a request larger than 32-bits can trigger a compact
reply).



About this.. Isn't it too permissive?

I think that actually having to very similar ways to do the same thing is 
usually a bad design. I think we don't want someone implement the logic, which 
tries to send 32bit commands/replies for small requests and 64bit 
command/replies for larger ones? Moreover, you don't allow doing it for 
commands. So, for symmetry, it may be good to be strict with replies too: in 
64bit mode only 64bit replies.

Now we of course have to support old 32bit commands and new 64bit commands. 
But, may be, we'll want to deprecate 32bit commands at some moment? I'm not 
sure we can deprecate them in protocol, but we can deprecate them in Qemu at 
least. And several years later we'll drop old code, keeping only support for 
64bit commands. Less code paths, less similar structures, simpler code, I think 
it worth it.


--
Best regards,
Vladimir



Re: [PATCH] spec: Add NBD_OPT_EXTENDED_HEADERS

2021-12-10 Thread Vladimir Sementsov-Ogievskiy

07.12.2021 12:08, Vladimir Sementsov-Ogievskiy wrote:

07.12.2021 02:00, Eric Blake wrote:

On Mon, Dec 06, 2021 at 02:40:45PM +0300, Vladimir Sementsov-Ogievskiy wrote:


[..]




+S: 64 bits, padding (MUST be zero)


Hmm. Extra 8 bytes to be power-of-2. Does 32 bytes really perform better than 
24 bytes?


Consider:
struct header[100];

if sizeof(header[0]) is a power of 2 <= the cache line size (and the
compiler prefers to start arrays aligned to the cache line) then we
are guaranteed that all array members each reside in a single cache
line.  But if it is not a power of 2, some of the array members
straddle two cache lines.

Will there be code that wants to create an array of headers?  Perhaps
so, because that is a logical way (along with scatter/gather to
combine the header with variable-sized payloads) of tracking the
headers for multiple commands issued in parallel.

Do I have actual performance numbers?  No. But there's plenty of
google hits for why sizing structs to a power of 2 is a good idea.


I have a thought:

If client stores headers in separate, nothing prevents make this padding in 
RAM. So you can define header struct with padding. But what a reason to make 
the padding in the stream? You can have and array of good-aligned structures, 
but fill only part of header structure reading from the socket. Note, that you 
can read only one header in one read() call anyway, as you have to analyze, 
does it have payload or not.

So, if we want to improve performance by padding the structures in RAM, it's 
not a reason for padding them in the wire, keeping in mind that we can't read 
more then one structure at once.


--
Best regards,
Vladimir



Re: [PATCH v5 09/31] block: introduce assert_bdrv_graph_writable

2021-12-10 Thread Hanna Reitz

On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote:

We want to be sure that the functions that write the child and
parent list of a bs are under BQL and drain.

BQL prevents from concurrent writings from the GS API, while
drains protect from I/O.

TODO: drains are missing in some functions using this assert.
Therefore a proper assertion will fail. Because adding drains
requires additional discussions, they will be added in future
series.

Signed-off-by: Emanuele Giuseppe Esposito 
---
  include/block/block_int-global-state.h | 10 +-
  block.c|  4 
  block/io.c | 11 +++
  3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/include/block/block_int-global-state.h 
b/include/block/block_int-global-state.h
index a1b7d0579d..fa96e8b449 100644
--- a/include/block/block_int-global-state.h
+++ b/include/block/block_int-global-state.h
@@ -312,4 +312,12 @@ void bdrv_remove_aio_context_notifier(BlockDriverState *bs,
   */
  void bdrv_drain_all_end_quiesce(BlockDriverState *bs);
  
-#endif /* BLOCK_INT_GLOBAL_STATE*/


This looks like it should be squashed into patch 7, sorry I missed this 
in v4...


(Rest of this patch looks good to me, for the record – and while I’m at 
it, for patches I didn’t reply to so far, I planned to send an R-b 
later.  But then there’s things like patches 2/3 looking good to me, but 
it turned out in my review for patch 4 that bdrv_lock_medium() is used 
in an I/O path, so I can’t really send an R-b now anymore...)


Hanna


+/**
+ * Make sure that the function is running under both drain and BQL.
+ * The latter protects from concurrent writings
+ * from the GS API, while the former prevents concurrent reads
+ * from I/O.
+ */
+void assert_bdrv_graph_writable(BlockDriverState *bs);
+
+#endif /* BLOCK_INT_GLOBAL_STATE */
diff --git a/block.c b/block.c
index 198ec636ff..522a273140 100644
--- a/block.c
+++ b/block.c
@@ -1416,6 +1416,7 @@ static void bdrv_child_cb_attach(BdrvChild *child)
  {
  BlockDriverState *bs = child->opaque;
  
+assert_bdrv_graph_writable(bs);

  QLIST_INSERT_HEAD(>children, child, next);
  
  if (child->role & BDRV_CHILD_COW) {

@@ -1435,6 +1436,7 @@ static void bdrv_child_cb_detach(BdrvChild *child)
  
  bdrv_unapply_subtree_drain(child, bs);
  
+assert_bdrv_graph_writable(bs);

  QLIST_REMOVE(child, next);
  }
  
@@ -2818,6 +2820,7 @@ static void bdrv_replace_child_noperm(BdrvChild **childp,

  if (child->klass->detach) {
  child->klass->detach(child);
  }
+assert_bdrv_graph_writable(old_bs);
  QLIST_REMOVE(child, next_parent);
  }
  
@@ -2827,6 +2830,7 @@ static void bdrv_replace_child_noperm(BdrvChild **childp,

  }
  
  if (new_bs) {

+assert_bdrv_graph_writable(new_bs);
  QLIST_INSERT_HEAD(_bs->parents, child, next_parent);
  
  /*

diff --git a/block/io.c b/block/io.c
index cb095deeec..3be08cad29 100644
--- a/block/io.c
+++ b/block/io.c
@@ -734,6 +734,17 @@ void bdrv_drain_all(void)
  bdrv_drain_all_end();
  }
  
+void assert_bdrv_graph_writable(BlockDriverState *bs)

+{
+/*
+ * TODO: this function is incomplete. Because the users of this
+ * assert lack the necessary drains, check only for BQL.
+ * Once the necessary drains are added,
+ * assert also for qatomic_read(>quiesce_counter) > 0
+ */
+assert(qemu_in_main_thread());
+}
+
  /**
   * Remove an active request from the tracked requests list
   *





Re: [PATCH 3/3] iotests: check: multiprocessing support

2021-12-10 Thread Kevin Wolf
Am 10.12.2021 um 15:46 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 10.12.2021 17:36, Kevin Wolf wrote:
> > Am 03.12.2021 um 13:22 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > Add -j  parameter, to run tests in several jobs simultaneously.
> > > For realization - simply utilize multiprocessing.Pool class.
> > > 
> > > Notes:
> > > 
> > > 1. Of course, tests can't run simultaneously in same TEST_DIR. So,
> > > use subdirectories TEST_DIR/testname/ and SOCK_DIR/testname/
> > > instead of simply TEST_DIR and SOCK_DIR
> > > 
> > > 2. multiprocessing.Pool.starmap function doesn't support passing
> > > context managers, so we can't simply pass "self". Happily, we need
> > > self only for read-only access, and it just works if it is defined
> > > in global space. So, add a temporary link TestRunner.shared_self
> > > during run_tests().
> > > 
> > > Signed-off-by: Vladimir Sementsov-Ogievskiy 
> > 
> > Just wondering, is it worth even supporting the mp=false case or can we
> > simplify the code a bit by always going through multiprocessing and
> > using nice directory names even if only one process is spawned?
> > 
> > Maybe John's observation that directory names get longer might be a
> > reason not to do that by default. Any other reasons you're aware of?
> 
> I just wanted to keep the behavior without a new option unchanged, to
> not deal with possible CI failures on "make check": who know what
> multiprocessing brings together with performance.

So basically just err on the side of caution. Makes sense.

Kevin




Re: [PATCH 0/2] block-backend: Retain permissions after migration

2021-12-10 Thread Kevin Wolf
Am 25.11.2021 um 14:53 hat Hanna Reitz geschrieben:
> Hi,
> 
> Peng Liang has reported an issue regarding migration of raw images here:
> https://lists.nongnu.org/archive/html/qemu-block/2021-11/msg00673.html
> 
> It turns out that after migrating, all permissions are shared when they
> weren’t before.  The cause of the problem is that we deliberately delay
> restricting the shared permissions until migration is really done (until
> the runstate is no longer INMIGRATE) and first share all permissions;
> but this causes us to lose the original shared permission mask and
> overwrites it with BLK_PERM_ALL, so once we do try to restrict the
> shared permissions, we only again share them all.
> 
> Fix this by saving the set of shared permissions through the first
> blk_perm_set() call that shares all; and add a regression test.
> 
> 
> I don’t believe we have to fix this in 6.2, because I think this bug has
> existed for four years now.  (I.e. it isn’t critical, and it’s no
> regression.)

Feels a bit like a hack, but I guess as long as it works... :-)

Reviewed-by: Kevin Wolf 




Re: [PATCH v5 06/31] block/block-backend.c: assertions for block-backend

2021-12-10 Thread Hanna Reitz

On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote:

All the global state (GS) API functions will check that
qemu_in_main_thread() returns true. If not, it means
that the safety of BQL cannot be guaranteed, and
they need to be moved to I/O.

Signed-off-by: Emanuele Giuseppe Esposito 
---
  block/block-backend.c  | 83 ++
  softmmu/qdev-monitor.c |  2 +
  2 files changed, 85 insertions(+)


So given my thoughts on patch 5, I believe that blk_set_perm() and 
blk_get_perm() should get assertions here.


Hanna




Re: [PATCH 3/3] iotests: check: multiprocessing support

2021-12-10 Thread Vladimir Sementsov-Ogievskiy

10.12.2021 17:36, Kevin Wolf wrote:

Am 03.12.2021 um 13:22 hat Vladimir Sementsov-Ogievskiy geschrieben:

Add -j  parameter, to run tests in several jobs simultaneously.
For realization - simply utilize multiprocessing.Pool class.

Notes:

1. Of course, tests can't run simultaneously in same TEST_DIR. So,
use subdirectories TEST_DIR/testname/ and SOCK_DIR/testname/
instead of simply TEST_DIR and SOCK_DIR

2. multiprocessing.Pool.starmap function doesn't support passing
context managers, so we can't simply pass "self". Happily, we need
self only for read-only access, and it just works if it is defined
in global space. So, add a temporary link TestRunner.shared_self
during run_tests().

Signed-off-by: Vladimir Sementsov-Ogievskiy 


Just wondering, is it worth even supporting the mp=false case or can we
simplify the code a bit by always going through multiprocessing and
using nice directory names even if only one process is spawned?

Maybe John's observation that directory names get longer might be a
reason not to do that by default. Any other reasons you're aware of?



I just wanted to keep the behavior without a new option unchanged, to not deal with 
possible CI failures on "make check": who know what multiprocessing brings 
together with performance.

--
Best regards,
Vladimir



Re: [PATCH 2/3] iotests/testrunner.py: move updating last_elapsed to run_tests

2021-12-10 Thread Vladimir Sementsov-Ogievskiy

10.12.2021 17:25, Kevin Wolf wrote:

Am 06.12.2021 um 18:59 hat John Snow geschrieben:

On Fri, Dec 3, 2021 at 7:22 AM Vladimir Sementsov-Ogievskiy <
vsement...@virtuozzo.com> wrote:


We are going to use do_run_test() in multiprocessing environment, where
we'll not be able to change original runner object.

Happily, the only thing we change is that last_elapsed and it's simple
to do it in run_tests() instead. All other accesses to self in
do_runt_test() and in run_test() are read-only.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  tests/qemu-iotests/testrunner.py | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/testrunner.py
b/tests/qemu-iotests/testrunner.py
index fa842252d3..a9f2feb58c 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -287,7 +287,6 @@ def do_run_test(self, test: str) -> TestResult:
diff=diff, casenotrun=casenotrun)
  else:
  f_bad.unlink()
-self.last_elapsed.update(test, elapsed)
  return TestResult(status='pass', elapsed=elapsed,
casenotrun=casenotrun)

@@ -353,6 +352,9 @@ def run_tests(self, tests: List[str]) -> bool:
  print('\n'.join(res.diff))
  elif res.status == 'not run':
  notrun.append(name)
+elif res.status == 'pass':
+assert res.elapsed is not None
+self.last_elapsed.update(t, res.elapsed)

  sys.stdout.flush()
  if res.interrupted:
--
2.31.1



(I continue to be annoyed by the "None" problem in mypy, but I suppose it
really can't be helped. Nothing for you to change with this patch or
series. I just wish we didn't need so many assertions ...)


I'm inclined to say it's a None problem in our code, not in mypy.
Essentially it comes from the fact that we're abusing a string
(res.status) and None values to distinguish different types of results
that have a different set of valid attributes.

Of course, Python already provides a language feature to distinguish
different types of results that have a different set of attributes and
that wouldn't run into this problem: subclasses.



Agree, you are right. I'll look if it is simple to refactor.

--
Best regards,
Vladimir



Re: [PATCH 1/3] iotests/testrunner.py: add doc string for run_test()

2021-12-10 Thread Vladimir Sementsov-Ogievskiy

10.12.2021 17:12, Kevin Wolf wrote:

Am 03.12.2021 um 13:22 hat Vladimir Sementsov-Ogievskiy geschrieben:

We are going to modify these methods and will add more documentation in
further commit. As a preparation add basic documentation.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  tests/qemu-iotests/testrunner.py | 13 +
  1 file changed, 13 insertions(+)

diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index 0e29c2fddd..fa842252d3 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -220,6 +220,12 @@ def find_reference(self, test: str) -> str:
  return f'{test}.out'
  
  def do_run_test(self, test: str) -> TestResult:

+"""
+Run one test
+
+:param test: test file path
+"""
+
  f_test = Path(test)
  f_bad = Path(f_test.name + '.out.bad')
  f_notrun = Path(f_test.name + '.notrun')
@@ -287,6 +293,13 @@ def do_run_test(self, test: str) -> TestResult:
  
  def run_test(self, test: str,

   test_field_width: Optional[int] = None) -> TestResult:
+"""
+Run one test and print short status
+
+:param test: test file path
+:param test_field_width: width for first field of status format
+"""
+
  last_el = self.last_elapsed.get(test)
  start = datetime.datetime.now().strftime('%H:%M:%S')


test_field_width is Optional[int], so the documentation should specify
what happens when you pass None.

Seems it doesn't change the behaviour, but just picks a default value of
8. 'test_field_width: int = 8' might have been the more obvious solution
for this in the first place.

Kevin



Agree, I'll make a follow-up patch for it.

--
Best regards,
Vladimir



Re: [PATCH v5 05/31] block-backend: special comments for blk_set/get_perm due to fuse

2021-12-10 Thread Hanna Reitz

On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote:

Fuse logic can be classified as I/O, so there is no BQL held
during its execution. And yet, it uses blk_{get/set}_perm
functions, that are classified as BQL and clearly require
the BQL lock. Since there is no easy solution for this,
add a couple of TODOs and FIXME in the relevant sections of the
code.


Well, the problem goes deeper, because we still consider them GS 
functions.  So it’s fine for the permission function 
raw_handle_perm_lock() to call bdrv_get_flags(), which is a GS function, 
and then you still get:


qemu-storage-daemon: ../block.c:6195: bdrv_get_flags: Assertion 
`qemu_in_main_thread()' failed.


It looks like in this case making bdrv_get_flags() not a GS function 
would be feasible and might solve the problem, but I guess we should 
instead think about adding something like


if (!exp->growable && !qemu_in_main_thread()) {
    /* Changing permissions like below only works in the main thread */
    return -EPERM;
}

to fuse_do_truncate().

Ideally, to make up for this, we should probably take the RESIZE 
permission in fuse_export_create() for writable exports in iothreads.



Signed-off-by: Emanuele Giuseppe Esposito 
---
  block/block-backend.c | 10 ++
  block/export/fuse.c   | 16 ++--
  2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/block/block-backend.c b/block/block-backend.c
index 1f0bda578e..7a4b50a8f0 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -888,6 +888,11 @@ int blk_set_perm(BlockBackend *blk, uint64_t perm, 
uint64_t shared_perm,
   Error **errp)
  {
  int ret;
+/*
+ * FIXME: blk_{get/set}_perm should be always called under
+ * BQL, but it is not the case right now (see block/export/fuse.c)
+ */
+/* assert(qemu_in_main_thread()); */
  
  if (blk->root && !blk->disable_perm) {

  ret = bdrv_child_try_set_perm(blk->root, perm, shared_perm, errp);
@@ -904,6 +909,11 @@ int blk_set_perm(BlockBackend *blk, uint64_t perm, 
uint64_t shared_perm,
  
  void blk_get_perm(BlockBackend *blk, uint64_t *perm, uint64_t *shared_perm)

  {
+/*
+ * FIXME: blk_{get/set}_perm should be always called under
+ * BQL, but it is not the case right now (see block/export/fuse.c)
+ */
+/* assert(qemu_in_main_thread()); */
  *perm = blk->perm;
  *shared_perm = blk->shared_perm;
  }
diff --git a/block/export/fuse.c b/block/export/fuse.c
index 823c126d23..7ceb8d783b 100644
--- a/block/export/fuse.c
+++ b/block/export/fuse.c
@@ -89,7 +89,10 @@ static int fuse_export_create(BlockExport *blk_exp,
  /* For growable exports, take the RESIZE permission */
  if (args->growable) {
  uint64_t blk_perm, blk_shared_perm;
-
+/*
+ * FIXME: blk_{get/set}_perm should not be here, as permissions
+ * should be modified only under BQL and here we are not!
+ */


I believe we are, BlockExportDriver.create() is called by blk_exp_add(), 
which looks very much like it runs in the main thread (calling 
bdrv_lookup_bs(), bdrv_try_set_aio_context(), blk_new(), and whatnot).


Hanna


  blk_get_perm(exp->common.blk, _perm, _shared_perm);
  
  ret = blk_set_perm(exp->common.blk, blk_perm | BLK_PERM_RESIZE,

@@ -400,6 +403,11 @@ static int fuse_do_truncate(const FuseExport *exp, int64_t 
size,
  
  /* Growable exports have a permanent RESIZE permission */

  if (!exp->growable) {
+
+/*
+ * FIXME: blk_{get/set}_perm should not be here, as permissions
+ * should be modified only under BQL and here we are not!
+ */
  blk_get_perm(exp->common.blk, _perm, _shared_perm);
  
  ret = blk_set_perm(exp->common.blk, blk_perm | BLK_PERM_RESIZE,

@@ -413,7 +421,11 @@ static int fuse_do_truncate(const FuseExport *exp, int64_t 
size,
 truncate_flags, NULL);
  
  if (!exp->growable) {

-/* Must succeed, because we are only giving up the RESIZE permission */
+/*
+ * Must succeed, because we are only giving up the RESIZE permission.
+ * FIXME: blk_{get/set}_perm should not be here, as permissions
+ * should be modified only under BQL and here we are not!
+ */
  blk_set_perm(exp->common.blk, blk_perm, blk_shared_perm, 
_abort);
  }
  





Re: [PATCH 3/3] iotests: check: multiprocessing support

2021-12-10 Thread Kevin Wolf
Am 03.12.2021 um 13:22 hat Vladimir Sementsov-Ogievskiy geschrieben:
> Add -j  parameter, to run tests in several jobs simultaneously.
> For realization - simply utilize multiprocessing.Pool class.
> 
> Notes:
> 
> 1. Of course, tests can't run simultaneously in same TEST_DIR. So,
>use subdirectories TEST_DIR/testname/ and SOCK_DIR/testname/
>instead of simply TEST_DIR and SOCK_DIR
> 
> 2. multiprocessing.Pool.starmap function doesn't support passing
>context managers, so we can't simply pass "self". Happily, we need
>self only for read-only access, and it just works if it is defined
>in global space. So, add a temporary link TestRunner.shared_self
>during run_tests().
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 

Just wondering, is it worth even supporting the mp=false case or can we
simplify the code a bit by always going through multiprocessing and
using nice directory names even if only one process is spawned?

Maybe John's observation that directory names get longer might be a
reason not to do that by default. Any other reasons you're aware of?

Kevin




Re: [PATCH 2/3] iotests/testrunner.py: move updating last_elapsed to run_tests

2021-12-10 Thread Kevin Wolf
Am 06.12.2021 um 18:59 hat John Snow geschrieben:
> On Fri, Dec 3, 2021 at 7:22 AM Vladimir Sementsov-Ogievskiy <
> vsement...@virtuozzo.com> wrote:
> 
> > We are going to use do_run_test() in multiprocessing environment, where
> > we'll not be able to change original runner object.
> >
> > Happily, the only thing we change is that last_elapsed and it's simple
> > to do it in run_tests() instead. All other accesses to self in
> > do_runt_test() and in run_test() are read-only.
> >
> > Signed-off-by: Vladimir Sementsov-Ogievskiy 
> > ---
> >  tests/qemu-iotests/testrunner.py | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/qemu-iotests/testrunner.py
> > b/tests/qemu-iotests/testrunner.py
> > index fa842252d3..a9f2feb58c 100644
> > --- a/tests/qemu-iotests/testrunner.py
> > +++ b/tests/qemu-iotests/testrunner.py
> > @@ -287,7 +287,6 @@ def do_run_test(self, test: str) -> TestResult:
> >diff=diff, casenotrun=casenotrun)
> >  else:
> >  f_bad.unlink()
> > -self.last_elapsed.update(test, elapsed)
> >  return TestResult(status='pass', elapsed=elapsed,
> >casenotrun=casenotrun)
> >
> > @@ -353,6 +352,9 @@ def run_tests(self, tests: List[str]) -> bool:
> >  print('\n'.join(res.diff))
> >  elif res.status == 'not run':
> >  notrun.append(name)
> > +elif res.status == 'pass':
> > +assert res.elapsed is not None
> > +self.last_elapsed.update(t, res.elapsed)
> >
> >  sys.stdout.flush()
> >  if res.interrupted:
> > --
> > 2.31.1
> >
> >
> (I continue to be annoyed by the "None" problem in mypy, but I suppose it
> really can't be helped. Nothing for you to change with this patch or
> series. I just wish we didn't need so many assertions ...)

I'm inclined to say it's a None problem in our code, not in mypy.
Essentially it comes from the fact that we're abusing a string
(res.status) and None values to distinguish different types of results
that have a different set of valid attributes.

Of course, Python already provides a language feature to distinguish
different types of results that have a different set of attributes and
that wouldn't run into this problem: subclasses.

Kevin




Re: [PATCH v5 04/31] include/sysemu/block-backend: split header into I/O and global state (GS) API

2021-12-10 Thread Hanna Reitz

On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote:

Similarly to the previous patches, split block-backend.h
in block-backend-io.h and block-backend-global-state.h

In addition, remove "block/block.h" include as it seems
it is not necessary anymore, together with "qemu/iov.h"

block-backend-common.h contains the structures shared between
the two headers, and the functions that can't be categorized as
I/O or global state.

Assertions are added in the next patch.

Signed-off-by: Emanuele Giuseppe Esposito 
---
  include/sysemu/block-backend-common.h   |  84 ++
  include/sysemu/block-backend-global-state.h | 121 +
  include/sysemu/block-backend-io.h   | 139 ++
  include/sysemu/block-backend.h  | 269 +---
  block/block-backend.c   |   9 +-
  5 files changed, 353 insertions(+), 269 deletions(-)
  create mode 100644 include/sysemu/block-backend-common.h
  create mode 100644 include/sysemu/block-backend-global-state.h
  create mode 100644 include/sysemu/block-backend-io.h


[...]


diff --git a/include/sysemu/block-backend-global-state.h 
b/include/sysemu/block-backend-global-state.h
new file mode 100644
index 00..77009bf7a2
--- /dev/null
+++ b/include/sysemu/block-backend-global-state.h


[...]


+int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);


I don’t quite follow what makes blk_ioctl() a GS function.  (It’s 
possible that I forgot something about this from v4, though, and can’t 
find it anymore...)


I suppose it can be said that in the context of the block layer, ioctl 
functions are generally used outside of I/O threads as something that 
kind of affects the global data state, but... bdrv_co_ioctl() is in 
block-io.h, and internally we definitely do use ioctl in I/O threads 
(various uses in file-posix.c, e.g. to zero out areas).  I also don’t 
understand why blk_ioctl() is GS, and blk_aio_ioctl() is I/O.


And if you have a virtio-scsi device in an iothread, and then create a 
scsi-generic device on it, will bdrv_co_ioctl() not run in the I/O 
thread still...?  Hm, testing (on patch 6), it appears that not – I 
think that’s because it mostly uses blk_aio_ioctl(), apart from 
*_realize() and scsi_SG_IO_FROM_DEV() (whatever that is; and this makes 
me a bit cautious, too).  Still, I’m not quite sure how it’s more global 
state than e.g. writing zeroes or, well, blk_aio_ioctl().


However, testing brought some other problems to light:

$ ls /dev/sg?
/dev/sg0

$ sudo modprobe scsi_debug max_luns=1 num_tgts=1 add_host=1

$ ls /dev/sg?
/dev/sg0 /dev/sg1

$ ./qemu-system-x86_64 \
    -object iothread,id=foo \
    -device virtio-scsi,iothread=foo \
    -blockdev host_device,filename=/dev/sg1,node-name=bar \
    -device scsi-generic,drive=bar
qemu-system-x86_64: ../block/block-backend.c:2038: 
blk_set_guest_block_size: Assertion `qemu_in_main_thread()' failed.
[1]    121353 IOT instruction (core dumped)  ./qemu-system-x86_64 
-object iothread,id=foo -device virtio-scsi,iothread=foo



And:

$ ./qemu-system-x86_64 \
    -object iothread,id=foo \
    -device virtio-scsi,iothread=foo \
    -blockdev null-co,read-zeroes=true,node-name=bar \
    -device scsi-hd,drive=bar \
    -cdrom ~/tmp/arch.iso \
    -enable-kvm \
    -m 1g
qemu-system-x86_64: ../block/block-backend.c:1918: 
blk_enable_write_cache: Assertion `qemu_in_main_thread()' failed.
[1]    127203 IOT instruction (core dumped)  ./qemu-system-x86_64 
-object iothread,id=foo -device virtio-scsi,iothread=foo


If I boot from a CD under virtio-scsi (in an iothread), I also get an 
error for blk_lock_medium(), and if I eject such a CD, for blk_eject() 
and blk_get_attached_dev_id() (called from blk_eject()).


It appears like scsi_disk_emulate_command() is always run in the BB’s 
AioContext, and so everything it (indirectly) calls cannot assume to be 
in the main thread.  As for blk_set_guest_block_size(), that one’s 
called from scsi-generic’s scsi_read_complete().


Hanna




[PATCH 1/2] scsi/scsi_bus: use host_status as parameter for scsi_sense_from_host_status()

2021-12-10 Thread Dongli Zhang
The scsi_sense_from_host_status() always returns GOOD since
req->host_status is 255 (-1) at this time. Change req->host_status to
host_status so that scsi_sense_from_host_status() will be able to return
the expected value.

Fixes: f3126d65b393("scsi: move host_status handling into SCSI drivers")
Cc: Joe Jin 
Signed-off-by: Dongli Zhang 
---
 hw/scsi/scsi-bus.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index 77325d8cc7..d46650bd8c 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -1465,7 +1465,7 @@ void scsi_req_complete_failed(SCSIRequest *req, int 
host_status)
 assert(req->ops != _unit_attention);
 
 if (!req->bus->info->fail) {
-status = scsi_sense_from_host_status(req->host_status, );
+status = scsi_sense_from_host_status(host_status, );
 if (status == CHECK_CONDITION) {
 scsi_req_build_sense(req, sense);
 }
-- 
2.17.1




[PATCH 2/2] scsi/utils: pass host_status = SCSI_HOST_ERROR to guest kernel

2021-12-10 Thread Dongli Zhang
For scsi_req_complete_failed() and when the req->bus->info->fail() is
implemented, the virtio-scsi passes SCSI_HOST_ERROR to the guest kernel as
VIRTIO_SCSI_S_FAILURE, while the pvscsi passes SCSI_HOST_ERROR to guest
kernel as BTSTAT_HASOFTWARE.

However, the scsi_req_complete_failed()->scsi_sense_from_host_status()
always returns GOOD for SCSI_HOST_ERROR, when the req->bus->info->fail() is
not implemented (e.g., megasas). As a result, the sense is not passed to
the guest kernel.

The SCSI_HOST_ERROR is reproduced on purpose by below QEMU command line:

-device megasas,id=vscsi0,bus=pci.0,addr=0x4 \
-drive file=/dev/sdc,format=raw,if=none,id=drive01 \
-device scsi-block,bus=vscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive01 \

... and when we remove the /dev/sdc from the host with:

"echo 1 > /sys/block/sdc/device/delete"

This patch passes sense_code_IO_ERROR to the guest kernel for
host_status = SCSI_HOST_ERROR.

(This issue is detected by running a testing code from Rui Loura).

Cc: Joe Jin 
Cc: Adnan Misherfi 
Cc: Rui Loura 
Signed-off-by: Dongli Zhang 
---
 scsi/utils.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scsi/utils.c b/scsi/utils.c
index 357b036671..086a1fea66 100644
--- a/scsi/utils.c
+++ b/scsi/utils.c
@@ -638,6 +638,9 @@ int scsi_sense_from_host_status(uint8_t host_status,
 case SCSI_HOST_ABORTED:
 *sense = SENSE_CODE(COMMAND_ABORTED);
 return CHECK_CONDITION;
+case SCSI_HOST_ERROR:
+*sense = SENSE_CODE(IO_ERROR);
+return CHECK_CONDITION;
 case SCSI_HOST_RESET:
 *sense = SENSE_CODE(RESET);
 return CHECK_CONDITION;
-- 
2.17.1




[PATCH 0/2] scsi: to fix issue on passing host_status to the guest kernel

2021-12-10 Thread Dongli Zhang
This patchset fixes the issue on passing 'host_status' to the guest kernel.

The 1st patch fixes the erroneous usage of req->host_status.

The 2nd patch is to pass the SCSI_HOST_ERROR to the guest kernel when the
req->bus->info->fail() is not implemented. I do not add 'Fixes:' because I
am not sure if to not pass SCSI_HOST_ERROR was on purpose, especially for
security reason.

Thank you very much!

Dongli Zhang





Re: [PATCH 1/3] iotests/testrunner.py: add doc string for run_test()

2021-12-10 Thread Kevin Wolf
Am 03.12.2021 um 13:22 hat Vladimir Sementsov-Ogievskiy geschrieben:
> We are going to modify these methods and will add more documentation in
> further commit. As a preparation add basic documentation.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>  tests/qemu-iotests/testrunner.py | 13 +
>  1 file changed, 13 insertions(+)
> 
> diff --git a/tests/qemu-iotests/testrunner.py 
> b/tests/qemu-iotests/testrunner.py
> index 0e29c2fddd..fa842252d3 100644
> --- a/tests/qemu-iotests/testrunner.py
> +++ b/tests/qemu-iotests/testrunner.py
> @@ -220,6 +220,12 @@ def find_reference(self, test: str) -> str:
>  return f'{test}.out'
>  
>  def do_run_test(self, test: str) -> TestResult:
> +"""
> +Run one test
> +
> +:param test: test file path
> +"""
> +
>  f_test = Path(test)
>  f_bad = Path(f_test.name + '.out.bad')
>  f_notrun = Path(f_test.name + '.notrun')
> @@ -287,6 +293,13 @@ def do_run_test(self, test: str) -> TestResult:
>  
>  def run_test(self, test: str,
>   test_field_width: Optional[int] = None) -> TestResult:
> +"""
> +Run one test and print short status
> +
> +:param test: test file path
> +:param test_field_width: width for first field of status format
> +"""
> +
>  last_el = self.last_elapsed.get(test)
>  start = datetime.datetime.now().strftime('%H:%M:%S')

test_field_width is Optional[int], so the documentation should specify
what happens when you pass None.

Seems it doesn't change the behaviour, but just picks a default value of
8. 'test_field_width: int = 8' might have been the more obvious solution
for this in the first place.

Kevin




Re: [RFC] block-backend: prevent dangling BDS pointer in blk_drain()

2021-12-10 Thread Kevin Wolf
Am 09.12.2021 um 15:23 hat Stefan Hajnoczi geschrieben:
> The BlockBackend root child can change during bdrv_drained_begin() when
> aio_poll() is invoked. In fact the BlockDriverState can reach refcnt 0
> and blk_drain() is left with a dangling BDS pointer.
> 
> One example is scsi_device_purge_requests(), which calls blk_drain() to
> wait for in-flight requests to cancel. If the backup blockjob is active,
> then the BlockBackend root child is a temporary filter BDS owned by the
> blockjob. The blockjob can complete during bdrv_drained_begin() and the
> last reference to the BDS is released when the temporary filter node is
> removed. This results in a use-after-free when blk_drain() calls
> bdrv_drained_end(bs) on the dangling pointer.
> 
> The general problem is that a function and its callers must not assume
> that bs is still valid across aio_poll(). Explicitly hold a reference to
> bs in blk_drain() to avoid the dangling pointer.
> 
> Signed-off-by: Stefan Hajnoczi 
> ---
> I found that BDS nodes are sometimes deleted with bs->quiesce_counter >
> 0 (at least when running "make check"), so it is currently not possible
> to put the bdrv_ref/unref() calls in bdrv_do_drained_begin() and
> bdrv_do_drained_end() because they will be unbalanced. That would have
> been a more general solution than only fixing blk_drain().

They are not supposed to end up unbalanced because detaching a child
calls bdrv_unapply_subtree_drain(). In fact, I think test-bdrv-drain
tests a few scenarios like this.

Do have more details about the case that failed for you?

Kevin




Re: [PATCH v4 0/3] hw/block/fdc: Fix CVE-2021-20196

2021-12-10 Thread Kevin Wolf
Am 24.11.2021 um 17:15 hat Philippe Mathieu-Daudé geschrieben:
> Since v3:
> - Preliminary extract blk_create_empty_drive()
> - qtest checks qtest_check_clang_sanitizer() enabled
> - qtest uses null-co:// driver instead of file
> 
> Philippe Mathieu-Daudé (3):
>   hw/block/fdc: Extract blk_create_empty_drive()
>   hw/block/fdc: Kludge missing floppy drive to fix CVE-2021-20196
>   tests/qtest/fdc-test: Add a regression test for CVE-2021-20196

If I may ask a meta question: No doubt that this is a bug and it's good
that we fixed it, but why was it assigned a CVE?

Any guest can legitimately shut down and we don't consider that a denial
of service. This bug was essentially just another undocumented way for
the guest kernel to shut down, as unprivileged users in the guest can't
normally access the I/O ports of the floppy controller. I don't think we
generally consider guests killing themselves a security problem as long
as it requires kernel or root privileges in the guest.

Kevin




Re: [PATCH 2/2] iotests/149: Skip on unsupported ciphers

2021-12-10 Thread Kevin Wolf
Am 17.11.2021 um 16:05 hat Hanna Reitz geschrieben:
> On 17.11.21 16:01, Hanna Reitz wrote:
> > Whenever qemu-img or qemu-io report that some cipher is unsupported,
> > skip the whole test, because that is probably because qemu has been
> > configured with the gnutls crypto backend.
> > 
> > We could taylor the algorithm list to what gnutls supports, but this is
> > a test that is run rather rarely anyway (because it requires
> > password-less sudo), and so it seems better and easier to skip it.  When
> > this test is intentionally run to check LUKS compatibility, it seems
> > better not to limit the algorithms but keep the list extensive.
> > 
> > Signed-off-by: Hanna Reitz 
> > ---
> >   tests/qemu-iotests/149 | 23 ++-
> >   1 file changed, 18 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
> > index 328fd05a4c..adcef86e88 100755
> > --- a/tests/qemu-iotests/149
> > +++ b/tests/qemu-iotests/149
> > @@ -230,6 +230,18 @@ def create_image(config, size_mb):
> >   fn.truncate(size_mb * 1024 * 1024)
> > +def check_cipher_support(output):
> > +"""Check the output of qemu-img or qemu-io for mention of the 
> > respective
> > +cipher algorithm being unsupported, and if so, skip this test.
> > +(Returns `output` for convenience.)"""
> > +
> > +if 'Unsupported cipher algorithm' in output:
> > +iotests.notrun('Unsupported cipher algorithm '
> > +   f'{config.cipher}-{config.keylen}-{config.mode}; '
> 
> Oops.  Just when I sent this I realized that during refactoring (putting
> this code into its own function) I forgot to pass `config` as a parameter.
> 
> Didn’t notice that because...  It seems to work just fine despite `config`
> not being defined here?  Python will forever remain a black box for me...

This is an old thread by now, but I think that it works is just because
it's defined as a global variable ('for config in configs') before
calling this function.

Kevin




Re: [PATCH 3/4] Move CONFIG_XFS handling to meson.build

2021-12-10 Thread Paolo Bonzini

On 12/10/21 09:46, Thomas Huth wrote:


platform_test_xfs_fd() is only used to decide whether to invoke 
XFS_IOC_DIOINFO; but failures of XFS_IOC_DIOINFO are ignored anyway, 
so we can get rid of is_xfs in BDRVRawState, too.


After staring at the code for a while, I wonder why we're not simply 
using fstat() here instead to get the st_blksize value... wouldn't that 
be better anyway since it also works with other file system types?


The value that XFS_IOC_DIOINFO returns is the logical sector size of the 
underlying device; it should be 512 or 4096, but more likely 512.  It 
can be smaller than st_blksize, because often it will be if it is 512 
but the st_blksize is usually 4096.


If it is wrong, QEMU will do unnecessary read/modify/write operations 
for disk writes that are not 4K-aligned.


Paolo



Re: [PATCH 3/4] Move CONFIG_XFS handling to meson.build

2021-12-10 Thread Thomas Huth

On 10/12/2021 09.39, Paolo Bonzini wrote:

On 12/10/21 08:53, Thomas Huth wrote:

On 02/11/2021 12.34, Paolo Bonzini wrote:

On 28/10/21 20:59, Thomas Huth wrote:

Checking for xfsctl() can be done more easily in meson.build. Also,
this is not a "real" feature like the other features that we provide
with the "--enable-xxx" and "--disable-xxx" switches for the
configure script, since this does not influence lots of code (it's
only about one call to xfsctl() in file-posix.c), so people don't
gain much with the ability to disable this with "--disable-xfsctl".
Let's rather treat this like the other cc.has_function() checks in
meson.build, i.e. don't add a new option for this in meson_options.txt.

Signed-off-by: Thomas Huth 


I think we should just use ioctl and copy the relevant definitions from 
Linux:


struct dioattr {
 u32   d_mem;  /* data buffer memory alignment */
 u32   d_miniosz;  /* min xfer size    */
 u32   d_maxiosz;  /* max xfer size    */
};

#define XFS_IOC_DIOINFO    _IOR ('X', 30, struct dioattr)


I've now had a closer look at this idea, but it's getting messy: We'd 
additionally also need the platform_test_xfs_fd() function that is called 
from file-posix.c ...


platform_test_xfs_fd() is only used to decide whether to invoke 
XFS_IOC_DIOINFO; but failures of XFS_IOC_DIOINFO are ignored anyway, so we 
can get rid of is_xfs in BDRVRawState, too.


After staring at the code for a while, I wonder why we're not simply using 
fstat() here instead to get the st_blksize value... wouldn't that be better 
anyway since it also works with other file system types?


 Thomas




Re: [PATCH 3/4] Move CONFIG_XFS handling to meson.build

2021-12-10 Thread Paolo Bonzini

On 12/10/21 08:53, Thomas Huth wrote:

On 02/11/2021 12.34, Paolo Bonzini wrote:

On 28/10/21 20:59, Thomas Huth wrote:

Checking for xfsctl() can be done more easily in meson.build. Also,
this is not a "real" feature like the other features that we provide
with the "--enable-xxx" and "--disable-xxx" switches for the
configure script, since this does not influence lots of code (it's
only about one call to xfsctl() in file-posix.c), so people don't
gain much with the ability to disable this with "--disable-xfsctl".
Let's rather treat this like the other cc.has_function() checks in
meson.build, i.e. don't add a new option for this in meson_options.txt.

Signed-off-by: Thomas Huth 


I think we should just use ioctl and copy the relevant definitions 
from Linux:


struct dioattr {
 u32   d_mem;  /* data buffer memory alignment */
 u32   d_miniosz;  /* min xfer size    */
 u32   d_maxiosz;  /* max xfer size    */
};

#define XFS_IOC_DIOINFO    _IOR ('X', 30, struct dioattr)


I've now had a closer look at this idea, but it's getting messy: We'd 
additionally also need the platform_test_xfs_fd() function that is 
called from file-posix.c ...


platform_test_xfs_fd() is only used to decide whether to invoke 
XFS_IOC_DIOINFO; but failures of XFS_IOC_DIOINFO are ignored anyway, so 
we can get rid of is_xfs in BDRVRawState, too.


Paolo



Re: [Libguestfs] [libnbd PATCH 00/13] libnbd patches for NBD_OPT_EXTENDED_HEADERS

2021-12-10 Thread Laszlo Ersek
On 12/04/21 00:17, Eric Blake wrote:
> Available here: 
> https://repo.or.cz/libnbd/ericb.git/shortlog/refs/tags/exthdr-v1
> 
> I also want to do followup patches to teach 'nbdinfo --map' and
> 'nbdcopy' to utilize 64-bit extents.
> 
> Eric Blake (13):
>   golang: Simplify nbd_block_status callback array copy
>   block_status: Refactor array storage
>   protocol: Add definitions for extended headers
>   protocol: Prepare to send 64-bit requests
>   protocol: Prepare to receive 64-bit replies
>   protocol: Accept 64-bit holes during pread
>   generator: Add struct nbd_extent in prep for 64-bit extents
>   block_status: Track 64-bit extents internally
>   block_status: Accept 64-bit extents during block status
>   api: Add [aio_]nbd_block_status_64
>   api: Add three functions for controlling extended headers
>   generator: Actually request extended headers
>   interop: Add test of 64-bit block status
> 
>  lib/internal.h|  31 ++-
>  lib/nbd-protocol.h|  61 -
>  generator/API.ml  | 237 --
>  generator/API.mli |   3 +-
>  generator/C.ml|  24 +-
>  generator/GoLang.ml   |  35 ++-
>  generator/Makefile.am |   3 +-
>  generator/OCaml.ml|  20 +-
>  generator/Python.ml   |  29 ++-
>  generator/state_machine.ml|  52 +++-
>  generator/states-issue-command.c  |  31 ++-
>  .../states-newstyle-opt-extended-headers.c|  90 +++
>  generator/states-newstyle-opt-starttls.c  |  10 +-
>  generator/states-reply-structured.c   | 220 
>  generator/states-reply.c  |  31 ++-
>  lib/handle.c  |  27 +-
>  lib/rw.c  | 105 +++-
>  python/t/110-defaults.py  |   3 +-
>  python/t/120-set-non-defaults.py  |   4 +-
>  python/t/465-block-status-64.py   |  56 +
>  ocaml/helpers.c   |  22 +-
>  ocaml/nbd-c.h |   3 +-
>  ocaml/tests/Makefile.am   |   5 +-
>  ocaml/tests/test_110_defaults.ml  |   4 +-
>  ocaml/tests/test_120_set_non_defaults.ml  |   5 +-
>  ocaml/tests/test_465_block_status_64.ml   |  58 +
>  tests/meta-base-allocation.c  | 111 +++-
>  interop/Makefile.am   |   6 +
>  interop/large-status.c| 186 ++
>  interop/large-status.sh   |  49 
>  .gitignore|   1 +
>  golang/Makefile.am|   3 +-
>  golang/handle.go  |   6 +
>  golang/libnbd_110_defaults_test.go|   8 +
>  golang/libnbd_120_set_non_defaults_test.go|  12 +
>  golang/libnbd_465_block_status_64_test.go | 119 +
>  36 files changed, 1511 insertions(+), 159 deletions(-)
>  create mode 100644 generator/states-newstyle-opt-extended-headers.c
>  create mode 100644 python/t/465-block-status-64.py
>  create mode 100644 ocaml/tests/test_465_block_status_64.ml
>  create mode 100644 interop/large-status.c
>  create mode 100755 interop/large-status.sh
>  create mode 100644 golang/libnbd_465_block_status_64_test.go
> 

I figured I should slowly / gradually review this series, and as a
*pre-requisite* for it, first apply the spec patch, and then read
through the spec with something like

$ git show --color -U1000

In other words, read the whole spec, just highlight the new additions.

Now, I see Vladimir has made several comments on the spec patch; will
those comments necessitate a respin of the libnbd series? If so, how
intrusive are the changes going to be? I'm hesitant to start my review
if significant changes are already foreseen.

Thanks,
Laszlo