Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
On Sun, May 16, 2021 at 09:44:53PM -0300, João Marcos Costa wrote: > Hello, > > Em ter., 11 de mai. de 2021 às 10:04, Richard Genoud < > richard.gen...@posteo.net> escreveu: > > > Hi all, > > > > Le 08/05/2021 à 23:51, Simon Glass a écrit : > > > Hi, > > > > > > On Thu, 4 Feb 2021 at 15:32, João Marcos Costa > > wrote: > > >> > > >> Em qua., 27 de jan. de 2021 às 12:15, Simon Glass > > escreveu: > > >>> > > >>> Hi Joao, > > >> > > >> Hello! > > >>> > > >>> This test works the first time I run it but fails the second time, > > >>> since the directory already exists. This makes it necessary to disable > > >>> the test for development. > > >>> > > >>> It also uses the wrong quoting style - we have settled on a single > > >>> quote by default in U-Boot. > > >>> > > >>> Finally, the tests and some functions need comments about what they do > > >>> and what the arguments are. > > >>> > > >>> Please can you take a look? > > >> > > >> Absolutely. Excuse me for such a late reply. > > > > > > Any word on this please? Have you been able to repeat this? > > Yes, for me, reading fragmented files doesn't work. > > The test "test_sqfs_load" is OK because it only tests the file length not > > its content. > > > > I've written a patch to check if the file is corrupted or not, and it > > fails : > > ./test/py/test.py --bd sandbox --build -k test_sqfs_load -v > > [...] > > > > I finally could get back to SquashFS support today, and I fixed a few bugs > concerning the fragmented files. However, I still need to run a few more > tests before submitting the patches. Please make sure the problem with repeatedly running the tests is fixed as well, that's what's blocking using the public gitlab CI runners, thanks! -- Tom signature.asc Description: PGP signature
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
Hello, Em ter., 11 de mai. de 2021 às 10:04, Richard Genoud < richard.gen...@posteo.net> escreveu: > Hi all, > > Le 08/05/2021 à 23:51, Simon Glass a écrit : > > Hi, > > > > On Thu, 4 Feb 2021 at 15:32, João Marcos Costa > wrote: > >> > >> Em qua., 27 de jan. de 2021 às 12:15, Simon Glass > escreveu: > >>> > >>> Hi Joao, > >> > >> Hello! > >>> > >>> This test works the first time I run it but fails the second time, > >>> since the directory already exists. This makes it necessary to disable > >>> the test for development. > >>> > >>> It also uses the wrong quoting style - we have settled on a single > >>> quote by default in U-Boot. > >>> > >>> Finally, the tests and some functions need comments about what they do > >>> and what the arguments are. > >>> > >>> Please can you take a look? > >> > >> Absolutely. Excuse me for such a late reply. > > > > Any word on this please? Have you been able to repeat this? > Yes, for me, reading fragmented files doesn't work. > The test "test_sqfs_load" is OK because it only tests the file length not > its content. > > I've written a patch to check if the file is corrupted or not, and it > fails : > ./test/py/test.py --bd sandbox --build -k test_sqfs_load -v > [...] > I finally could get back to SquashFS support today, and I fixed a few bugs concerning the fragmented files. However, I still need to run a few more tests before submitting the patches. Best regards, Joao Marcos www.linkedin.com/in/jmarcoscosta/ https://github.com/jmarcoscosta
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
Hi all,
Le 08/05/2021 à 23:51, Simon Glass a écrit :
Hi,
On Thu, 4 Feb 2021 at 15:32, João Marcos Costa wrote:
Em qua., 27 de jan. de 2021 às 12:15, Simon Glass escreveu:
Hi Joao,
Hello!
This test works the first time I run it but fails the second time,
since the directory already exists. This makes it necessary to disable
the test for development.
It also uses the wrong quoting style - we have settled on a single
quote by default in U-Boot.
Finally, the tests and some functions need comments about what they do
and what the arguments are.
Please can you take a look?
Absolutely. Excuse me for such a late reply.
Any word on this please? Have you been able to repeat this?
Yes, for me, reading fragmented files doesn't work.
The test "test_sqfs_load" is OK because it only tests the file length not its
content.
I've written a patch to check if the file is corrupted or not, and it fails :
./test/py/test.py --bd sandbox --build -k test_sqfs_load -v
[...]
AssertionError: assert not 'ERROR' in 'crc32 for 0100 ... 010013eb ==>
df8e6fe2 != d1522690 ** ERROR **'
[...]
Here's the patch onto v2021.07-rc2 :
--8<
Subject: [PATCH] test/py: SquashFS: Check if loaded file is corrupted
After loading the file in memory, its content should be checked for
errors.
Signed-off-by: Richard Genoud
---
test/py/tests/test_fs/test_squashfs/sqfs_common.py| 5 -
test/py/tests/test_fs/test_squashfs/test_sqfs_load.py | 6 +-
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/test/py/tests/test_fs/test_squashfs/sqfs_common.py
b/test/py/tests/test_fs/test_squashfs/sqfs_common.py
index c96f92c1d8f..a7673c73762 100644
--- a/test/py/tests/test_fs/test_squashfs/sqfs_common.py
+++ b/test/py/tests/test_fs/test_squashfs/sqfs_common.py
@@ -6,6 +6,7 @@ import os
import random
import string
import subprocess
+import zlib
def sqfs_get_random_letters(size):
letters = []
@@ -19,12 +20,14 @@ def sqfs_generate_file(path, size):
file = open(path, "w")
file.write(content)
file.close()
+return zlib.crc32(content.encode())
class Compression:
def __init__(self, name, files, sizes, block_size = 4096):
self.name = name
self.files = files
self.sizes = sizes
+self.crc = []
self.mksquashfs_opts = " -b " + str(block_size) + " -comp " + self.name
def add_opt(self, opt):
@@ -34,7 +37,7 @@ class Compression:
src = os.path.join(build_dir, "sqfs_src/")
os.mkdir(src)
for (f, s) in zip(self.files, self.sizes):
-sqfs_generate_file(src + f, s)
+self.crc.append(sqfs_generate_file(src + f, s))
# the symbolic link always targets the first file
os.symlink(self.files[0], src + "sym")
diff --git a/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
b/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
index 9e900623846..2ab4660036e 100644
--- a/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
+++ b/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
@@ -4,6 +4,7 @@
import os
import pytest
+import zlib
from sqfs_common import *
@pytest.mark.boardspec('sandbox')
@@ -14,6 +15,7 @@ from sqfs_common import *
def test_sqfs_load(u_boot_console):
build_dir = u_boot_console.config.build_dir
command = "sqfsload host 0 $kernel_addr_r "
+sum_command = "crc32 -v $kernel_addr_r $filesize "
for opt in comp_opts:
# generate and load the squashfs image
@@ -30,10 +32,12 @@ def test_sqfs_load(u_boot_console):
output = u_boot_console.run_command(command + "xxx")
assert "File not found." in output
-for (f, s) in zip(opt.files, opt.sizes):
+for (f, s, c) in zip(opt.files, opt.sizes, opt.crc):
try:
output = u_boot_console.run_command(command + f)
assert str(s) in output
+output = u_boot_console.run_command(sum_command + format(c,
'08x'))
+assert not 'ERROR' in output
except:
assert False
opt.cleanup(build_dir)
--
2.20.1
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
Hi, On Thu, 4 Feb 2021 at 15:32, João Marcos Costa wrote: > > > > Em qua., 27 de jan. de 2021 às 12:15, Simon Glass > escreveu: >> >> Hi Joao, > > Hello! >> >> >> >> This test works the first time I run it but fails the second time, >> since the directory already exists. This makes it necessary to disable >> the test for development. >> >> It also uses the wrong quoting style - we have settled on a single >> quote by default in U-Boot. >> >> Finally, the tests and some functions need comments about what they do >> and what the arguments are. >> >> Please can you take a look? > > Absolutely. Excuse me for such a late reply. Any word on this please? Have you been able to repeat this? Regards, Simon
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
Em qua., 27 de jan. de 2021 às 12:15, Simon Glass escreveu: > Hi Joao, Hello! > > > This test works the first time I run it but fails the second time, > since the directory already exists. This makes it necessary to disable > the test for development. > > It also uses the wrong quoting style - we have settled on a single > quote by default in U-Boot. > > Finally, the tests and some functions need comments about what they do > and what the arguments are. > > Please can you take a look? Absolutely. Excuse me for such a late reply. > > Thanks, > Simon > -- Atenciosamente, João Marcos Costa www.linkedin.com/in/jmarcoscosta/ https://github.com/jmarcoscosta
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
Hi Joao, On Wed, 25 Nov 2020 at 01:58, Richard Genoud wrote: > > Hi, > > Le 20/11/2020 à 02:35, Tom Rini a écrit : > > On Tue, Nov 03, 2020 at 12:11:25PM +0100, Richard Genoud wrote: > > > >> The code for reading a fragmented file is not functionnal. > >> It's better to signal this to the user. > >> > >> Signed-off-by: Richard Genoud > > > > This change causes the test.py squashfs tests to fail. I am unsure if > > the problem is with the tests or this exposing further problems in the > > code. > Actually, reading a fragmented file doesn't work. > The test only check if the file is read, but not it's content. > > With this following patch, we'll see that the file content is not the same : > > > From 68f87301c059aaae8e90e42fbec9b560aee0c6eb Mon Sep 17 00:00:00 2001 > From: Richard Genoud > Date: Tue, 24 Nov 2020 17:45:07 +0100 > Subject: [PATCH] test/py: SquashFS: Check if loaded file is corrupted > > After loading the file in memory, its content should be checked for > errors. > > Signed-off-by: Richard Genoud > --- > test/py/tests/test_fs/test_squashfs/sqfs_common.py| 5 - > test/py/tests/test_fs/test_squashfs/test_sqfs_load.py | 6 +- > 2 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/test/py/tests/test_fs/test_squashfs/sqfs_common.py > b/test/py/tests/test_fs/test_squashfs/sqfs_common.py > index c96f92c1d8f..a7673c73762 100644 > --- a/test/py/tests/test_fs/test_squashfs/sqfs_common.py > +++ b/test/py/tests/test_fs/test_squashfs/sqfs_common.py This test works the first time I run it but fails the second time, since the directory already exists. This makes it necessary to disable the test for development. It also uses the wrong quoting style - we have settled on a single quote by default in U-Boot. Finally, the tests and some functions need comments about what they do and what the arguments are. Please can you take a look? Thanks, Simon
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
Hi,
Le 20/11/2020 à 02:35, Tom Rini a écrit :
On Tue, Nov 03, 2020 at 12:11:25PM +0100, Richard Genoud wrote:
The code for reading a fragmented file is not functionnal.
It's better to signal this to the user.
Signed-off-by: Richard Genoud
This change causes the test.py squashfs tests to fail. I am unsure if
the problem is with the tests or this exposing further problems in the
code.
Actually, reading a fragmented file doesn't work.
The test only check if the file is read, but not it's content.
With this following patch, we'll see that the file content is not the same :
From 68f87301c059aaae8e90e42fbec9b560aee0c6eb Mon Sep 17 00:00:00 2001
From: Richard Genoud
Date: Tue, 24 Nov 2020 17:45:07 +0100
Subject: [PATCH] test/py: SquashFS: Check if loaded file is corrupted
After loading the file in memory, its content should be checked for
errors.
Signed-off-by: Richard Genoud
---
test/py/tests/test_fs/test_squashfs/sqfs_common.py| 5 -
test/py/tests/test_fs/test_squashfs/test_sqfs_load.py | 6 +-
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/test/py/tests/test_fs/test_squashfs/sqfs_common.py
b/test/py/tests/test_fs/test_squashfs/sqfs_common.py
index c96f92c1d8f..a7673c73762 100644
--- a/test/py/tests/test_fs/test_squashfs/sqfs_common.py
+++ b/test/py/tests/test_fs/test_squashfs/sqfs_common.py
@@ -6,6 +6,7 @@ import os
import random
import string
import subprocess
+import zlib
def sqfs_get_random_letters(size):
letters = []
@@ -19,12 +20,14 @@ def sqfs_generate_file(path, size):
file = open(path, "w")
file.write(content)
file.close()
+return zlib.crc32(content.encode())
class Compression:
def __init__(self, name, files, sizes, block_size = 4096):
self.name = name
self.files = files
self.sizes = sizes
+self.crc = []
self.mksquashfs_opts = " -b " + str(block_size) + " -comp " + self.name
def add_opt(self, opt):
@@ -34,7 +37,7 @@ class Compression:
src = os.path.join(build_dir, "sqfs_src/")
os.mkdir(src)
for (f, s) in zip(self.files, self.sizes):
-sqfs_generate_file(src + f, s)
+self.crc.append(sqfs_generate_file(src + f, s))
# the symbolic link always targets the first file
os.symlink(self.files[0], src + "sym")
diff --git a/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
b/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
index 9e900623846..2ab4660036e 100644
--- a/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
+++ b/test/py/tests/test_fs/test_squashfs/test_sqfs_load.py
@@ -4,6 +4,7 @@
import os
import pytest
+import zlib
from sqfs_common import *
@pytest.mark.boardspec('sandbox')
@@ -14,6 +15,7 @@ from sqfs_common import *
def test_sqfs_load(u_boot_console):
build_dir = u_boot_console.config.build_dir
command = "sqfsload host 0 $kernel_addr_r "
+sum_command = "crc32 -v $kernel_addr_r $filesize "
for opt in comp_opts:
# generate and load the squashfs image
@@ -30,10 +32,12 @@ def test_sqfs_load(u_boot_console):
output = u_boot_console.run_command(command + "xxx")
assert "File not found." in output
-for (f, s) in zip(opt.files, opt.sizes):
+for (f, s, c) in zip(opt.files, opt.sizes, opt.crc):
try:
output = u_boot_console.run_command(command + f)
assert str(s) in output
+output = u_boot_console.run_command(sum_command + format(c,
'08x'))
+assert not 'ERROR' in output
except:
assert False
opt.cleanup(build_dir)
Re: [PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
On Tue, Nov 03, 2020 at 12:11:25PM +0100, Richard Genoud wrote: > The code for reading a fragmented file is not functionnal. > It's better to signal this to the user. > > Signed-off-by: Richard Genoud This change causes the test.py squashfs tests to fail. I am unsure if the problem is with the tests or this exposing further problems in the code. -- Tom signature.asc Description: PGP signature
[PATCH v2 27/28] fs/squashfs: sqfs_read: fragmented files are not supported
The code for reading a fragmented file is not functionnal.
It's better to signal this to the user.
Signed-off-by: Richard Genoud
---
fs/squashfs/sqfs.c | 7 +++
1 file changed, 7 insertions(+)
diff --git a/fs/squashfs/sqfs.c b/fs/squashfs/sqfs.c
index f63a06fd40f..a96c1d4f564 100644
--- a/fs/squashfs/sqfs.c
+++ b/fs/squashfs/sqfs.c
@@ -1498,6 +1498,13 @@ int sqfs_read(const char *filename, void *buf, loff_t
offset, loff_t len,
goto out;
}
+ printf("Error: reading a fragmented file is not supported yet.\n");
+ ret = -EINVAL;
+ goto out;
+
+ /*
+* TODO: reading a fragmented file doesn't work
+*/
start = frag_entry.start / ctxt.cur_dev->blksz;
table_size = SQFS_BLOCK_SIZE(frag_entry.size);
table_offset = frag_entry.start - (start * ctxt.cur_dev->blksz);
