Re: [PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Chris Johns]

On 18/8/2023 11:51 pm, Kinsey Moore wrote:
> On Thu, Aug 17, 2023 at 7:25 PM Chris Johns  > wrote:
> 
> On 17/8/2023 10:43 pm, Kinsey Moore wrote:
> > On Wed, Aug 16, 2023 at 9:47 PM Chris Johns  
> > >> wrote:
> >
> >     On 17/8/2023 7:36 am, Kinsey Moore wrote:
> >     > Before the fatal-error test result type was introduced, 
> minimum.exe was
> >     > classified as an invalid test since it lacked a proper test 
> header and
> >     > trailer. This applies the test exclusions to all test states to 
> avoid
> >     > this happening again in the future.
> >
> >     I do not think this right. What state are you seeing?
> >
> >
> > The minimum test fails with a fatal error:
> >     "*** FATAL ***",
> >     "fatal source: 0 (INTERNAL_ERROR_CORE)",
> >     "fatal code: 5 (INTERNAL_ERROR_THREAD_EXITTED)",
> >     "RTEMS version: 6.0.0.c76c98c344d803fa80361884c4cc79f0b3607ec8",
> >     "RTEMS tools: 12.3.1 20230626 (RTEMS 6, RSB
> > 8e568b2ca3489d6bfa48e1d29618ea9b48a5b408, Newlib 4c7d0df)",
> >     "executing thread ID: 0x09010001",
> >
> > As far as I'm aware, this is a normal exit for minimum since it does 
> nothing
> > which includes not properly exiting the test by allowing Init() to 
> return.
> >
> >
> >     While we cannot determine the pass state because there is no end
> message we can
> >     determine if the test has timed out or is too long and they should 
> be
> a fail.
> >
> >
> > This is not timing out or taking too long as per the above. I could 
> move the
> > check to cover just the invalid/fatal-error checks instead of all of 
> them
> if you
> > prefer.
> 
> This looks to me like an issue in minimum and it looks to me like minimal 
> has
> some other issue. What is printing out the message? If there is code in 
> minimal
> to print this error then why not print a normal test start and end banner?
> 
> My understanding is minimal has not means to output anything.
> 
> 
> It seems to be intentional:
> static void *Init( uintptr_t ignored )
> {
>   /* initialize application */
> 
>   /* Real application would call idle loop functionality */
> 
>   /* but in this case, just return and fall into a fatal error */
> 
>   return NULL;
> }
> 
> Even if no console is requested (not even simple console), BSP_output_char is
> still available.

Is this happening for all BSPs and all archs?

When the original filter for minimal was made minimal never reported anything so
what has changed to make this happen?

I think what we have is right because it does not suppress real issues minimal
could have if there is a regression. I think there is more to this we need to
understand before we decide to change the tester filtering.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Kinsey Moore]

On Thu, Aug 17, 2023 at 7:25 PM Chris Johns  wrote:

> On 17/8/2023 10:43 pm, Kinsey Moore wrote:
> > On Wed, Aug 16, 2023 at 9:47 PM Chris Johns  > > wrote:
> >
> > On 17/8/2023 7:36 am, Kinsey Moore wrote:
> > > Before the fatal-error test result type was introduced,
> minimum.exe was
> > > classified as an invalid test since it lacked a proper test header
> and
> > > trailer. This applies the test exclusions to all test states to
> avoid
> > > this happening again in the future.
> >
> > I do not think this right. What state are you seeing?
> >
> >
> > The minimum test fails with a fatal error:
> > "*** FATAL ***",
> > "fatal source: 0 (INTERNAL_ERROR_CORE)",
> > "fatal code: 5 (INTERNAL_ERROR_THREAD_EXITTED)",
> > "RTEMS version: 6.0.0.c76c98c344d803fa80361884c4cc79f0b3607ec8",
> > "RTEMS tools: 12.3.1 20230626 (RTEMS 6, RSB
> > 8e568b2ca3489d6bfa48e1d29618ea9b48a5b408, Newlib 4c7d0df)",
> > "executing thread ID: 0x09010001",
> >
> > As far as I'm aware, this is a normal exit for minimum since it does
> nothing
> > which includes not properly exiting the test by allowing Init() to
> return.
> >
> >
> > While we cannot determine the pass state because there is no end
> message we can
> > determine if the test has timed out or is too long and they should
> be a fail.
> >
> >
> > This is not timing out or taking too long as per the above. I could move
> the
> > check to cover just the invalid/fatal-error checks instead of all of
> them if you
> > prefer.
>
> This looks to me like an issue in minimum and it looks to me like minimal
> has
> some other issue. What is printing out the message? If there is code in
> minimal
> to print this error then why not print a normal test start and end banner?
>
> My understanding is minimal has not means to output anything.
>

It seems to be intentional:
static void *Init( uintptr_t ignored )
{
  /* initialize application */

  /* Real application would call idle loop functionality */

  /* but in this case, just return and fall into a fatal error */

  return NULL;
}

Even if no console is requested (not even simple console), BSP_output_char
is still available.

Kinsey
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Chris Johns]

On 17/8/2023 10:43 pm, Kinsey Moore wrote:
> On Wed, Aug 16, 2023 at 9:47 PM Chris Johns  > wrote:
> 
> On 17/8/2023 7:36 am, Kinsey Moore wrote:
> > Before the fatal-error test result type was introduced, minimum.exe was
> > classified as an invalid test since it lacked a proper test header and
> > trailer. This applies the test exclusions to all test states to avoid
> > this happening again in the future.
> 
> I do not think this right. What state are you seeing?
> 
> 
> The minimum test fails with a fatal error:
>     "*** FATAL ***",
>     "fatal source: 0 (INTERNAL_ERROR_CORE)",
>     "fatal code: 5 (INTERNAL_ERROR_THREAD_EXITTED)",
>     "RTEMS version: 6.0.0.c76c98c344d803fa80361884c4cc79f0b3607ec8",
>     "RTEMS tools: 12.3.1 20230626 (RTEMS 6, RSB
> 8e568b2ca3489d6bfa48e1d29618ea9b48a5b408, Newlib 4c7d0df)",
>     "executing thread ID: 0x09010001",
> 
> As far as I'm aware, this is a normal exit for minimum since it does nothing
> which includes not properly exiting the test by allowing Init() to return.
> 
> 
> While we cannot determine the pass state because there is no end message 
> we can
> determine if the test has timed out or is too long and they should be a 
> fail.
> 
> 
> This is not timing out or taking too long as per the above. I could move the
> check to cover just the invalid/fatal-error checks instead of all of them if 
> you
> prefer.

This looks to me like an issue in minimum and it looks to me like minimal has
some other issue. What is printing out the message? If there is code in minimal
to print this error then why not print a normal test start and end banner?

My understanding is minimal has not means to output anything.

Chris

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Kinsey Moore]

On Wed, Aug 16, 2023 at 9:47 PM Chris Johns  wrote:

> On 17/8/2023 7:36 am, Kinsey Moore wrote:
> > Before the fatal-error test result type was introduced, minimum.exe was
> > classified as an invalid test since it lacked a proper test header and
> > trailer. This applies the test exclusions to all test states to avoid
> > this happening again in the future.
>
> I do not think this right. What state are you seeing?
>

The minimum test fails with a fatal error:
"*** FATAL ***",
"fatal source: 0 (INTERNAL_ERROR_CORE)",
"fatal code: 5 (INTERNAL_ERROR_THREAD_EXITTED)",
"RTEMS version: 6.0.0.c76c98c344d803fa80361884c4cc79f0b3607ec8",
"RTEMS tools: 12.3.1 20230626 (RTEMS 6, RSB
8e568b2ca3489d6bfa48e1d29618ea9b48a5b408, Newlib 4c7d0df)",
"executing thread ID: 0x09010001",

As far as I'm aware, this is a normal exit for minimum since it does
nothing which includes not properly exiting the test by allowing Init() to
return.


> While we cannot determine the pass state because there is no end message
> we can
> determine if the test has timed out or is too long and they should be a
> fail.
>

This is not timing out or taking too long as per the above. I could move
the check to cover just the invalid/fatal-error checks instead of all of
them if you prefer.

Kinsey
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Chris Johns]

On 17/8/2023 7:36 am, Kinsey Moore wrote:
> Before the fatal-error test result type was introduced, minimum.exe was
> classified as an invalid test since it lacked a proper test header and
> trailer. This applies the test exclusions to all test states to avoid
> this happening again in the future.

I do not think this right. What state are you seeing?

While we cannot determine the pass state because there is no end message we can
determine if the test has timed out or is too long and they should be a fail.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

[PATCH rtems-tools] tester/report: Apply test excludes to fatal-error

[Kinsey Moore]

Before the fatal-error test result type was introduced, minimum.exe was
classified as an invalid test since it lacked a proper test header and
trailer. This applies the test exclusions to all test states to avoid
this happening again in the future.
---
 tester/rt/report.py | 15 +++
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/tester/rt/report.py b/tester/rt/report.py
index a688dc8..5c369e6 100644
--- a/tester/rt/report.py
+++ b/tester/rt/report.py
@@ -192,7 +192,11 @@ class report(object):
 if tools != self.config['tools']:
 state = 'WRONG_TOOLS'
 if state is None or state == 'EXPECTED_PASS':
-if start and end:
+exe_name = path.basename(name).split('.')[0]
+if exe_name in test_fail_excludes:
+status = 'passed'
+self.passed += 1
+elif start and end:
 if state is None or state == 'EXPECTED_PASS':
 status = 'passed'
 self.passed += 1
@@ -210,13 +214,8 @@ class report(object):
 status = 'failed'
 self.failed += 1
 else:
-exe_name = path.basename(name).split('.')[0]
-if exe_name in test_fail_excludes:
-status = 'passed'
-self.passed += 1
-else:
-status = 'invalid'
-self.invalids += 1
+status = 'invalid'
+self.invalids += 1
 else:
 if state == 'EXPECTED_FAIL':
 if start and end:
-- 
2.39.2

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel