Re: release 3.1.0 critical bugs still outstanding

[Steve Lawrence]

Agreed, I think the CLI change is worth getting in for 3.1.0. Shouldn't
take too long to get it merged.

I'll volunteer to be release manager 3.1.0. Probably for the best since
I'm probably the most familiar with the process/script, and it's
possible something could be broken with this being the first
non-incubator release.

Though, I would recommend that other PMC/committers go through the
process to create and publish a gpg key so that when it does come time
to do a release, at least that part is out of the way.


On 5/11/21 10:10 AM, Interrante, John A (GE Research, US) wrote:
> Yes, we will be in good shape for the 3.1.0 release after we update the CLI 
> help information, which I'd like to complete today.  We've also got at least 
> 4-5 days to add validation.md and anything else we want to the daffodil-site 
> before the earliest possible official release announcement could go out.
> 
> I'm taking this Thursday and Friday off which is a consideration in 
> volunteering to be the release manager.  I'd have to generate a signing key, 
> add its public part to the KEYS file, commit it to the Daffodil release 
> distribution SVN repository, send the fingerprint to the Apache key server, 
> build a release candidate, and start a vote before I go out of town on 
> Thursday.  Steve, perhaps I should wait for the following release unless you 
> think I'd be able to do all these steps quickly within a few hours.
> 
> John
> 
> -Original Message-
> From: Beckerle, Mike  
> Sent: Monday, May 10, 2021 2:43 PM
> To: dev@daffodil.apache.org
> Subject: EXT: Re: release 3.1.0 critical bugs still outstanding
> 
> I agree we should do the release.
> 
> I am in the thick of debugging DAFFODIL-1422, but there's a bunch of 
> refactoring here, 30 files touched, changes in diagnostic behavior, etc. 
> Perhaps best to put it off until after the 3.1.0 release.
> 
> 
> 
> 
> 
> From: Steve Lawrence 
> Sent: Monday, May 10, 2021 1:59 PM
> To: dev@daffodil.apache.org 
> Subject: Re: release 3.1.0 critical bugs still outstanding
> 
> We have fixed the later two mentioned issues. The current list of critical 
> issues is now:
> 
> DAFFODIL-1422: disallow doctype decls in all XML & XSD that we read in
> DAFFODIL-2400: New SAX API causes performance degredations.
> DAFFODIL-2473: Need bit-wise AND, OR, NOT, and shift operations
> 
> I agree the the first two can likely be postponed without issue. The last one 
> doesn't even seem critical to me, unless there are very important formats 
> that require this functions that I'm not aware of. I suggest we also postpone 
> that ticket as well.
> 
> If others agree, I think we are ready for the 3.1.0 release?
> 
> Does any want to volunteer to be the release manager? I've done it a handful 
> of times so don't mind, but it might be good to get others experience 
> depending on availability. By this point, the workflow is pretty well 
> documented here:
> 
> https://cwiki.apache.org/confluence/display/DAFFODIL/Release+Workflow
> 
> 
> On 5/3/21 5:25 PM, Beckerle, Mike wrote:
>> Of the 4 remaining "critical bugs or improvements" I think we should 
>> postpone and release note these first two:
>>
>>   *
>>
>>   *   Improvement: https://issues.apache.org/jira/browse/DAFFODIL-2400 - New 
>> SAX API causes performance degradations.
>>  *It is a mystery why the SAX API is slower. The whole point of SAX 
>> is lighter weight.
>>   *   Improvement: https://issues.apache.org/jira/browse/DAFFODIL-1422 - 
>> disallow doctype decls in all XML and XSD we read in.
>>  *   Assigned to Mike Beckerle. Unlikely to be finished in time for 
>> release 3.1.0. Substantial code refactoring to do this right.
>>
>> These next two seem rather important to fix:
>>
>>   *   Bug: https://issues.apache.org/jira/browse/DAFFODIL-2183 - Unparse 
>> complex nilled element fails
>>  *   There are data formats where I advised people a best-practice is to 
>> use complex nilled elements to model a specific situation.
>>   *   Bug: https://issues.apache.org/jira/browse/DAFFODIL-2399 - error 
>> diagnostics output even though there is an infoset
>>  *   This one is assigned to Steve Lawrence
>>  *   Seems rather important. Was a user-reported issue I believe.
>>
>> The 5th critical ticket is for a new feature (bitwise and/or/xor, and shift 
>> functions), so we can postpone that one.
>>
>> So only DAFFODIL-2183 really needs someone to take it on.
>>
>> 
>> From: Interrante, John A (GE Research, US) 
>> Sent: Monday, May 3, 2021 4:57 PM
>> To: dev@daffodil.apache.org ...
>>
>> Are any of the 5 critical bugs (2 of which need developers to work on them) 
>> going to hold up the 3.1.0 release?  The report doesn't say so, but I had 
>> the impression you'd added the remaining critical bugs (which were unlikely 
>> to be hit by people) to the 3.1.0 release notes so that the 3.1.0 release 
>> still could go out.  If any critical bugs are 

RE: release 3.1.0 critical bugs still outstanding

[Interrante, John A (GE Research, US)]

Yes, we will be in good shape for the 3.1.0 release after we update the CLI 
help information, which I'd like to complete today.  We've also got at least 
4-5 days to add validation.md and anything else we want to the daffodil-site 
before the earliest possible official release announcement could go out.

I'm taking this Thursday and Friday off which is a consideration in 
volunteering to be the release manager.  I'd have to generate a signing key, 
add its public part to the KEYS file, commit it to the Daffodil release 
distribution SVN repository, send the fingerprint to the Apache key server, 
build a release candidate, and start a vote before I go out of town on 
Thursday.  Steve, perhaps I should wait for the following release unless you 
think I'd be able to do all these steps quickly within a few hours.

John

-Original Message-
From: Beckerle, Mike  
Sent: Monday, May 10, 2021 2:43 PM
To: dev@daffodil.apache.org
Subject: EXT: Re: release 3.1.0 critical bugs still outstanding

I agree we should do the release.

I am in the thick of debugging DAFFODIL-1422, but there's a bunch of 
refactoring here, 30 files touched, changes in diagnostic behavior, etc. 
Perhaps best to put it off until after the 3.1.0 release.





From: Steve Lawrence 
Sent: Monday, May 10, 2021 1:59 PM
To: dev@daffodil.apache.org 
Subject: Re: release 3.1.0 critical bugs still outstanding

We have fixed the later two mentioned issues. The current list of critical 
issues is now:

DAFFODIL-1422: disallow doctype decls in all XML & XSD that we read in
DAFFODIL-2400: New SAX API causes performance degredations.
DAFFODIL-2473: Need bit-wise AND, OR, NOT, and shift operations

I agree the the first two can likely be postponed without issue. The last one 
doesn't even seem critical to me, unless there are very important formats that 
require this functions that I'm not aware of. I suggest we also postpone that 
ticket as well.

If others agree, I think we are ready for the 3.1.0 release?

Does any want to volunteer to be the release manager? I've done it a handful of 
times so don't mind, but it might be good to get others experience depending on 
availability. By this point, the workflow is pretty well documented here:

https://cwiki.apache.org/confluence/display/DAFFODIL/Release+Workflow


On 5/3/21 5:25 PM, Beckerle, Mike wrote:
> Of the 4 remaining "critical bugs or improvements" I think we should postpone 
> and release note these first two:
>
>   *
>
>   *   Improvement: https://issues.apache.org/jira/browse/DAFFODIL-2400 - New 
> SAX API causes performance degradations.
>  *It is a mystery why the SAX API is slower. The whole point of SAX 
> is lighter weight.
>   *   Improvement: https://issues.apache.org/jira/browse/DAFFODIL-1422 - 
> disallow doctype decls in all XML and XSD we read in.
>  *   Assigned to Mike Beckerle. Unlikely to be finished in time for 
> release 3.1.0. Substantial code refactoring to do this right.
>
> These next two seem rather important to fix:
>
>   *   Bug: https://issues.apache.org/jira/browse/DAFFODIL-2183 - Unparse 
> complex nilled element fails
>  *   There are data formats where I advised people a best-practice is to 
> use complex nilled elements to model a specific situation.
>   *   Bug: https://issues.apache.org/jira/browse/DAFFODIL-2399 - error 
> diagnostics output even though there is an infoset
>  *   This one is assigned to Steve Lawrence
>  *   Seems rather important. Was a user-reported issue I believe.
>
> The 5th critical ticket is for a new feature (bitwise and/or/xor, and shift 
> functions), so we can postpone that one.
>
> So only DAFFODIL-2183 really needs someone to take it on.
>
> 
> From: Interrante, John A (GE Research, US) 
> Sent: Monday, May 3, 2021 4:57 PM
> To: dev@daffodil.apache.org ...
>
> Are any of the 5 critical bugs (2 of which need developers to work on them) 
> going to hold up the 3.1.0 release?  The report doesn't say so, but I had the 
> impression you'd added the remaining critical bugs (which were unlikely to be 
> hit by people) to the 3.1.0 release notes so that the 3.1.0 release still 
> could go out.  If any critical bugs are holding up 3.1.0, please post links 
> to them so we can help if we have time.
>
> John
>
>
>

Re: sbt testOnly command line for running exactly one integration test?

[Steve Lawrence]

Might be an issue with using the old sbt syntax? Old syntax should work,
I think it's only deprecated, but the new syntax is:

 sbt IntegrationTest/testOnly ...

Also, integration tests are only in the CLI subproject, so things can be
a small bit quicker if you specify that, e.g.

  sbt daffodil-cli/IntegrationTest/testOnly ..

Lastly, sbt views each space separated thing as new sbt command, so if
your command has spaces in it, like you usually want with testOnly, then
you need to wrap it in quotes, e.g.

  sbt "daffodil-cli/IntegrationTest/testOnly
org.apache.daffodil.foo.TestSuite -- --tests=test_that_you_care_about"

The integration tests literally execute the CLI, so there's no way to
get around staging jars without a fundamental change in how we test. We
could maybe consider changing our CLI tests so that they call the main()
function in org.apache.daffodil.Main. That would avoid the need to stage
things. But we'd need to figure out how to capture stdin/stdout for each
unique run. Maybe each test would need a separate fork, which sbt can
do? I think we would also need to move away from expect since I think
that essentially spawns a shell and then provides input/output.

I've also wondered if there's a away to sort of share the process space
or JVM. I imagine respawning the JVM for every integration test it what
causes the majority of the slowness for the CLI tests. Would be nice if
we could spawn a single JVM and share it among different daffodil
executions. I'm not sure if anything like that exists.


On 5/11/21 9:47 AM, Beckerle, Mike wrote:
> I am wasting a bunch of time because just two integration tests are failing 
> in 
> my current work.
> 
> I can't get 'sbt it:testOnly ... ' to work.
> 
> I've used sbt testOnly before with regular tests.
> 
> Can it be made to work for integration tests?
> 
> I've asked questions about sbt testOnly before. This time I will make an sbt 
> notes wiki page.
> 
> Second issue: the build-test-rebuild-test cycle for integration tests is 
> terribly long.
> 
> Can one shortcut the full 'sbt stage' operation that makes the doc jars and 
> such, e.g., is there a sub-operation that just refreshes the code jars?
> 
> 
> Mike Beckerle | Principal Engineer
> 
> mbecke...@owlcyberdefense.com 
> 
> P +1-781-330-0412
> 

sbt testOnly command line for running exactly one integration test?

[Beckerle, Mike]

I am wasting a bunch of time because just two integration tests are failing in 
my current work.

I can't get 'sbt it:testOnly ... ' to work.

I've used sbt testOnly before with regular tests.

Can it be made to work for integration tests?

I've asked questions about sbt testOnly before. This time I will make an sbt 
notes wiki page.

Second issue: the build-test-rebuild-test cycle for integration tests is 
terribly long.

Can one shortcut the full 'sbt stage' operation that makes the doc jars and 
such, e.g., is there a sub-operation that just refreshes the code jars?


Mike Beckerle | Principal Engineer

[cid:055ceb22-3135-4a39-b05f-6f75286db035]

mbecke...@owlcyberdefense.com

P +1-781-330-0412