Re: rewrite StatelessXmlReporter fo JAXB

[Tibor Digana]

Hi Andreas,
I will not change the reporter. The reason is the risk with memory
consumption. That's why we opened this discussion. I will always prevent
from growing Surefire memory consumption.

I do not want to change encoding because there is no way to do and does not
make sense. Maybe additional documentation would help the users.

Closing streams is on my list. I would like to finish populating existing
changes made so far and then communicate with all in ML.

In 2.20.1 I want to support JDK 9 and I would like to have another format
of stream we are sending from forked JVM to Maven. The stream format is in
good progress and should fix SUREFIRE-1222 and maybe those blocker issues
we have now.


On Mon, Apr 17, 2017 at 8:33 PM, Andreas Gudian 
wrote:

> What ever you do, make sure you never try to put unbounded sysout/syserr of
> a test case into memory, ever. Not even for a short time, as there may be
> users with a ton of output that is currently buffered in the
> overflow-files. Changing this can cause serious performance issues (gc
> pauses, swapping) or just break the current behavior.
>
> Writing some XML is no rocket science, especially as our schema is very
> simple and barely changes anyway. The only thing to take care is encoding
> and closing all streams at the right time - but that won't be much simpler
> using JAXB.
>
> Just my 2 cent ;-)
>
>
> Michael Osipov  schrieb am So. 16. Apr. 2017 um
> 17:12:
>
> > Am 2017-04-16 um 16:52 schrieb Tibor Digana:
> > >>> That would still create a string in memory, won't it?
> > > It would still leak in memory while getter is called.
> > > Streaming to XML is better in this case.
> > >
> > > Do we have any alternative?
> >
> > I have to think about it and understand the whole system.
> > But why not use the regular XMLStreamWriter paired with this:
> > http://stackoverflow.com/a/2514/696632.
> >
> > You'll be able to stream whatever you want.
> >
> > > On Sun, Apr 16, 2017 at 4:40 PM, Michael Osipov 
> > wrote:
> > >
> > >> Am 2017-04-16 um 16:15 schrieb Tibor Digana:
> > >>
> > >>> Since we have XSD [1] we can generate classes and call setters in
> > >>> particular places in StatelessXmlReporter.java.
> > >>> Regarding memory consumption system-out may consume really a lot of
> > >>> memory.
> > >>> Therefore this is my proposal:
> > >>>
> > >>> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
> > >>> public Utf8RecodingDeferredFileOutputStream getSystemOut()
> > >>>
> > >>> class CdataSystemAdapter
> > >>> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo(
> > >>> stream )
> > >>> return ""
> > >>>
> > >>
> > >> That would still create a string in memory, won't it?
> > >>
> > >> [1]
> > >>> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/
> > >>> surefire-test-report.xsd
> > >>>
> > >>
> > >> This needs a rewrite, too much duplication, no typing but xs:string
> > only.
> > >>
> > >> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
> > >>> ml-node+s40175n5906482...@n5.nabble.com> wrote:
> > >>>
> > >>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
> > 
> >  Hi,
> > >
> > > I want to first talk with you about XML marshaller
> > StatelessXmlReporter
> > >
> >  [1]
> > 
> > > in Surefire which is built on the top of SharedUtils'
> > >
> >  PrettyPrintXMLWriter.
> > 
> > >
> > > I found out issues after a contributor opened this issue [2].
> > >
> > > The problem is that we are using two streams, OutputStream and
> > Writer,
> > >
> >  to
> > 
> > > create XML in file system for one reason. We write CDATA directly
> to
> > the
> > > stream apart of xml facility. We have bug with escaping illegal
> > >
> >  characters.
> > 
> > >
> > > Instead, my proposal is to create XSD and Jaxb stubs and marshal it
> > to
> > >
> >  XML
> > 
> > > file and we do need to care about low level XML and escaping
> > characters.
> > > The code will be easier to understand. The elements system-out and
> > > system-err can be large but we have the stream already in temp
> files
> > > Utf8RecodingDeferredFileOutputStream which means that the Jaxb
> > >
> >  mashaller
> > 
> > > will not keep String in memory too long - only while Marshaller
> needs
> > >
> >  the
> > 
> > > string.
> > >
> > > What is your opinion?
> > >
> > 
> >  Looking at the current code -- a lot of boilerplate. I am quite
> >  experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
> > 
> >  Consider that JAXB always holds the model in memory. If you intend
> to
> >  maintain streams within an XML you can resort fragments and other
> > stuff
> >  MOXy supports. But let me tell you that it won't be that easy as you
> >  think if you want everything as streaming with little memory usage.
> > 
> >  Michael
> > 
> >  BTW: this should go to dev@
> > 
> > >>

Re: rewrite StatelessXmlReporter fo JAXB

[Andreas Gudian]

What ever you do, make sure you never try to put unbounded sysout/syserr of
a test case into memory, ever. Not even for a short time, as there may be
users with a ton of output that is currently buffered in the
overflow-files. Changing this can cause serious performance issues (gc
pauses, swapping) or just break the current behavior.

Writing some XML is no rocket science, especially as our schema is very
simple and barely changes anyway. The only thing to take care is encoding
and closing all streams at the right time - but that won't be much simpler
using JAXB.

Just my 2 cent ;-)


Michael Osipov  schrieb am So. 16. Apr. 2017 um 17:12:

> Am 2017-04-16 um 16:52 schrieb Tibor Digana:
> >>> That would still create a string in memory, won't it?
> > It would still leak in memory while getter is called.
> > Streaming to XML is better in this case.
> >
> > Do we have any alternative?
>
> I have to think about it and understand the whole system.
> But why not use the regular XMLStreamWriter paired with this:
> http://stackoverflow.com/a/2514/696632.
>
> You'll be able to stream whatever you want.
>
> > On Sun, Apr 16, 2017 at 4:40 PM, Michael Osipov 
> wrote:
> >
> >> Am 2017-04-16 um 16:15 schrieb Tibor Digana:
> >>
> >>> Since we have XSD [1] we can generate classes and call setters in
> >>> particular places in StatelessXmlReporter.java.
> >>> Regarding memory consumption system-out may consume really a lot of
> >>> memory.
> >>> Therefore this is my proposal:
> >>>
> >>> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
> >>> public Utf8RecodingDeferredFileOutputStream getSystemOut()
> >>>
> >>> class CdataSystemAdapter
> >>> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo(
> >>> stream )
> >>> return ""
> >>>
> >>
> >> That would still create a string in memory, won't it?
> >>
> >> [1]
> >>> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/
> >>> surefire-test-report.xsd
> >>>
> >>
> >> This needs a rewrite, too much duplication, no typing but xs:string
> only.
> >>
> >> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
> >>> ml-node+s40175n5906482...@n5.nabble.com> wrote:
> >>>
> >>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
> 
>  Hi,
> >
> > I want to first talk with you about XML marshaller
> StatelessXmlReporter
> >
>  [1]
> 
> > in Surefire which is built on the top of SharedUtils'
> >
>  PrettyPrintXMLWriter.
> 
> >
> > I found out issues after a contributor opened this issue [2].
> >
> > The problem is that we are using two streams, OutputStream and
> Writer,
> >
>  to
> 
> > create XML in file system for one reason. We write CDATA directly to
> the
> > stream apart of xml facility. We have bug with escaping illegal
> >
>  characters.
> 
> >
> > Instead, my proposal is to create XSD and Jaxb stubs and marshal it
> to
> >
>  XML
> 
> > file and we do need to care about low level XML and escaping
> characters.
> > The code will be easier to understand. The elements system-out and
> > system-err can be large but we have the stream already in temp files
> > Utf8RecodingDeferredFileOutputStream which means that the Jaxb
> >
>  mashaller
> 
> > will not keep String in memory too long - only while Marshaller needs
> >
>  the
> 
> > string.
> >
> > What is your opinion?
> >
> 
>  Looking at the current code -- a lot of boilerplate. I am quite
>  experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
> 
>  Consider that JAXB always holds the model in memory. If you intend to
>  maintain streams within an XML you can resort fragments and other
> stuff
>  MOXy supports. But let me tell you that it won't be that easy as you
>  think if you want everything as streaming with little memory usage.
> 
>  Michael
> 
>  BTW: this should go to dev@
> 
>  [1] https://github.com/highsource/maven-jaxb2-plugin
> 
>  -
>  To unsubscribe, e-mail: [hidden email]
>  
>  For additional commands, e-mail: [hidden email]
>  
> 
> 
> 
>  --
>  If you reply to this email, your message will be added to the
> discussion
>  below:
>  http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlRepo
>  rter-fo-JAXB-
>  tp5906482.html
>  To start a new topic under Maven Developers, email
>  ml-node+s40175n142166...@n5.nabble.com
>  To unsubscribe from Maven Developers, click here
>    acro=unsubscribe_by_code&node=142166&code=dGlib3JkaWdhbmFAYX
>  BhY2hlLm9yZ3wxNDIxNjZ8LTI4OTQ5MjEwMg==>
>  .
>  NAML
>  

Re: Publish Maven releases on SDKMAN!

[Manfred Moser]

I agree with Stephen... we should NOT do binary releases for any other system 
than the existing tar.gz and zip to Central/Apache Archive. If users of a 
specific ecosystem want to distribute Maven within their system .. fine. But 
its up to them to do it. All others like Linux distro packaging, brew, ports 
and many others do that. sdkman should be in the same category ... and if they 
dont want to distribute Maven .. thats fine with me (and us..) too.

manfred

Stephen Connolly wrote on 2017-04-16 06:49:

> On Sun 16 Apr 2017 at 11:32, Marco Vermeulen  wrote:
> 
>> Hi Maven folks,
>>
>> I really haven't come here to start a flame war or to give justification
>> about why/how I built SDKMAN. Even less, justifying why I gave it such a
>> "silly" name. I have come here with friendly intent to help your community
>> publish their own releases on SDKMAN. I've come back since I was urged to
>> do so [1] recently on Twitter by the @ASFMavenProject account.
> 
> 
> Twitter is not a good place for discussion.
> 
> We currently have 30 steps (some complex) to get a release out the door.
> 
> We want every committer capable of acting as release manager for core (I am
> currently acting in that role because we had a failure to get releases out)
> 
> To adopt the push model would require:
> 
> 1. Adding another step.
> 2. Either keeping shared credentials in a PMC only shared space (and
> restrictions on release manager to PMC) or that the PMC has to step up and
> take responsibility for *another* step
> 3. Even then, we have to remember to do it.
> 
> A pull model can be very simple: you can define the *exact* format of the
> page on maven.apache.org. We use a template to populate the info. Every
> time we release, the template will be regenerated as part of the site
> release anyway (no extra manual step)
> 
> I want to remove steps from the release process, not add. Make it easy for
> me to remove steps and you are my friend. Suggestions to add more do not
> make me happy
> 
> Publishing
>> is as simple as a small API call which can be handled by anyone wishing to
>> take up this responsibility in your community.
>>
>> I won't be defending myself or arguing with anyone on this forum going
>> forward. I'm simply asking you for help. Plain and simple. I won't be
>> changing the way SDKMAN works, since it already works well and is very
>> reliable. All our vendors love it, and are very happy to integrate with us.
>> In the process we all win: vendors are given control of their own releases
>> and get another channel to distribute their software free of charge, and
>> users have a convenient easy-to-use way to install their favourite
>> software. I get to stand in the middle and watch it all happen asking
>> nothing in return. I like contributing my free time to make developers
>> happy and easing their installation pains. This is what I do to make our
>> open source world a better place.
>>
>> If anyone in the community wants to step up and help with this process,
>> feel free to contact me and I would be more than happy to get you set up. I
>> would issue some API credentials and this would empower you to do a simple
>> REST call to make Maven available to all our users across all platforms.
>> Feel free to send me a mail at ma...@sdkman.io
>>
>> Many thanks!
>> Marco.
>>
>> [1] https://twitter.com/ASFMavenProject/status/851132246669103108
>>
>> On Sun, 16 Apr 2017 at 10:52 Aldrin Leal  wrote:
>>
>> > just fyi: github releases has an atom feed.
>> >
>> > https://github.com/apache/maven/releases.atom
>> >
>> > --
>> > -- Aldrin Leal,  / http://about.me/aldrinleal
>> >
>> > On Sun, Apr 16, 2017 at 4:49 AM, Marco Vermeulen > >
>> > wrote:
>> >
>> > > Hi Heinz,
>> > >
>> > > On Sun, 16 Apr 2017 at 08:41 Karl Heinz Marbaise 
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > On 16/04/17 00:56, Marco Vermeulen wrote:
>> > > > > Hi Maven folks,
>> > > > >
>> > > > > Some time ago I asked the Maven dev community whether they would be
>> > > > willing
>> > > > > to publish their releases on SDKMAN! [1] using our Vendor API [2].
>> > > > > Unfortunately, my request was met with scepticism and ultimately
>> > > resulted
>> > > > > in no action taken.
>> > > >
>> > > > What happended out of the idea to scan automatically[1] for new
>> > versions
>> > > > and insert the data automatically via a crawler which can for example
>> > > > scan maven central[2] where all releases of Maven will be distributed
>> > > > (also there is a REST API on Maven Central)...or the distirbution
>> > > pages...
>> > > >
>> > > >
>> > > The idea to scan/crawl is not feasible for me as crawlers are brittle
>> and
>> > > time consuming to maintain. I provide SDKMAN in my spare time to help
>> > > others, so don't want to spend my time maintaining something like this.
>> > > Moreover, why write a piece of infrastructure when a simple API call
>> from
>> > > your side would do the trick? It would take almost no effort.
>> > >
>> > >
>> > > >
>> > > > [1